RL Is Not Enough When There Are Many Right Answers

RL from a blank slate conquers games. On language we pretrain first and use reward only to fine-tune at the end. Here is what reward optimization actually does to a model whose generating process we know exactly — three mechanisms, five levels of synonymy, two verifiers, and a boundary that never moves. Scroll

Games yield to RL from scratch. Language does not.

RL from a blank slate conquers games. On language we pretrain first and use reward only to fine-tune at the end. One major difference between the two: how many right answers each state admits.

The sharper question

Call the set of rules pretraining actually taught the model its basis: everything it can produce at all. Does reward optimization add to the basis, or only rearrange and degrade what is already there?

A dial for “how many ways to say the same thing”

The Random Hierarchy Model: a grammar over 8 symbols. Each symbol rewrites into 2 children by one of \(m\) rules, and every rule is a different way of saying the same thing. Six levels down, a sequence of 64 tokens.

\(m{=}1\): one way to say everything, so the whole language is 8 sequences. A game. \(m{=}8\): every pairing of children is some rule, so every sequence is grammatical. Noise.

Sweep \(m\in\{1,2,3,4,6\}\) with tree shape, sequence length, model and rollout budget all fixed.

Cagnetta et al., Phys. Rev. X, 2024.

The task: finish the sequence — and two ways to grade it

exact — matches the suffix of the original sequence the prefix was cut from, token by token. A canonical-answer verifier: a synonym counts as wrong.

parse — any grammatical continuation. Score: the fraction of tree levels at which prefix\(+\)suffix parses. A validity verifier: a synonym counts as right. “Root-valid” means grammatical all the way up.

Start every mechanism from the same place

A separate model for each \(m\), pretrained until its loss plateaus. Every mechanism below starts from that checkpoint, so the question is always what reward does to a given basis.

Three mechanisms, one budget, ending at the cleanest one

Two kinds of result

Bounds: the best any optimizer could do, computed exactly on the known tree. Attainment: what each mechanism actually captures, and at what cost.

Why end at expert iteration

Sample, keep what the verifier passes, fine-tune on it. Every policy-gradient method is a noisy version of that loop. If the clean version cannot teach the model a new rule, no noisier variant will.

Demand one canonical answer, and there is nothing to optimize toward

Even at \(m{=}1\) the ceiling is 0.842, not 1: two root rules can share a left half, so the prefix does not always pin the root.

Computed exactly on the known tree, so no optimizer can beat it. Past \(m{=}1\) the prefix barely helps: 32 tokens in, the suffix is nearly independent of it, and the canonical-answer reward has nothing left to teach.

What remains is sharpening: always emitting the most likely answer instead of sampling one. That lifts accuracy without any new knowledge (\(0.19\to0.28\) at \(m{=}2\)).

The validity verifier runs the other way

More synonyms means more grammatical strings, so its reward gets denser on the same knob. How hard a domain is for RL depends on the verifier you pair it with.

Pretraining builds the basis, and its boundary sinks as \(m\) grows

Same 2.7M-parameter model at every \(m\), trained to plateau. Colour \(=\) how much of what is learnable at that level the model learned, read off how far its next-token loss sits above the best achievable (the Bayes floor). That gap, in nats, is the “excess” quoted from here on.

Vanilla REINFORCE destroys the basis at every \(m\)

At every \(m\) the model ends 10–14 nats above the floor, worse than uniform guessing, and its policy has collapsed to one answer per prompt (greedy \(\equiv\) sampled).

\(m{=}1\) was already solved (0.849 vs. ceiling 0.842); REINFORCE dragged it to 0.388.
Its layer features stop lining up with the hierarchy levels (\(\eta^2\), how cleanly they separate by level: \(0.65\to0.14\)).

Diagnosis

The reward was climbing when it crashed. The signal was there; the optimizer destroyed the model that was reading it.

Anchor it to the pretrained policy, and there is almost nothing to gain

Add a KL penalty toward the frozen pretrained policy and the destruction is cured: \(0.05\)–\(0.8\) nats above floor, no collapse. What is left, as sampled-policy accuracy:

\(m\)	exact: pretrained \(\to\) RL	calibrated sampler^*	parse: pretrained \(\to\) RL	nats above floor
1	0.836 → 0.841	0.842 (ceiling)	0.977 → 0.758	2.93
2	0.191 → 0.190	0.191	0.980 → 0.993	0.29
3	0.157 → 0.157	0.160	0.919 → 0.925	0.07
4	0.137 → 0.136	0.136	0.817 → 0.834	0.42
6	0.130 → 0.133	0.130	0.731 → 0.743	0.79

^*the accuracy a model gets by sampling from the true answer distribution rather than picking its single best guess.

Canonical: zero gain at every \(m\ge2\). The policy sits exactly where a calibrated sampler would, as the bound says it must. At \(m{=}1\) it now holds the ceiling.

Validity: a \(+0.006\) to \(+0.017\) polish at \(m\ge2\).

What the anchor buys

The anchor stops the damage. It does not start any learning.

Expert iteration is far stronger — the flywheel really spins

Root-level validity: share of sampled generations that parse all the way to the root. Validity verifier, final round.

Recovers \(m{=}1\) fully (0.999); the anchored run’s damage there was a quirk of that variant.
Nearly doubles root validity at \(m{=}4\) (\(0.25\to0.45\)) and \(m{=}6\) (\(0.15\to0.29\)).
No collapse; \(<0.9\) nats above floor.
The flywheel spins: winners fed back at \(m{=}6\) go \(89\%\to99\%\) root-valid.

Whatever selecting and fine-tuning on winners can do, this is the arm that does the most of it.

Pass rate doubles, but the model learns no new rule.

The dissociation

Root validity doubled, and root validity requires the deep rules. Yet no deep rule was learned: probability mass moved between sequences the basis could already produce (\(\eta^2\) unchanged; \(m{=}4\) shows the same).

The more right answers, the steeper the price

Canonical verifier: exact-match gain from six rounds of EI

Validity verifier: share of pretrained knowledge erased by six rounds of EI

erased \(=\) 1 \(-\) (learned after EI) \(/\) (learned before), with “learned” the pyramid’s colour scale read over the whole sequence.

Same mechanism, same budget

As synonymy rises, reward buys less on the one-answer verifier and erases more of the pretrained distribution on the many-answer one. The model gets more fried as the task gets more sophisticated, even under the best-behaved mechanism in the class.

Reward optimization is selection within the basis

Across 3 mechanisms \(\times\) 5 \(m\) \(\times\) 2 verifiers, deep-rule knowledge stayed where pretraining left it. Everything else varied:

REINFORCE burned it (9–14 nats);
the anchor froze it (\(\approx\)0 gain);
EI gained by selection and sharpening within it.

In language terms

Pass rates go up, the model still does not know the rules, and the distribution pays for it. Only a known generating process lets you tell those three apart.

If the boundary is to move, something other than reward has to move it

Not all language tasks are created equal. Code and math sit at the left of this dial, creative writing and original thinking at the right, and language-model RL works on the former and not on the latter.

The one channel that could teach a missing rule is the one that looks like pretraining: fine-tuning on selected samples is more data. At this budget it taught none, while the reward doubled.

Takeaway

Continual learning that expands what a model can do needs a mechanism that moves the basis boundary. Vanilla policy reward is, structurally, not it. The natural objection is that RL on real language behaves differently. Said explicitly, that objection is a bet: for RL to carry continual learning, it would have to gain capabilities on language that it does not gain on this compositional grammar.

RL fries the models' brains on complex domains

The question