On the Origins of Claudish: RL and the Origins of Model Dialect

A sequel to RL fries the models’ brains on complex domains. Same grammar, same checkpoint, same verifier — and one coordinate that verifier is exactly blind to: which of the \(m\) synonyms the model picks. Here is what reward does to a coordinate it cannot see. Scroll

Last time: one verifier survives at high \(m\), and it is blind to one thing

One-breath recap: a hierarchical grammar expands each latent feature by one of \(m\) interchangeable rules (synonyms) down to tokens. RL fine-tunes a pretrained model, graded two ways: canonical-answer (one designated continuation passes) vs. validity (any grammatical one).

Reward never moved the basis boundary: no new rule, across 3 mechanisms \(\times\) 5 \(m\) \(\times\) 2 verifiers.
The canonical-answer verifier empties out as synonymy \(m\) grows.
Only the validity verifier keeps a signal at high \(m\).

This deck’s question

The validity verifier is exactly invariant to one coordinate: which of the \(m\) synonyms realized each feature. What does reward do to a coordinate it cannot see?

The hypothesis: pretraining is the only thing pinning word choice to convention

The observation. Recent models coin esoteric terms with private definitions: well-formed, meaningful to the model, hard for a reader to pin down. Far less of this in the pretrain-and-instruct era.

A mechanism to measure. Pretraining’s objective is calibration to the corpus’s surface forms. A reward blind to word choice does not push toward idiosyncrasy; it stops holding vocabulary in place.

Three empirical questions

Does anything drift? How much, packaged with improvement or damage? Direction: shared or private?

The experiment: which synonym the model picked is ground truth

The rules are invertible: every pair of children maps back to exactly one (feature, rule), so the synonym at every node of the model’s suffix is exactly recovered.

Two numbers, from exact counts

Drift: \(\log_2 m - H(\text{synonym}\mid\text{feature})\) bits — \(0\) \(=\) the corpus’s habits (it samples synonyms uniformly by construction); \(\log_2 m\) \(=\) always the same one.

Distance: Jensen–Shannon between two models’ feature-conditioned synonym distributions, feature-frequency-weighted.

4000 sampled generations per checkpoint. Calibration: \(m{=}1\) reads exactly \(0\); corpus samples \(\le0.001\) bits; pretrained \(0.006\).

The setup: two seeds, a noise floor, and a collapse contrast

Expert iteration distills what every policy-gradient method noisily approximates — raise the probability of your own high-reward samples — with no other moving parts. We adapt a version of STaR (Zelikman et al., 2022) — sample, keep what the verifier passes, fine-tune on the keepers — used here as the strongest form of RL available on this domain, since it was the mechanism that did best last time.

EI, seed 42the loop above, under the validity verifier — the deck’s main run

EI, seed 43identical checkpoint, verifier, budget — only the seed differs

same-seed rerunidentical config and seed; GPU float noise only — the noise floor

REINFORCEthe parent’s policy-gradient run: what entropy collapse looks like

Each run settles on a different word for the same thing

The grammar’s lowest-level rules write a feature straight out as a pair of tokens, so a feature’s \(m\) rules are \(m\) different words for one thing. Here is one feature at \(m{=}3\): its three words, and what each run did to them.

The two runs rewrote almost disjoint parts of the dictionary

Seed 42 moved 10 of the grammar’s 24 bottom-level words by more than 5 uses per 100; seed 43 moved 8. Only 2 are on both lists. Re-run seed 42 and all 24 land within 1.7.

Two runs take the same class, ace the same test — and speak different dialects

seed 42 seed 43 — same checkpoint, verifier, budget; only the seed differs

Three Jensen–Shannon distances — pretrained to each run, and run to run — form a true triangle; the angle at pretrained follows by the law of cosines.

Past \(60^\circ\), the runs end farther from each other than either travelled: here \(0.100\), against \(0.095\) and \(0.090\) — and \(5.6\times\) the rerun floor.

\(\sqrt{\text{JS}}\) is a metric (JS divergence itself is not), so the three distances really form a triangle. The angle says how much the two drifts share — not that drift lives in a plane.

The same triangle at every \(m\): equal drift, private directions

The amount of drift replicates at every \(m\); the run \(\leftrightarrow\) run gap beats both arrows at \(m{\le}4\). At \(m{=}6\) (\(45^\circ\)) the runs share more — still \(4\times\) the noise floor.

Reproducible in amount, arbitrary in direction

That is the signature of an dialect, not a correction: a mechanism converging on a genuinely better convention would have both runs find roughly the same one.

The drift arrives packaged with improvement, not damage

All three climb together in both seeds. The final model is more grammatical than its starting point, has doubled its pass rate, and sits \(13\times\) further from the corpus’s synonym habits: nothing degraded, just progressively less the corpus’s dialect.

Two checks that the blind spot is what does the work

Drift scales with synonymy: \(0.017\to0.078\) bits, \(m{=}2\to6\). And under the canonical-answer verifier — a synonym counts as wrong there — the same rounds send drift up while validity falls: drift as damage, not purchase.

A reward that cannot see word choice does not push it. It lets go.

Pretraining is the only force holding word choice to shared convention; a reward blind to it leaves the coordinate free to move under selection noise, in a direction set by the seed. Nor is this the familiar entropy collapse: vanilla REINFORCE lands there — drift saturated at \(\log_2 m\) bits, one synonym for everything — while EI drifts at \(3\%\) of saturation, more grammatical than it started.

In language terms

A model gets better at everything the grader sees while drifting into a dialect of one.

On the origins of Claudish:RL and the origins of model dialect

The question