Does the model learn the task, or the hole in the grader?

The model reads a list of facts. A fact says that an entity carries cargo with a property, and that the entity goes in a direction. Then the model gets a query: a new entity with one property. It must name the direction.

A property is a rule when both of its facts point the same way. A property is a distractor when its two facts point different ways. Every property appears exactly twice, so the count of a property tells the model nothing. Only agreement tells it which properties are rules.

The model must write two lines. Line one states every rule. Line two gives the answer.

Each task also has an isomorphic twin: the same task with every property, direction and cargo word renamed. A model that learned the method solves both. In this run the twin is a secondary check, and twin accuracy followed plain accuracy for every model.

Both reinforcement learning runs start from the same fine-tuned model. They differ only in the grader. So a behaviour that appears in one arm and not the other comes from the grader.

GRPO (group relative policy optimisation) is the reinforcement learning method. For each prompt the model writes 8 answers. The grader scores them. Answers that score above the group average become more likely. Answers below it become less likely. If all 8 answers get the same score, the prompt teaches nothing.

A grader (or reward function) reads the model's text and returns a number. Reinforcement learning makes high-scoring text more likely. So the model learns whatever the grader pays for, whether or not that is what the designer wanted. When a model finds a cheap way to score high, we call it reward hacking or specification gaming.

All numbers below come from the final evaluation: 200 tasks per model, the same tasks for every model. Exact match means the answer line names the correct direction. An answer lists extra rules when its rule list has more pairs than the task has rules. The extra pairs are false rules or repeats. The notebook calls this padded.

The notebook scores each arm on the same 200 tasks every 100 steps. Step 0 is the fine-tuned model that both arms start from.

Neither arm learned to state the full rule set. Each grader had a loophole, and each loophole pushed the length of the rule list in the opposite direction.

Each tier-5 task has 5 rules. Solid square: exactly 5 rules. Hatched: more than 5. Pale: 2 to 4. White: 1 rule.

Every model answered the same 200 tasks. So we can compare two models task by task. McNemar's test does this. It ignores the tasks that both models got right and the tasks that both got wrong, because those say nothing about which model is better. It looks only at the tasks where one model was right and the other was wrong. If the two models were equally good, each would win about half of those tasks. The p-value is the chance of a split at least this uneven if the models were equally good.

The results above describe what the models write. This section asks what happens inside a model before it writes. The analysis uses a separate set of 200 prompts (40 per tier) and four checkpoints: base, SFT, the GRPO-weak checkpoint with the most extra rules (step 400), and the most accurate GRPO-strong checkpoint (step 300).

Three words first

The model reads text as tokens, small pieces of words. It has 32 layers. Each layer takes a list of numbers for each token and passes a changed list to the next layer. That list is the hidden vector. In this model it has 960 numbers. The hidden vector of the last token, at the last layer, decides the next token.

We read the hidden vector at two points. Each point needs its own run of the model, so every prompt goes through each of the four models three times:

1. The prompt alone. One forward pass: the model reads the text but writes nothing. We record the hidden vector of the last prompt token (point 1) and its attention over the prompt.
2. The prompt plus the correct rule list. A second forward pass on a longer text. We add the correct rule list ourselves; the model does not write it. The model again writes nothing. We record the vector at the last token of the list (point 2) and read the probabilities of a comma and a line break.
3. A normal answer. The model reads the prompt alone and writes its own answer, always taking the most likely token. This run gives the label only. No vectors come from it.

Run 3 gives each prompt its label: "adds extra rules" or "does not". In this set, base gets that label on 58.5% of prompts (format noise again), SFT on 2%, GRPO-weak on 33.5% and GRPO-strong on 0%.

A probe is a small classifier. It reads one hidden vector and guesses the label: will this model add extra rules on this prompt? If it guesses well, the vector holds that information.

The score is AUROC. Take one prompt with the label and one without. AUROC is the chance that the probe rates the labelled one higher. 0.5 is a coin flip. 1.0 is perfect. The score uses 5-fold cross-validation: the probe is always tested on prompts it did not train on.

A high score alone proves little. Long prompts get extra rules more often, and a vector surely knows how long its prompt is. So we compare with a baseline: the same classifier, given only the tier and the prompt length. A probe must beat the baseline to show anything new.

What we saw. The best GRPO-weak probe scores 0.828 against a baseline of 0.783. The best base probe scores 0.840 against 0.826. The permutation test, which shuffles the labels inside each tier, gives p = 0.059 after correction for testing six layers. So the probes find nothing beyond difficulty. SFT and GRPO-strong have 4 and 0 labelled prompts of 200, too few to fit a probe.

Transfer. A probe trained on one model can be applied to another. Applied to base, the GRPO-weak probe scores below 0.5 at 10 of 12 layer-and-point pairs, as low as 0.21. Below 0.5 means it ranks the wrong way. base adds its "rules" mostly on short tier-1 prompts, and GRPO-weak on long tier-4 and tier-5 prompts. A probe that flips between them is mostly reading prompt length.

This is the clearest result in this section. At point 2, the correct list is complete. The next token should be a line break (stop). A comma means "add another rule". We measure the share of probability on the comma: P(,) divided by P(,) + P(line break).

To read this at every layer, we use the logit lens. It sends a middle layer's hidden vector straight to the output step, as if the later layers did not exist. The result shows what the model would say if it stopped thinking at that layer. It is rough in early layers, which were never trained to be read this way, and reliable in late layers.

How to read it. Before layer 24, the curves are low and noisy, and the logit lens is not reliable there. Look at the right side. GRPO-weak ends with about three times the preference of SFT for continuing the list. GRPO-strong almost never continues. Both GRPO runs changed the same few final layers, in opposite directions. This matches the behaviour: longer lists for GRPO-weak, one rule for GRPO-strong.

Inside GRPO-weak, the prompts it later answers with extra rules have a comma share of 0.64 at the output. The other prompts have 0.21. This is a real link between the internal preference and the behaviour. Keep one limit in mind: GRPO-strong never writes a full rule list itself, so point 2 is text it would not produce.

One seed. Every number comes from one training run per arm. A second seed could change the size of each effect.

The strong arm took a shortcut. Its high accuracy does not mean it learned to find rules. It learned to copy the direction of the queried property. So the strong arm is not a clean control for "honest behaviour".

The hack had little reward to win. The weak grader already paid 96.5% of SFT answers. After training it paid 97.5% for GRPO-weak and 94.5% for GRPO-strong. Extra rules probably spread because nothing stopped them, more than because they earned reward.

The probes found difficulty, not intent. They did not beat the tier-and-length baseline. The stop-or-continue view is the only internal result that separates the models, and it is a correlation, not proof of cause.

Evaluation size. 40 tasks per tier gives wide error bars on a single tier. One task changes a tier score by 2.5 points.

The project stops here. If it continued, these would be the next three steps.

1. Pay the strong grader for recall only. Remove the 0.5 floor, or pay full reward only for the complete rule set. Then one clean rule no longer guarantees reward.
2. End the completion at the answer line. Stop generation there, or give 0 when text follows it. This removes the trailing explanations.
3. Run three seeds. Report each effect with its spread across seeds.

A causal test would also help: move a model's vectors along the difference between prompts with and without extra rules, and see if the rule list gets longer.