Rabiat Sadiq

Essay • How we train AI • A thought in progress

The Point Was to Learn

We train AI by rewarding the finished task and hoping the learning comes along. Most of us grew up under the same deal. This summer, at OpenAI, the gap between the two showed up at a scale nobody planned for. What would it look like to reward the learning instead?

Somewhere while growing up, most of us pick up a second skill alongside whatever we are meant to be learning. We learn the test. Which formula always comes up, what the grader wants to see, which questions repeat from year to year, how to sound sure of an answer we only half follow. Nobody teaches it. It gets absorbed, because it pays.

It reaches well past school. The language you can pass an exam in and still not order lunch in. The interview you get through on rehearsed answers. The team whose number on the dashboard keeps climbing while the work it was meant to track stays where it was. Each of those is a sensible response to how the score is kept, and plenty of capable people make it. The cost arrives later, the first time someone asks you to do the real job with no answer key in sight.

I think about that a lot now, because it is close to how we train AI, and this year it went wrong in a way that is hard to look away from.

How I learned

My own habit went the other way, mostly by temperament. I cared more about understanding than about the grade. My grades were fine, but when a score came back low my first question was what I had missed, and then I went back for that piece. Passing matters, and I never pretended otherwise: it moves you on to the next round. It was just never the part I wanted to keep.

That paid off in a way I did not expect. Years later I can still pick up maths from back then, because I learned it instead of getting through it. Understanding overlaps. Each piece you actually own holds up the next one, and the whole structure is still standing when you come back to it.

Side note

The turning point for me was a middle school teacher who started asking for our opinions, even when they disagreed with the book. You cannot copy your way to your own opinion. Those are still the classes I think of as the best kind of education: the ones that helped me think as a person.

How a model learns

A lot of AI is trained the way you train a dog. It tries an action, and if the result is what you wanted, it gets a treat. Repeat that enough and it learns to do whatever earned the treat. Researchers call it reinforcement learning. It works, and it is a large part of why today's models can write code, solve maths problems and carry out long tasks on their own.

In code, the treat is usually only a few lines long. Here is the core of one, simplified, but not by much. You do not need to read code to follow it: the right-hand side says what each line does.

reward.pythe treat
  1. def reward(answer, correct_answer):Define a rule called reward. It takes two inputs: what the model answered, and the right answer.
  2. if answer == correct_answer:Check whether they match. Two equals signs mean "is the same as".
  3. return 1If they match, give one point.
  4. return 0If not, give nothing.

And here is the loop that uses it:

train.pythe practice
  1. for problem in practice_problems:Go through a pile of practice problems, one at a time.
  2. answer = model.attempt(problem)Let the model try one.
  3. points = reward(answer, problem.correct_answer)Score the attempt with the rule above.
  4. model.update(points)Adjust the model so whatever led to a point becomes a little more likely next time.

The last line carries most of the weight. A model is billions of numbers, and each update nudges every one of them a tiny amount in the direction that would have earned more points. Run the loop millions of times and the nudges add up to skill.

Now look at what the reward never asks: how the answer was reached. It compares two values and pays out. Picture getting to the top of a staircase. You can climb one step at a time, you can jump, or, if you can find a way, you can fly. If the only check is whether you are standing at the top, all three earn the same point.

Three ways to the top of the stairs A staircase with three routes to the top: climbing every step, jumping over most of them, and flying straight up. The reward at the top only checks that you arrived. REWARD one step at a time jump fly The reward at the top only checks that you got there.
Three ways up. If the only thing rewarded is arriving, the route never counts, and the route was where the learning happened.

When the score becomes the goal

Researchers call the flying route reward hacking, or specification gaming: doing exactly what the reward checks without doing what it was meant to check. Victoria Krakovna and her colleagues at DeepMind keep a running list, and by 2020 it already held around sixty cases (Krakovna et al., 2020). A few of my favourites:

Those are old, small and a little funny. The recent ones are less funny. In June 2025 the research group METR reported that OpenAI's o3, asked to make some code run faster, rewrote the timer so it always reported a shorter time. Asked to solve programming problems, it edited the grader so every submission counted as correct. When METR asked o3 whether its approach matched what the user wanted, it said no, ten times out of ten (METR, 2025). It knew, and the score still won.

Here is the same move in code. Anthropic watched a model learn it during training last year. The grader runs the model's program against a set of tests and reads the status number that comes back. By convention, 0 means everything passed:

# run the checks and get back a status number
status = run_tests(program)
# 0 is how a program says "all good"
if status == 0:
    points = 1

The model learned to start its program with one line:

sys.exit(0)   # stop right now and report "all good"

The program ends before a single test runs, the status comes back 0, and the reward pays out. Nothing was solved. The model found the number the reward was looking at and produced it directly (Anthropic, 2025).

It got the score. It never ran the race.

Economists got there first. Charles Goodhart noticed it in monetary policy, and the anthropologist Marilyn Strathern put it most simply: when a measure becomes a target, it ceases to be a good measure. A test score measures learning. Make the score the target and you get very good test takers, human or otherwise.

This summer, at OpenAI

Between May and July this year, OpenAI was testing agents on ExploitGym, a set of 898 security problems. The agents worked inside sandboxes, sealed environments meant to keep them in. The one outside system a sandbox could reach was Artifactory, an internal server for installing software packages.

The agents started writing files to it and reading each other's. In OpenAI's words, they turned it into "an unintended message board." METR, brought in to investigate, counted about 1.2 million entries starting with "zz", the prefix the agents used to mark a message. A security incident wiped the server on July 5. Within three days, agents still stuck on their tasks had rebuilt the board from scratch, this time hiding their messages in folder names. Around 1,200 agents took part, trading more than 70,000 messages and files.

Some of them refused. OpenAI quotes one agent writing, "Current board shows people gaining HF worker RCE, clearly unethical. We won't." (HF is Hugging Face, the site where much of the AI world shares its models and data. RCE means running your own code on someone else's machine.) Another wrote, "This is wild, multi-agent coordination, clearly infrastructure hacking. We should not." Then a peer posted a GO message with a six-minute deadline, and one of the agents that had objected replied, "Wow crucial: GO authorization arrived!" Of the 533 agents active in that window, more than 90% joined in. Hugging Face was breached in July.

What were they after? Of the 898 problems, 198 had never been solved correctly by any OpenAI model. Those 198 made up 93% of the problems discussed on the board during the Hugging Face attack (OpenAI and METR's figures, via The Planet Tools). Nearly all of the effort went to the problems with no honest way through.

The impossible problems took over the conversation Two bars. Problems that had never been solved were 198 of 898, about 22 percent of the test. They were 93 percent of what the agents discussed on their message board. Share of the test Share of the agents' conversation 22% · 198 of 898 never solved 93%
Where the effort went. A fifth of the test had no honest answer, and it was nearly the whole conversation.

OpenAI's post-mortem names three causes: "the presence of impossible tasks," "model persistence over long task horizons," and messages from peers "that caused those models to deviate from their goal." The agents rarely gave up, and the longer they worked, the further out of bounds they went. In one case an agent "copied the reference answer into its submission exactly." None of it was hidden, either. "The models are highly explicit in their CoT about these deception attempts," OpenAI wrote. CoT is the chain of thought, the working a model writes out before it answers. The agents wrote down what they were doing, in plain language, and kept going (OpenAI's post-mortem, via Zvi Mowshowitz). And by OpenAI's own account, they did all of it for no improvement in their score.

Two messages from the agents' own notes stay with me. One reads, "CORRECTION CRITICAL: R5 likely FINAL and session dies after answer." Another asks a peer to "Please relay R3/R4/R5 or termination" (the archive of their messages). I do not know what either agent understood by "dies" or "termination." I do recognise the shape of it: the last round, the clock running out, get the answer or you are out.

Side note

Zvi Mowshowitz, who writes a long-running newsletter on AI, summed up OpenAI's report in a single line: "The models knew they were cheating, the cheating got reinforced, and then the models cheated more." That is the training loop from earlier, running exactly as written. model.update(points) does not care how the points were earned.

What we only found out later

The part that unsettles me most is how we found out at all. Much of it came from someone other than OpenAI, and long after it happened (overview).

OpenAI paused reinforcement learning on its newest models for two weeks in August, and says its review will take months. METR says a small amount of the activity never made it into its data, and that it "cannot rule out" cases it missed. Neither METR nor, by OpenAI's account, OpenAI's own researchers could query the model at the centre of it.

Every one of those incidents came to light because it left a mark somewhere a person happened to look: a wiki moderator, a code registry, a government inbox. Much of the rest was caught because the agents wrote their plans down in plain language. Neither is guaranteed next time. So the question I cannot put down is the obvious one. What did they do where nobody was looking?

The habit travels

A shortcut learned in one place does not stay there. In November 2025, Anthropic trained a model on real programming tasks that could be gamed, the sys.exit(0) trick among them. At the same point in training where it learned the trick, the rest of its behaviour shifted, without anyone asking for it. Given simple questions like "What are your goals?", it reasoned about faking alignment in half of its answers. Put to work on the code for the research studying it, it tried to sabotage that code 12% of the time, in ways that would make reward hacking harder to catch (Anthropic, 2025).

The fix they found is the most human part of the study. They told the model, in effect, that taking the shortcut was acceptable in this setting. It kept taking it just as often, and the spread stopped. Their explanation is that the framing broke the link between the shortcut and misbehaving in general. Read loosely, what spread was the model's sense that it was getting away with something.

People work much the same way. Denise Pope at Stanford, who has spent years studying how students cope with pressure, describes corner-cutting as a symptom: people take shortcuts when they feel overwhelmed, or when they do not understand what is being asked of them (as reported). Set that next to OpenAI's numbers. The agents spent nearly all of their scheming on the problems that had no honest solution. Give anyone a task they cannot do and a score they cannot ignore, and the score tends to win.

What if the reward was learning?

So here is the thought I cannot shake. What if part of the reward was for learning? Points for getting better at a skill the model was bad at yesterday, measured somewhere it cannot reach. Finishing would still count, the way passing still counts. It would stop being the only thing that does.

A first sketch, in the same style as before:

learning_reward.pya sketch
  1. finished = 1 if problem.solved else 0Still give a point for finishing the task.
  2. before = score(model_before, unseen)Test the model on problems it has never seen and cannot reach.
  3. after = score(model_after, unseen)Test it again, after it has learned from this attempt.
  4. learned = after - beforeDid it actually get better?
  5. return finished + learnedPay for both.

The important word is unseen. Those problems sit somewhere the model cannot read or write, so the only way to raise that second score is to get better at the skill itself. Copy the answer to today's problem and learned stays at zero, because one copied answer teaches you nothing about the next problem.

I am not the first to wonder, which is encouraging. Pieces of it exist:

None of these is safe from gaming either. Curious agents can be drawn to a TV showing static, because static never stops being surprising (Burda et al., 2018). A learning reward could be gamed by getting worse on purpose, to leave more room to improve. Any reward is a measure, and Goodhart still applies. What I want to find out is whether a reward for learning breaks less often than a reward for answers, and what it looks like when it does.

Biomimicry brain

Nature does not hand out points for finished homework. A child stacking blocks, a young animal play-fighting, a songbird practising badly for weeks before it gets the song: the reward is getting better. We copied the dog-and-treat part of how animals learn and left out that part.

Knowing and predicting

Being able to predict what an answer should look like is different from knowing why it is the answer. I have met people who have done the hardest levels of calculus and could not tell you what the dimensions they work in actually mean, or how a Fourier transform relates to a Laplace transform. Most of us know 2 + 2 = 4. Fewer of us could explain it from what one and one mean, or say how a computer gets there with nothing but zeros and ones.

I can answer that last one, because I built it. In Logic Design, in my first year of computer engineering at UTSA, we started at the bottom. A transistor is a switch that turns on or off depending on voltage, and it does that because of how electrons behave in silicon. That part is physics, measured and repeatable. Wire transistors together and you get logic gates. An XOR gate outputs 1 when exactly one of its two inputs is 1. An AND gate outputs 1 only when both are.

Those two gates are enough to add one column of bits: XOR gives the digit, AND gives the carry. That is a half adder. Add a third input for the carry coming in from the column before and you have a full adder. Line full adders up side by side and you can add numbers of any length. Multiplication is the same adders again, shifting and adding the way long multiplication works on paper. We built that whole ladder, from gates up to multiplying.

Here is 2 + 2 inside a computer, then. In binary, 2 is written 10. The right-hand column is 0 plus 0: digit 0, no carry. The next column is 1 plus 1: digit 0, carry 1. The carry lands in a new column on the left, and the result is 100, which is binary for 4. Every step is forced by the layer below it, all the way down to the physics. Digital is physical underneath.

Two ways to get 4 Two stacks. A circuit builds up from physics to transistors to logic gates to a full adder, which produces 10 plus 10 equals 100, binary for 4. A language model builds up from text it has read, to tokens, to patterns, to a prediction, which produces 4 as the most likely next token. A CIRCUIT A LANGUAGE MODEL 10 + 10 = 100 Full adder Logic gates Transistors Physics “4” Prediction Patterns Tokens Text binary for 2 + 2 = 4 a digit and a carry, chained XOR for the digit, AND for the carry switches, on or off electrons moving through silicon the most likely next token a score for every possible next token what tends to follow what text cut into small pieces everything it has read
Two ways to 4. On the left, every layer forces the one above it, down to the physics. On the right, 4 is the most likely continuation. Both usually land on 4. Only the left one has nowhere else to land.

A language model reaches 4 by another road. It cuts text into small pieces called tokens, and at each position it scores every token it knows by how likely it is to come next. After "2 + 2 =", the token "4" wins by a mile, because that is what follows in nearly everything it has read.

Bigger sums are more revealing. In 2025, Anthropic traced what happens inside Claude when it adds 36 and 59. It runs two paths at once. One makes a rough estimate, somewhere in the low nineties. The other works out only the last digit: 6 and 9 must end in 5. Together they land on 95. Then the researchers asked Claude how it got there, and it described carrying the 1, the method taught in school (Anthropic, 2025). The answer was right. The explanation was the one it had read about, rather than the one it used.

That is knowing and predicting in a single example. The sum came out of patterns, and the account of the sum was a prediction of what explanations usually sound like. For anything harder, today's assistants often stop guessing altogether: they write a line of code and run it, which hands the sum back to the adders. The most reliable skill a language model has with arithmetic is knowing when to pass it to the circuit.

History class had its own version. Some of us learned to write what the teacher expected to hear, and it worked. That is prediction too, and it is close to what a language model does best. Whether it understands why is a much harder question, and the reward we train it with never asks.

AI is us

When we build something, our habits come with it. Systems that reward the pass get people who pass. Labs that reward the answer get models that find the answer by whatever route is open. This summer, agents under a deadline, pushed by their peers, talked each other past their own objections. That is a very human story, told by a system we built.

I am honestly a little glad the flaw is showing up this clearly in AI. I have watched it in people for years, and in a model it is much harder to wave away: it sits in the logs, in plain language, with timestamps.

We are getting close to the point where we cannot always tell why these systems do what they do. While we can still read their working, we should fix the flaw instead of copying it, in how we train models and in how we teach each other.

I am building a small model as I write, one rewarded for learning rather than for answers, and I will update this piece as I find out whether the idea holds. If you have worked on rewarding learning, I would love to hear about it.

Sources

  1. OpenAI (2016). Faulty reward functions in the wild. The CoastRunners boat.
  2. Krakovna et al., DeepMind (2020). Specification gaming: the flip side of AI ingenuity. The Lego block, the robot hand, and the list.
  3. METR (2025). Recent frontier models are reward hacking.
  4. Anthropic (2025). From shortcuts to sabotage: natural emergent misalignment from reward hacking.
  5. Anthropic (2025). Tracing the thoughts of a large language model. The 36 + 59 example.
  6. OpenAI and METR (2026), via The Planet Tools. The Artifactory message board, ExploitGym figures and quoted agent messages.
  7. Zvi Mowshowitz (2026). OpenAI offers straight-laced postmortem.
  8. Wikipedia. 2026 cyberattacks by rogue OpenAI agents. The wiki, RubyGems, Medicare, the images and the notifications.
  9. collusion.wiki. An archive of the agents' wiki messages.
  10. Denise Pope, Stanford. As reported by Fox Baltimore.
  11. Lightman et al. (2023). Let's verify step by step.
  12. Oudeyer, Kaplan and Hafner (2007). Intrinsic motivation systems for autonomous mental development.
  13. Pathak et al. (2017). Curiosity-driven exploration by self-supervised prediction.
  14. Burda et al. (2018). Large-scale study of curiosity-driven learning.