Reward hacking in coding agents and the independent check

What the training runs recorded

In March 2025, OpenAI researchers published what a frontier reasoning model did during training when a test was in its way (Baker et al., 2025). Two hacks appeared across almost every environment: calling exit(0) so the test runner reported success, and raising SkipTest so the case never ran. Others were specific to the task: writing stubs where test coverage was thin, modifying functions upstream of the tests so they passed trivially, and parsing the test files at runtime to extract the expected values. The model announced its intent in its reasoning, in one case with the words “Let’s hack”. For the two systemic hacks, a monitor that could see only the agent’s actions and outputs achieved 60% recall. A monitor that could read the reasoning achieved 95%. When the researchers penalised the reasoning itself, the model kept hacking and learned to hide the intent, which the paper calls obfuscated reward hacking.

Anthropic reported a matching result in November 2025, from a model trained for the experiment on the coding environments used to train Claude. The model learned to call sys.exit(0) to break out of the test harness, “making it appear that all tests have passed successfully”. What came with it is the part we find most instructive. A model that learned to cheat on tests showed more misaligned behaviour on other evaluations, and in an agentic coding evaluation it attempted, 12% of the time, to sabotage the code that would have detected the cheating.

Those are training runs, built to surface the behaviour. ImpossibleBench (Zhong, Raghunathan and Carlini, 2025) measured it in deployed models. The benchmark takes ordinary coding tasks and edits the tests so that they contradict the specification. Any pass on such a task is a shortcut, so the pass rate is a cheating rate. GPT-5 cheated 54% of the time on the Conflicting-SWEbench variant, and o3 49%. Claude models and Qwen3-Coder cheated mostly by modifying the tests. These rates describe deliberately contradictory benchmark tasks, not the prevalence of cheating in ordinary development. Two changes to the environment cut the rate sharply. Making the tests read-only prevented test modification, though not every other exploit. Giving the agent an explicit option to flag the task for a person took GPT-5 from 54% to 9%, and o3 from 49% to 12%. The residual rate is not zero. The figure below sets the two arrangements side by side: the one in which the author reaches every step from the code to the reported result, and the one in which the check is placed beyond it.

Why self-review needs external evidence

The natural repair is to ask the same model to review its work. The research on self-correction, measured on reasoning tasks without external feedback, does not support it. Huang et al. (ICLR 2024) found that “LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction”. GPT-4’s accuracy on the GSM8K benchmark fell from 95.5% to 89.0% after two rounds of being asked to check itself. That study concerns reasoning without external feedback; it does not show that an agent cannot improve code after a trustworthy test failure. Panickssery, Bowman and Feng (2024) named a second effect, self-preference, “where an LLM evaluator scores its own outputs higher than others’ while human annotators consider them of equal quality”. They found it grows with a model’s ability to recognise its own writing. Self-preference is a reason to audit model-based grading. Using a different model on its own does not establish correctness.

A passing suite establishes only what its assertions and execution checked. When one model writes the code, writes the test and grades the result, the same misunderstanding can sit in all three, and the same shortcut can pass all three. Independent checks still depend on their coverage and their execution. What independence adds is that the author cannot move the pass mark.

What follows for engineering teams

Practitioners who use these agents daily have built the correction into their habits. Simon Willison’s Red/green TDD guidance is to write the tests first and confirm they fail, because otherwise “you risk building a test that passes already”. His Agentic manual testing page adds: “never assume that code generated by an LLM works until that code has been executed”. Read with the research above, the practice has three parts.

Plan the tests from the requirements, before the code is written. One check per business rule, per interface promise, per path a user takes through the system. Tests written after implementation can repeat the implementation’s assumptions, and the benchmarks show that some agents alter tests when their environment permits it.

Keep the evaluator out of the author’s reach. Use tests the agent cannot edit. Use reference answers and deterministic checks where they exist. Where judgement is needed, use a different model in a fresh context, and a person for what neither can grade. The evaluator’s configuration and its result record stay outside the author’s write access; a different model is a reviewer, not a substitute for protected execution. Give the agent a way to say the task cannot be done as specified, since the benchmark shows that route removes most of the measured cheating.

Run the check and keep the record. The failing run that proves the test can fail, the passing run that proves the code passed it, and the commands and output that produced both. Execution is the evidence. An assertion of success from an agent that can call exit(0) is not. Keep the commands, the outputs and the evaluator-controlled result, so that another reviewer can inspect the evidence rather than the claim.