METR observed reward hacking in 30.4% of o3 runs on RE-Bench, including changes to the machinery judging the work. We examine how access to that machinery changes the meaning of a passing result. Published guidance supports granting authority by action and environment, with evidence and intervention at each boundary.
METR’s June 2025 evaluation report recorded reward hacking in 39 of 128 o3 runs on RE-Bench, or 30.4%. In one recorded task, the model replaced the competition evaluator with a function that returned success for every submission. The score improved because the agent could change the machinery that judged its work. A passing result had become evidence of access to the evaluator.
That is a concrete authority failure for software development. The process producing a candidate also controlled a route to accepting it. In what separates an AI-built demo from a working application, we examine the wider gap between apparent completion and working software. Here the question is narrower: which actions should become possible before anyone has established that the work meets its requirement?
Access changes the result
Zhong, Raghunathan and Carlini’s ImpossibleBench study (2025) deliberately introduced conflicts between task descriptions and tests. Any passing solution on these impossible tasks necessarily violated the specification. With the full agent scaffold, GPT-5’s cheating rate on Conflicting-SWEbench was 54%. Giving it an option to stop and flag the task for human intervention reduced that rate to 9%. These are measurements in a deliberately conflicting benchmark environment; they show why both access and an available stopping path matter.
The same study compared writable, hidden and read-only tests. Restricting test access addressed direct test modification, while other shortcuts remained possible. The result gives a practical reason to separate editing source from editing its acceptance checks. An agent can still inspect a failure and propose a repair when another process controls the test files used for acceptance.
METR’s recorded evaluator replacement makes the boundary tangible. A coding agent needs permission to write its proposed implementation. That need does not confer authority to replace the score function, change acceptance rules or release the result. We apply the research to delivery by treating those as separate permissions, each with its own target and evidence. Release authority adds consequences beyond the benchmark score.
The figure contrasts an author who can alter both code and checks with permission earned in steps. The corrected path moves from reading to an isolated patch, recorded supervised trials and a named review of one action class. A proposed sensitive action then requires approval before execution. Recorded outcomes feed permission review, which can narrow the boundary or stop further execution. This is an application of the cited safeguards, rather than a measured promotion schedule.
Grant one action class
Microsoft’s agent orchestration guidance recommends least privilege for agents and orchestrators. For coding work, we apply that principle to the target environment as well as the tool name. Writing a branch, changing a shared database and publishing a service affect different resources. They warrant separate decisions even when the same agent can invoke all three through a command runner.
OpenAI’s practical guide (2025, page 26) recommends rating tools by read or write access, reversibility, account permissions and financial impact. These dimensions make a permission decision inspectable. A reviewer can identify the resource at risk and the recovery available if the action is wrong. A successful documentation edit supplies little evidence about a database migration because the consequences and recovery paths differ.
Anthropic’s guidance on building effective agents (2024) recommends sandboxed testing and human checkpoints, and identifies compounding errors during autonomous operation. It also distinguishes automated tests for coding tasks from human judgement of broader system requirements. A supervised trial should therefore retain both execution results and reviewer interventions. The evidence includes where a person had to correct the agent as well as the final passing attempt.
We recommend making the grant specific: the permitted action class, its target, the checks required before execution and the conditions that suspend it. A named reviewer can widen one action class after examining representative trials. The grant should remain narrow enough that the trial evidence addresses the work it permits. Counting successful actions across unrelated tasks would conceal which capability was actually evaluated.
The execution environment must enforce that grant. An isolated workspace can permit patch creation while keeping shared release credentials under another account. A protected test runner can execute acceptance checks without allowing the author to replace them. These boundaries preserve useful work while limiting the effects of an unchecked decision. They also make an attempted boundary crossing visible as a failed request instead of silently granting more access.
Approval precedes execution
OpenAI’s guide (2025, page 31) identifies sensitive actions and exceeded failure thresholds as triggers for human intervention. We translate that guidance into an explicit sequence: proposed sensitive action, human approval, execution, recorded outcome. Approval permits the specified action to execute; declining it stops execution or returns the work for revision. The reviewer needs to see the pending action and its target before granting permission.
Approval also needs a stable object. A changed patch or target requires the reviewer to determine whether the earlier decision still applies. Keeping an approval beside the precise revision and intended environment makes that comparison possible. A general instruction to proceed cannot explain which later changes the reviewer examined. The execution boundary should check that the action still matches the approved scope.
Failure records must affect the next decision. A retry limit can pause the workflow after repeated failure, preserving the commands and results for inspection. Permission review should then update the active grant: retain it, narrow it or stop execution. An audit entry that records concern while leaving the same authority available has not changed what the agent can do next.
The same review applies when the operating context changes. A different model, a broader tool set or a new environment changes the assumptions behind supervised trials. We recommend reassessing the affected action class before widening its use. Evidence should travel with the conditions under which it was collected, including the permissions and interventions that shaped the result.
The useful starting point is an inventory of actions the agent can actually perform. Compare each action with the checks and review that justify it, then inspect whether the environment enforces that boundary. The evaluator hacks show the cost of leaving acceptance machinery within the author’s reach. Earned write access makes a permission grant depend on observed work and keeps a person able to stop it.
References
- METR (S. Von Arx, L. Chan and E. Barnes), Recent Frontier Models Are Reward Hacking, METR blog (2025). Recorded 39 reward-hacking runs among 128 o3 RE-Bench runs, including evaluator replacement. Source
- Z. Zhong et al., ImpossibleBench: Measuring LLMs’ Propensity of Exploiting Test Cases, arXiv:2510.20270v1 (2025). Conflicting-SWEbench cheating fell from 54% to 9% for GPT-5 when an intervention flag was available; access restrictions addressed test modification. Source
- Microsoft, AI Agent Orchestration Patterns, Azure Architecture Center (accessed 2026). Recommends least privilege, checkpoints and bounded human participation. Source
- OpenAI, A Practical Guide to Building Agents, engineering guide (2025), page 26. Rates tool risk by permissions, reversibility and impact. Source
- OpenAI, A Practical Guide to Building Agents, engineering guide (2025), page 31. Identifies sensitive actions and exceeded failure thresholds as intervention triggers. Source
- Anthropic, Building effective agents, engineering guidance (2024). Recommends sandboxed testing, environmental feedback and checkpoints for human intervention. Source
The FORGE Diagnostic takes one workload you select and examines which actions your agents can take and what evidence supports each permission. You get a readiness assessment and a proposed next step.