We look at why long agent runs go wrong and what the research says about preventing it. Agent success falls as tasks get longer, and a mistake made early is carried into everything built on top of it. Studies of how models solve hard problems, and of how multi-agent systems fail, point to the same remedy that vendor guidance now prescribes. Cut the task into small pieces, give each one an observable outcome, and check it before the next piece begins.
METR measured how long a task, timed by skilled humans, frontier models could complete with a 50% success rate. In its March 2025 study, that horizon was about 50 minutes for Claude 3.7 Sonnet. It had doubled roughly every seven months since 2019. Longer human task duration went with lower agent success, and a whole feature handed over in one prompt is a long task.
The builder’s version of this result is familiar. The agent writes across many files and reports that it is finished. Review then finds that one requirement was skipped, another was half built, and an early wrong assumption runs through the rest. This is one of the failure modes set out in what separates an AI-built demo from a working application.
In this post we summarise the research on breaking problems into smaller ones, the evidence on where agent systems fail, and the published guidance on running the pieces.
Decomposition in the research
Wang and colleagues, at ACL 2023, analysed where a model’s step-by-step reasoning goes wrong on maths word problems. They found three kinds of error: calculation errors, missing-step errors and semantic misunderstanding. Their method, plan-and-solve, first has the model devise “a plan to divide the entire task into smaller subtasks,” then carry out the subtasks in order. On a sample of 100 problems, the improved variant, PS+, cut missing-step errors from 12% to 7% against plain step-by-step prompting.
Zhou and colleagues, at ICLR 2023, went further with least-to-most prompting: “break down a complex problem into a series of simpler subproblems and then solve them in sequence,” each one using the answers already found. On the length split of SCAN, a benchmark of compositional instructions, code-davinci-002 scored 16.2% with chain-of-thought prompting and 99.7% with least-to-most prompting. That is evidence about following compositional instructions, and it shows how much depends on the order in which the parts are solved.
The same papers mark two limits that matter for software. In the plan-and-solve analysis, semantic misunderstanding stayed at 26 to 27% whether or not the model planned first. A model can produce a coherent plan for a task it has misunderstood. The least-to-most authors found that a decomposition written for one domain did not carry over to another. They also observed that “nearly all problems in GSM8K can be accurately solved” when the model is given “the correct decomposition.” The decomposition carries most of the difficulty. Cutting the work up is a design task, and it needs someone who knows the domain to check it. The figure below shows what the placement of the check changes.
Where agent runs fail
Cemri and colleagues (2025) annotated more than 1,600 execution traces from seven multi-agent frameworks. They sorted the failures into fourteen modes in three categories: system design issues, misalignment between agents, and task verification. System design issues, which include disobeying the task specification, formed the largest category. Verification failures, where a step’s output was not checked or was checked wrongly, formed a category of their own.
The same study tested a fix. In a 32-task case study, the authors changed a software-building framework so that it repeated its reviews and ended only when a supervising agent confirmed that every review was satisfied. Task success rose from 25.0% to 40.6%, an increase of 15.6 percentage points.
A single agent shows the same pattern at a smaller scale. Anthropic’s guidance for Claude Code puts it plainly: “Claude stops when the work looks done. Without a check it can run, ‘looks done’ is the only signal available.” Its advice is to give the agent something that returns a pass or a fail, such as a test suite, a build or a comparison against a fixture, so that the loop closes on evidence.
Running the pieces
Vendor guidance for long agent work now describes the same structure. OpenAI’s cookbook on execution plans for multi-hour Codex work asks the plan to “anchor the plan with observable outcomes,” and to “prefer additive, testable changes that can be validated as you go.” It states that “validation is not optional.” Each milestone describes what will exist at its end, the commands to run and the result to expect. For an internal change, it suggests demonstrating the effect with tests that fail before the change and pass after it.
Microsoft’s architecture guidance on agent orchestration gives the reason to check between steps instead of only at the end. It warns that “low-confidence, malformed, or off-topic responses can cascade through a pipeline.” A check at each boundary can stop a detected error before later steps depend on it. None of these sources sets a universal size for a piece; that is a judgement made for each task.
What this means for teams building with agents
Taken together, the research and the guidance describe a working method:
- Settle the requirement before anything is cut. The misread requirement was the error that planning did not reduce. Confirm it with the person who owns it, and write it down, before the first piece is defined.
- Cut the work into pieces with an observable outcome each. A piece should be small enough to keep the agent’s context focused on it. Its acceptance should be stated in advance, as a test that must pass or a behaviour a person can see.
- Check each piece before the next begins. Run the check and keep the output. A piece that fails is fixed and rechecked where it failed, not discovered three pieces later.
- Keep the plan and the record together. Record what was cut, what passed and what was set aside in a document the next step reads. A fresh context or a different person can then continue the work without reconstructing it.
Running without intermediate checks is a bet that nothing went wrong early. The failure research shows that system design and verification account for a large share of what does go wrong. The cost of either is set by how much was built on top before anyone looked.
References
- T. Kwa, B. West, J. Becker et al., Measuring AI Ability to Complete Long Tasks, METR (2025). 50% time horizon of about 50 minutes for Claude 3.7 Sonnet; doubling roughly every seven months since 2019. Source
- L. Wang et al., Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models, ACL (2023). Three error types; missing-step errors 12% to 7% for PS+ on 100 GSM8K problems; semantic misunderstanding 26 to 27% with or without a plan. Source
- D. Zhou et al., Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, ICLR (2023). SCAN length split with code-davinci-002: 16.2% chain-of-thought, 99.7% least-to-most; decompositions do not transfer across domains. Source
- M. Cemri et al., Why Do Multi-Agent LLM Systems Fail?, arXiv 2503.13657 (2025). 1,600+ traces, 7 frameworks, 14 failure modes in 3 categories; on 32 ProgramDev-v0 tasks a revised ChatDev topology raised success from 25.0% to 40.6%. Source
- Anthropic, Best practices for Claude Code, Claude Code documentation (2026). The agent stops when the work looks done; give it a check it can run. Source
- OpenAI, Using PLANS.md for multi-hour problem solving, OpenAI Cookbook (2026). Observable outcomes, additive testable changes, validation not optional. Source
- Microsoft, AI agent orchestration patterns, Azure Architecture Center (2026). Low-confidence or malformed output can cascade through a pipeline. Source
The FORGE Diagnostic takes one workload you select and shows how it can be cut into pieces with an observable outcome each, checked before the next begins. You get a readiness assessment and a proposed next step.