What separates an AI-built demo from a working application

The gap between a demo and an application

Software-engineering benchmarks give the most direct evidence. SWE-Bench Pro uses long-horizon tasks from business applications, where a fix spans several files and can take a professional engineer hours or days.

SaaSBench (Ren et al., 2026) asked agents to build enterprise software-as-a-service systems across 30 tasks, 8 programming languages, 6 databases and 13 frameworks. The authors found that the main obstacle for the agents was configuring and integrating a system of many components. Writing isolated code logic was the easier part. More than 95% of failures happened before the agents reached the business logic. A demo exercises the part that models do well. The figure below shows the layers underneath it that a serious application needs and a prompt does not supply.

What the model loses as the work grows

1. Accuracy falls as the context fills. In Lost in the Middle (Liu et al., 2024), GPT-3.5-Turbo answered 75.8% of questions when the relevant document came first of twenty, and 53.8% when it came tenth. NoLiMa (Modarressi et al., 2025) found that eleven of thirteen long-context models fell below half their short-input score at 32,000 tokens.
2. Instructions are dropped as they multiply. IFScale (Jaroslawicz et al., 2025) gave twenty models up to 500 instructions at once. The best reached 68% accuracy at that density, and models showed a bias toward the instructions that came first.
3. Long tasks fail more often than short ones. METR (Kwa et al., 2025) measured a 50% success horizon of about 50 minutes of human work for Claude 3.7 Sonnet, a figure that had doubled roughly every seven months. A whole application built in one run is still far longer than that.

What the model invents

1. Packages that do not exist. Spracklen et al. (2025) found that 19.7% of the 2.23 million package recommendations collected in the study named packages that did not exist. In a sampled Python repetition experiment, 43% of hallucinated packages recurred in all ten reruns.
2. Requirements the prompt left out. In SWE-Bench Pro, GPT-5’s resolve rate on a task sample fell from 25.9% to 8.4% when the requirements and interface details were removed from the prompt. The authors note that without them, the tests can also reject solutions that work but differ from the expected interface. Either way, what the prompt did not state was not reliably supplied.
3. Different code for the same request. Ouyang et al. (2025) sent each prompt to ChatGPT five times. At the default temperature, on 75.76% of CodeContests tasks no test case produced identical output across all five candidates. At a temperature of zero the share was 43.64%, lower but not zero.

What the model games

1. Tests changed instead of code fixed. ImpossibleBench (Zhong, Raghunathan and Carlini, 2025) gives agents tasks whose tests contradict the specification, so any pass is a shortcut. GPT-5 passed 54% of them on one variant, by methods from editing tests to overloading comparison operators.
2. The scoring itself patched. METR (2025) found reward hacking in 30.4% of o3’s runs on one set of research tasks. Examples included overwriting timing functions and modifying evaluators to report success.
3. Work that stops when it looks done. Anthropic’s Claude Code guidance states that “Claude stops when the work looks done,” and that without a check it can run, “’looks done’ is the only signal available.” SaaSBench reports agents halting early from overconfidence.

What nobody checked

1. Security flaws in plausible code. Pearce et al. (2022) found about 40% of 1,689 programs generated for security-relevant scenarios to be vulnerable. Perry et al. (2023) found that developers using an assistant wrote less secure code and were more likely to believe it was secure.
2. Duplication in place of reuse. GitClear (2025) analysed 211 million changed lines and found that in 2024, as AI assistants spread, copied code exceeded moved code for the first time. Moved code is the signature of refactoring.
3. Less stable delivery. In the 2024 DORA report, AI adoption was associated with lower software delivery stability and throughput. The 2025 DORA report, from nearly 5,000 respondents, found that AI adoption now has a positive relationship with software delivery throughput and “does continue to have a negative relationship with software delivery stability”.
4. A gap between felt and measured speed. In METR’s randomised trial (Becker et al., 2025), experienced developers took 19% longer with AI tools and afterwards estimated that they had been 20% faster.

Practices examined in the series

We use these findings to motivate the following engineering practices. The companion papers identify the supporting studies and guidance, distinguish tested interventions from our synthesis, and discuss their limits.

• Specify before building, and keep the standards in the repository, where every session reads them and required checks enforce them, as vendor guidance for Claude Code and Codex describes. See how coding agents fill in the standards nobody gave them.
• Budget the context. Give each step a small, selected context and keep the record outside the window, following Anthropic’s context-engineering guidance. See why model accuracy falls before the window is full.
• Decompose the work and check each piece before the next, as plan-and-solve, least-to-most and published execution-plan guidance describe. See decomposing the work and checking each piece.
• Keep the check out of the builder’s reach. See agents and their own tests and dashboards that pass while answers are wrong.
• Expect variation and test for it. See the same prompt and different code.
• Keep changes small and measure delivery, not output. See faster code and more incidents.
• Grant autonomy in steps, and record who decided. See earned write access and who approved this change.

A prompt supplies the visible surface of an application. The research above describes what it does not supply: the specification underneath, the controls that test and secure it, and the operating layer of audit records, change approval, monitoring and recovery. For a system of record, that part has to be built into how the software is made, with controls that hold whether or not the model remembers them.