We review three studies that measure how well language models use long inputs. Each finds that accuracy falls as the input grows, well before the advertised context limit. Information in the middle of a long input is used least reliably. Teams that build agents have responded by treating context as a budget: each step gets a small, selected context, and the standards and the running record stay outside the window.
When a model was given twenty documents and a question, it answered 75.8% correctly if the answer sat in the first document and 53.8% if it sat in the tenth (Liu et al., 2024). With no documents at all it scored 56.1%. Builders who use a coding agent on a growing project report the working-software version of this. Late in a long session, with the specification, the architecture notes and hours of conversation in its context, the agent skips requirements it followed earlier. The advertised window is far from full.
The usual response is to give the model more: the whole specification, the full coding standard, every relevant file. The research on long inputs suggests that this adds to the problem. How reliably a model uses a fact depends on how much else is in its context and where the fact sits. This is one of the failure modes set out in what separates an AI-built demo from a working application.
In this post we summarise three measurements of that effect and the guidance that vendors and practitioners have published in response.
What the measurements show
Liu and colleagues, in Lost in the Middle, gave models a question and twenty retrieved documents, one of which held the answer. They then moved that document through the input while keeping the input the same size. GPT-3.5-Turbo’s accuracy of 75.8% with the answer first fell to 53.8% with it tenth, below its 56.1% with no documents. The authors conclude that “current language models do not robustly make use of information in long input contexts.” The extended-context versions of the models they tested did no better.
NoLiMa, presented at ICML 2025, tested thirteen models that each claim a context of at least 128,000 tokens. Its questions require the model to infer the answer, not to match the question’s words in the text. At 32,000 tokens, eleven of the thirteen scored below half of their own short-input accuracy. GPT-4o went from 99.3% to 69.7%.
Chroma’s Context Rot report (Hong, Troynikov and Huber, 2025) tested eighteen models from four providers. It found that performance “grows increasingly unreliable as input length grows,” even on simple tasks. A single passage that resembled the answer without being it lowered accuracy, and four such passages lowered it further. On a conversational memory benchmark, focused prompts of about 300 tokens outperformed the full history of about 113,000 tokens for every model family tested. The focused prompts were built from the benchmark’s labelled relevant passages with manual adjustment, so the result shows the value of the right selection more than of shortness alone. Behaviour under uncertainty also differed: the Claude models tended to abstain, while the GPT models more often gave a confident wrong answer.
These are question-answering measurements, not coding benchmarks. For software work, an omitted requirement may produce code that ignores it without an explicit warning. The figure below contrasts the two ways of spending a context window.
Context as a budget
Teams that build agents have changed how they work in response. Anthropic’s guidance on context engineering (2025) describes context as “a finite resource with diminishing marginal returns.” The goal it sets is “the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome.” Its guidance for Claude Code opens with the same constraint: the context window “fills up fast, and performance degrades as it fills,” and it is “the most important resource to manage.”
The recommended techniques follow from that. With just-in-time retrieval, the agent holds lightweight references and loads the detail when a step needs it. With progressive disclosure, it finds context by exploring instead of receiving everything at once. Compaction condenses a long session into a summary of the decisions and open issues. Structured note-taking writes progress to files outside the window, where a later step can read it. Sub-agents take a focused task in a separate context and return a short result.
LangChain’s engineering team groups the same practice into four moves: write context out of the window, select only what the current step needs, compress what accumulates, and isolate work into separate contexts. It borrows Drew Breunig’s names for the ways a context goes wrong. Poisoning is a hallucination that enters the context and is then relied on. Distraction is a context so long that it overwhelms what the model learned in training. Confusion is irrelevant material that influences the answer, and clash is two parts of the context that disagree.
What this means for teams building with agents
The studies and the guidance point to a way of working that differs from filling the window and hoping:
- Cut the work into pieces that fit. A focused context can reduce irrelevant material; measure its effect on the team’s own tasks. The benchmarks put the reliable range far below the advertised limit.
- Select the context for each step. At each step, select the relevant rules, interfaces and evidence, then record progress for the next step. Shorter inputs can still omit a critical requirement, so test whether the selection holds what the step needs.
- Keep the standards and the design outside the model’s memory. They live in the repository, versioned and checked there, and are read into a step as needed. A rule the model must remember across a long session is a rule the studies suggest it may lose.
- Write the record outside the window. Progress, decisions and open questions go into notes the next step reads. External notes reduce the dependence on one session, provided later steps retrieve and check them.
- Check the result, not the session. A final check that runs a named test decides whether the change is accepted, and a failure goes back to the step that produced it.
Select context for each step and test whether the selection preserves the information needed to complete it. Keep progress outside the window so later steps can retrieve and check it.
References
- N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni and P. Liang, Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics (2024). GPT-3.5-Turbo with 20 documents: 75.8% with the answer first, 53.8% with it tenth, 56.1% closed-book. Source
- A. Modarressi et al., NoLiMa: Long-Context Evaluation Beyond Literal Matching, ICML (2025). 13 models claiming at least 128K tokens; 11 below half their short-input score at 32K; GPT-4o 99.3% to 69.7%. Source
- K. Hong, A. Troynikov and J. Huber, Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma (2025). 18 models; degradation with length and distractors; focused prompts of about 300 tokens, built from labelled passages, beat histories of about 113,000 tokens; abstention differs by model family. Source
- Anthropic, Effective context engineering for AI agents, Anthropic Engineering (2025). Context as a finite resource; just-in-time retrieval, progressive disclosure, compaction, structured note-taking, sub-agents. Source
- Anthropic, Best practices for Claude Code, Claude Code documentation (2026). The window fills fast and performance degrades as it fills. Source
- L. Martin, Context Engineering for Agents, LangChain Blog (2025). Write, select, compress, isolate; Breunig's poisoning, distraction, confusion and clash. Source
The FORGE Diagnostic takes one workload you select and shows how the work can be cut to fit the window, with the standards read from the repository for each step. You get a readiness assessment and a proposed next step.