Evaluating variation in generated code

What repeated runs reveal

Atil and colleagues (2025) tested five models on eight reasoning and knowledge tasks, with ten runs per condition at temperature zero. In their few-shot college-math results, GPT-4o gave the same extracted answer across all ten runs for 50.0% of questions. That is answer agreement, rather than a coding success rate. Their distinction between full response agreement and extracted answer agreement helps explain why visually different text and different behaviour need separate measures.

For code, two implementations can satisfy the same requirement while using different names, control flow or libraries. Conversely, nearly identical implementations can share the same defect. Text comparison is useful for examining what changed, but executable acceptance checks determine whether the required behaviour holds. The requirement and the check need to be defined before a reviewer chooses the most appealing candidate.

We separate the process into candidate generation, deterministic gates and retention of the accepted source. Every run creates a candidate whose results concern that candidate alone. A failed check sends a revision through the same checks again. A passing check supports the selected source only for the behaviour exercised, so coverage remains a separate review decision.

The figure shows one recorded request producing three candidates with separate check results. Its branches are illustrative outcomes, not the studies’ measured frequencies. Revision returns to the candidate path; it has no direct route to acceptance. Of the two passing candidates, one is selected. The selected source and its passing results are retained together, then the build uses that fixed source. Regeneration from the prompt creates a new candidate and returns to checking.

Evaluate the selection procedure

Anthropic’s evaluation guidance (2026) distinguishes pass@k, the probability of at least one success in k attempts, from pass^k, the probability that all k attempts succeed. These quantities answer different operational questions. A developer choosing a working candidate from several attempts needs to know the cost and reliability of that selection. An unattended operation needs evidence about the next attempt it will consume.

The arithmetic makes the distinction concrete. Assume independent trials, each with a constant 90% chance of passing. Across three trials, the probability of at least one pass is 1 minus 0.1 cubed, or 99.9%. The probability that all three pass is 0.9 cubed, or 72.9%. This is an illustrative calculation under stated assumptions; correlated failures in real tasks require measurement of the actual procedure.

Selection is part of that procedure. If a reviewer selects among several outputs during evaluation, unattended deployment must reproduce that selection step or be evaluated separately. Removing the reviewer changes how the accepted candidate is found. Reporting only the best attempt leaves the reader unable to tell how much checking, repair or rejection was required to obtain it.

Anthropic recommends deterministic graders where possible and stable evaluation environments. For coding work, executable checks give every candidate the same acceptance rule. A controlled environment helps separate model variation from flaky tests, changing dependencies or unavailable services. A passing result then refers to a known source revision under recorded conditions, making comparisons between attempts more useful.

We recommend retaining the failed attempts as well as the selected one when evaluating a workflow. Record which check rejected each candidate and whether a person intervened before the next run. The resulting record describes the work required to obtain an acceptable output. It also reveals whether repeated attempts fail on the same requirement, which a headline eventual-success score can hide.

Evaluation should include the task distribution and operating conditions intended for use. A change to the model, available context or candidate-selection rule changes that procedure. Repeating representative tasks under the new conditions allows a meaningful comparison with the earlier evaluation. The decision to use the procedure should refer to those observed outcomes and the costs of checking them.

Preserve what passed

After selection, source control provides a fixed input for the build. Build reproducibility concerns that input, dependencies and the build environment. It does not require another model response. Regenerating an implementation from the original request introduces a fresh candidate and therefore a fresh need for acceptance evidence. Even an unchanged prompt cannot carry yesterday’s passing results into today’s generated code.

OpenSSF’s SLSA v1.0 (2023) describes build provenance that records the build platform, process and top-level inputs. We apply that traceability principle to the earlier generation step by retaining the model identity, settings, instructions and available context. These records explain the conditions of generation. The exact selected source remains the object that downstream checks and builds must identify.

The check record and the source are linked references. The record identifies the source revision and the observed results; it does not create or repair that source. The build takes the retained revision as input and produces an artifact whose provenance points back to it. Together, these links allow a reviewer to follow the delivered artifact to the candidate that passed.

A later edit needs the same care. Its earlier results remain useful history, but a reviewer must determine which checks cover the changed behaviour. Running the affected checks against the new revision creates evidence for that revision. Keeping the command, environment and output avoids confusing an earlier success with a result produced after the edit.

For teams building beyond a demo, variation is a reason to evaluate the whole generation and selection procedure. Once a candidate has passed, retaining it removes regeneration from the delivery path. The build should consume the code that earned acceptance. Another response to the same prompt starts another evaluation, however familiar the resulting interface looks.