Silent failures and the limits of AI incident diagnosis

How agents score on diagnosis

IBM’s ITBench, published at ICML 2025, gave agents built on the strongest available models realistic site reliability scenarios. They resolved 11.4% of them (Jha et al., 2025). Artificial Analysis and IBM followed with ITBench-AA in May 2026: 59 Kubernetes incidents, each a snapshot of alerts, events, traces, metrics and logs. The score is average precision at full recall. A diagnosis that misses any root cause scores zero, and one that finds them all is marked down for every false positive it adds. This is a different measure from ITBench’s resolution rate, and neither is a probability that a given diagnosis is wrong. At launch, “all frontier models score below 50%”, the best at 47%. As of 23 September 2026 the leaderboard’s best was 56.2%, for GPT-5.6 Sol (max). In the benchmark, some longer investigations added false positives; models “that over-investigate tend to surface upstream fault-injection mechanisms or co-occurring symptoms as false positives”.

ClickHouse’s engineers ran a smaller trial in August 2025 (Palacin and Brown, 2025). They gave five models telemetry from a demonstration application with four injected faults, which they describe as relatively easy to detect. None of the models consistently identified the root cause without guidance, two of the four faults needed a hint, and a single investigation used between 41,823 and 1,969,111 tokens. Their conclusion was that “autonomous RCA is not there yet”. Their recommendation was to use the models to summarise, suggest and document while engineers stay in control.

An agent investigating an unfamiliar system depends on the evidence its tools can retrieve. Telemetry may omit intended behaviour, dependencies and known failure modes. Documenting that knowledge can reduce what an operator must reconstruct. The figure below shows the two states a service can be in while every infrastructure check is green, with the three study results beside them. Its third band shows where an alarm derived from the promise sends a detected failure. The three study results are independent observations about detection, resolution and diagnosis.

The failures nobody is paged for

Wu’s case study covers a single personal-assistant agent runtime over eight weeks (Wu, 2026). The system had 4,286 unit tests and 827 governance checks and recorded 22 incidents. About 70% of the documented silent failures were detected through human observation rather than by any test or audit. The longest-lived failures, from thirteen hours to sixty days, sat in the seams between components, where no test runs. In the worst class, the model turned the error into fluent, plausible narrative delivered to the user. The author calls this fail-plausible: the observer “is not just blind, it is convincingly lied to by the failure itself”. One system does not establish how common this is elsewhere. It establishes that the class exists and what it looks like.

An infrastructure check cannot see this class. The process is running, the request returned a success code, the latency is normal. What is wrong is the content of the answer, and only a check that knows what the answer should have been can tell.

Where the alarms come from

Google’s SRE practice described the discipline in 2016. The Site Reliability Engineering book gives monitoring two questions to answer, “what’s broken, and why?”, and sets rules for interrupting a person: “every page should be actionable”, a page “should require intelligence”, and it should concern “a novel problem or an event that hasn’t been seen before”. Pages that fire too often teach people to “second-guess, skim, or even ignore incoming alerts”. Teams should prepare this before deployment and update it as incidents reveal new failure modes. The alarms follow from what the service promised, and an automated response follows from what has been rehearsed. An InfoQ analysis from August 2026 states the human side plainly: AI “has commoditized code generation, removing the comprehension that formed for free during implementation”, and “a system that is not understood cannot evolve safely”.

What follows for engineering teams

We draw the following recommendations from the benchmarks, case study and SRE guidance.

What production watches is derived from what the system promised. The availability targets, the user paths that must work, the dependencies whose failure must be survived. Checks derived from those can reveal failures that infrastructure metrics miss. For example, a synthetic order can check that an accepted request appears in the downstream order system. A healthy dashboard then means the implemented checks passed; untested behaviours can still fail. A service returning success codes while giving wrong answers passes the infrastructure checks. Whether it is keeping its promises is the check that has to be designed.

The operating knowledge is produced with the system. The runbook, the dependency map, the known failure modes and the rehearsed responses are written from the design, at build time, rather than reconstructed by whoever is paged. An operator that already knows the system, human or automated, is the difference between a diagnosis and a search.

Automation acts only where it has been rehearsed. The fixes an automated operator runs alone are the ones it has performed in rehearsal and had checked. A fix that does not clear the alarm hands the incident to a person, with the evidence assembled. The novel case, where the best model scored 47% at launch on ITBench-AA, goes to a person rather than to a longer investigation, and the escalation depends on the incident’s risk and the procedure’s validated scope.

An application that arrives with its operating knowledge can be run by people who did not build it, and increasingly by automation that did not either. Preparing that knowledge during development reduces the amount an unfamiliar operator has to reconstruct during an incident.