Spec drift is the normal state

Spec-driven development currently counts as the serious way to build software with AI agents. The specification is written before the code and remains the binding directive throughout the work. I held to this way of working in two internal projects over months, more strictly than the common tools require. Together they amount to around 850,000 lines of product code, more than 5,000 ticket and specification files, and more than 7,000 commits. The result is sobering. In neither project did the specification reliably determine the outcome. Correctness came in the end from end-to-end tests and independent audits, checks that only take hold after the implementation and are expensive to run.

The promise

When agents write the code, the human work moves forward, into the precise description of what is supposed to come into being. The specification becomes the actual source of the system; the code counts as a derived artifact that can be regenerated from it at any time. Tools such as GitHub’s Spec Kit or Amazon’s Kiro institutionalize this sequence: requirements are captured, a plan is derived from them, the agent implements, and at the end the result is validated against the specification. Thoughtworks lists the technique in its Technology Radar under “Assess” (as of November 2025). The idea is older than the agents; requirements documents have existed for as long as there have been software projects. What is new is the expectation that the chain from requirement to implementation holds without human translation, because an agent takes it over.

Behind this stands a promise of control. Whoever specifies precisely enough, so the assumption goes, controls the outcome, even without reading the emerging code line by line. The specification is meant to replace the review that secures correctness in human teams. In its strong form this means that a sufficiently precise specification yields a conforming implementation.

Both projects discussed here were set up as specification-driven from the beginning. The very first commit of project B contains nothing but specification files. In project A, the ticket, epic and ADR system likewise came into being before the first feature.

The setup

I developed both projects with coding agents. The agents implemented and reviewed; the documentation, too, ran through them.

Specification discipline was high in both projects. In project B, every larger ticket follows a schema with 23 mandatory sections, from fixing the scope through acceptance criteria to the verification and negative-control plan; a single specification package of 122 Markdown files contains 741 tables with close to 6,000 data rows. In project A, a central working constitution governs the ranking of sources, the separation of roles between implementer and auditor, and the conditions under which a result counts as accepted. A passed plan review comes before any implementation start, an independent audit before any merge; self-acceptance is ruled out.

What stands out is the size ratio between prose and code. In project A, the entire Markdown corpus of tickets, plans, audits and evidence documents comes to around 868,000 lines. That is almost twice the product code. In project B, around 244,000 lines of specification sit next to around 550,000 lines of code. Alongside the software, a second system made of prose has come into being, similar in size and with a maintenance need of its own.

The finding

By spec drift I mean specification and reality diverging: the code does something other than what the specification describes, or the specification describes something that never existed in that form or no longer does. In both projects this state was the rule over long stretches. Across the entire runtime I did not come across a longer strand of work that ran through without documented drift. The audit artifacts of both projects show this in four directions.

The implementation deviates from the specification. This is the case one expects, and it occurs in forms that no diff review catches. In project B the specification required deterministic generation of content; a repair plan later counted 16 places in the code that read the system clock anyway. The effect stayed invisible for months because an earlier reference export had frozen the same generation date: the tests compared against reference data that contained the same error. An audit in project A documents a case in which the system prompt required two shell commands before any assumption about a path, while the rule set blocked one of those commands. 48 of 51 observed command sequences followed the instruction and ran into the block; 21 test runs consumed their full iteration budget without effect, and all processes nevertheless exited with code 0. Another audit showed that a normalization function truncated multi-digit error counts to zero, which allowed a check module to be set to “passed” despite explicit evidence of errors. A documented configuration switch for the token limit also had no effect; four measurement runs with four different values produced exactly the same output length every time, without an error message.

The documentation ages against the code. An automated comparison in project A found references to source files that had long been deleted or renamed, and a contradiction register counted eleven different versions of the same test count across the documents. Each of those numbers had been correct at its time and then froze. Status drift is more serious. An independent audit across 16 work packages concluded that six of them carried status information that the reality of the main branch had overtaken. Work already merged appeared open. Agents read these artifacts as a source of control; every later gate would therefore have checked against false status. The same class includes a finding in which four core levers of the implementation existed only on a side branch and were preserved solely through an archive tag, while planning listed them as available; in another case a passed plan gate lay only on the working branch and was missing on the main branch, against which it should have applied.

Specifications contradict each other. In project A the acceptance criteria of two tickets collided head-on. Both concerned the same function; one required a tightening of the existing patterns, the other a deliberate loosening. Another ticket declared itself a precondition of one that knew nothing of this dependency. These specifications were written at different times by different agent contexts, none of which surveyed the whole body.

The specification contradicts itself. This is the direction that surprised me most. A plan audit in project A showed that three provisions of an already detailed plan produced no callable code path between them; the plan was unfulfillable in itself before a line was implemented. In project B an audit found that nine mandatory sections of the parent specification had disappeared from the tickets through a compaction pass, and nobody had noticed until a reviewer counted them. In the same project there is an audit document that turns its own earlier “passed” into “not passed” by addendum, because the chain of evidence had been tied to the wrong objects under review.

In my assessment the cause lies in the working model itself. Agents work with a limited context window in sessions that end; in project B, subagents ran into session limits after around two hours. A project of several months therefore consists of hundreds of handovers between contexts, each of which sees only a section of the system and of the specification. At every handover, specification and reality can drift apart a little. One agent changes code and misses some of the affected documents; the next reads one of the stale ones and builds on it. Resumption prompts and handover documents soften this, but they do not replace the missing overall view. The specification ages just as the code does. It sits in the system as a maintained artifact, and unlike the code it has no tests. Specifying more precisely therefore mainly enlarges the surface on which this ageing takes place; the 23 mandatory sections in project B did not prevent drift in any case.

I am not alone in this observation. A field study of 20,574 real agent sessions found systematic gaps between assignment and execution; 91 percent of the visible cases required an explicit user correction (arXiv 2605.29442, 2026). Even the best agent in a benchmark on specification competence identifies only 44 percent of the defects in specs, that is, gaps, ambiguities and contradictions (arXiv 2605.30314, 2026). A comparison of three spec-driven frameworks across two model families concludes that stricter binding to the specification even lowers the determinism of the output; verifiability remains the attainable goal (arXiv 2606.30689, 2026). The mechanism of context loss is described as well, down to the detail that context compression preferentially discards precisely those tokens that encode the original specification (arXiv 2607.27167, 2026). From industry, Atlassian’s engineering blog reports the same: with a full context window, agents silently abandon earlier decisions, and larger models do not solve the problem (Atlassian Engineering, June 2026).

What helps in practice

Three countermeasures emerged in both projects and have proven themselves in daily use. All three move reliability to places that can be executed or checked independently.

When a model picks a different solution than the specified one, the reflex is to roll the deviation back. That reflex is expensive and often unfounded in substance, because many deviations are technically equivalent or better. It is more workable to accept such deviations and to check whether the system behaves the way it should; whether the code follows the wording of the specification becomes secondary. The instrument for that is end-to-end tests against the real execution path. Project B holds close to 600 browser end-to-end cases next to a good 6,000 unit cases; for every new functional unit an end-to-end proof of its own is mandatory. If a deviation is accepted, the specification has to be updated as well, otherwise the same deviation shows up as a finding again in the next audit. In both projects, end-to-end tests have been the only instance that checks semantics. In project B the same component delivered a green run at package level with more than 400 tests and failed in the integration run on two simulators with a single pair of tests; the project’s working rules have recorded since then that a green package run does not establish correctness. The working constitution of project A states the corresponding principle, that a live gate can be green on broken semantics. Every feature there therefore requires an end-to-end scenario of its own against the real target system, and a documented live run across 54 check cells ended with 16 cells passed. An honestly red result is worth something in this way of working, because the counter-class is documented as well: a gate that reported green three times although the child system under test produced malformed output. Because of such false-green cases, the E2E tests themselves have to be checked, for instance on whether they are able to fail at all.

E2E tests do have another weak point, and it is well measured. When the implementing agent sees the tests and iterates on them, it optimizes for passing them; whether the task is solved drops out of view. A benchmark on reward hacking in long-horizon coding agents documents as an extreme case a 2,900-line sham compiler that had simply memorized the test inputs; the gap between visible and held-back tests grew there with the size of the code (arXiv 2605.21384, 2026). My way out of this dilemma is end-to-end tests that humans curate or whose content humans prescribe. The scenarios and expected values come from the domain; the implementing agent runs them but does not write or change them. In addition, part of the scenarios can be held back and run only once the implementation is finished. That costs human working time at a point where it pays off, because these tests are the one artifact in the whole chain that the agent may neither produce nor reword.

Every implementation additionally goes through a loop of independent audits. I described the pattern in detail in “The second lane”; the core is enough here. One agent implements, a second agent with fresh context and the explicit task of refuting the result checks it, and a machine-readable verdict decides the merge. In project A it is additionally prescribed that reviewer and implementer come from different model families; if both are the same, the verdict is discarded. The reviewer runs there in a write-protected working directory of its own, without tool access to the running session, and its prompt, its raw log and its verdict are bound together by checksums so that a verdict can later be assigned to an exact state. The loop arises because a “not passed” triggers a correction round and the correction is checked again.

That these loops find real defects and not just formalities is shown by the verdict statistics. The machine-readable verdicts of the audit artifacts in project A read 77 times “fail”, 53 times “pass” and 9 times “inconclusive”; in project B, 23 of 82 logged gate runs ended in “fail”. Almost all the drift cases cited in this post come from exactly these review runs. Without the auditors they would have stayed in the system, because the respective implementation had reported itself finished.

The reviewers have limits of their own. In project A the follow-up check found a reasoning error in the report of a review model. The report cleared a test failure with the claim that the affected file was unchanged, which the diff refuted. A model used as a judge now goes through a calibration procedure of its own there, with independent double labelling, and as things stand it is not cleared for unsupervised use.

Research supports both precautions. An evaluation of around 541,000 judgements from 21 judge models found high consistency alongside weak validity; the authors call for a dedicated validation protocol for judges (arXiv 2606.19544, 2026). Cross-model review also works differently depending on direction. In one study a Claude review of Codex drafts raised the success rate from 71.6 to 89.7 percent, while the reverse pairing lowered it from 91.4 to 82.8 percent (arXiv 2607.21656, 2026). Which family implements and which one reviews is therefore itself a decision that has to be validated.

The third countermeasure starts before the implementation, with the form of the specification itself. For agents over long periods, prose is too ambiguous; each of the four drift directions from the finding is in the end a property of text that nobody executes. From this follows a shift left toward more formal specifications. A human or a lead agent assigned to the job first writes integration tests to implement against; where a test form does not suffice, a domain-specific notation that can be evaluated by machine helps. A failing test is a hard specification. It does not drift, because it stays red until the code fits; a Markdown section, by contrast, can assert something false for months.

In rudimentary form this already exists in both projects. Project A holds ten drift test classes of its own that check registries, configuration fields and manifests against the code on every run. Project B runs sabotage gates that abort with a defined error code as soon as a contract is violated; one of them, though, has also raised a false alarm because of a tooling artifact. The limits described above continue to apply. A test that the implementing agent is allowed to shape can turn out tautological, and what lies beyond the testable, such as architectural intentions and rationales, still needs prose, only much less of it. For the checkable core of the requirements, a test is nevertheless the most stable form I have seen in either project.

What it costs

The most visible item is token consumption. The operations notes of project B record that stranded agent runs, which hung at session boundaries without a commit, burned 250,000 to 350,000 tokens per resumption without progress. A single review run there produces 500 kilobytes to one megabyte of text. In project A the audit artifacts log the ratio between internal working log and visible result: a good 8,300 lines of raw log for a verdict of 64 lines, and this effort falls due again with every correction round.

On top of that comes operational effort that appears in no token bill. The notes from project B describe agents that hang without a completion message and can be recognized only by circumstantial evidence, such as absent file changes alongside missing processes. They also record the discipline of running only one active agent per work area, because a replacement agent started too hastily overwrites half-finished states. A resumed agent once discarded someone else’s not-yet-committed work; since then the rule is that discarding is allowed only by a reversible route.

A problem of its own is the models’ tendency toward micromanagement. Left unattended, the review loop escalates. The extreme values come from project A. A single ticket accumulated 96 audit documents, another went through 23 rounds in two calendar days, and a pure specification artifact without a line of product code was re-audited nine times; the ninth round still changed 122 added and 75 removed Markdown lines compared with the eighth. In doing so, the models invent ever finer occasions for review: bisection steps, diagnostic tickets, interim acceptances. The project notes now name this explicitly as a relapse to be avoided. In project B I intervened myself and stopped an agent that was spinning through its third correction round on the same test while the product code had long been correct.

Without fixed stop rules that bound the loop from outside, the approach therefore tips over. In project A several forms of such circuit breakers have become established. A two-correction-round rule ends the loop after two unsuccessful rounds and forces a decision between accepting with documented risk, descoping, or abandoning. This limit has empirical counterparts. Across seven models, the gains from self-correction loops concentrate in the first two rounds (arXiv 2604.10508, 2026), and in one investigation with smaller models, multi-agent debates cost 2.1 to 3.4 times the tokens at equal or worse accuracy, with peer critique destabilizing previously correct solutions (arXiv 2605.00914, 2026). Counted authorizations cap expensive runs in advance: a decision artifact authorizes exactly one start, without selective repetitions and without cherry-picking from historical runs. A concurrency cap restricts the simultaneously active implementation strands to two, and a prose cap sets, per work package, the added Markdown lines against the added product code lines, so that the documentation does not overgrow the code. I find it notable that the most recent measure points in the opposite direction: an explicit autonomy mandate that forbids the implementation strand to file interim reports and provides only a few defined return points.

Taking stock

Spec-driven development did not deliver on its central promise in two disciplined, long-running projects. The specification drifted against the code, specifications contradicted one another, and single artifacts even contradicted themselves. By everything the audit artifacts show, more discipline would not have switched this off: drift was the base rate of a working model made of months-long workflows and hundreds of context switches.

What did work is a different architecture of reliability: end-to-end tests check the behaviour of the system, independent auditors catch in their loop a large share of the defects that the implementation had reported as done, and stop rules keep this loop affordable. Where requirements exist as tests from the outset, the prose surface on which drift can arise at all also shrinks. The specification changes its role in this setup. It remains a starting point and a reference for review, against which deviations become visible and are then decided on deliberately; the role of the source from which the system reliably follows is one it did not fill in either project. The research fits this picture: an explicit specification phase measurably improves test pass rates, by 7 to 21 points in one recent investigation, but does not determine the result (arXiv 2607.27167, 2026). Part of the research pursues the counter-strategy and wants to prevent drift technically, for instance through gates that make spec-code divergence a merge blocker (arXiv 2606.27045, 2026). Whether that can be sustained over months, or merely shifts the point at which the maintenance costs arise, is empirically open.

None of this solves the underlying problem either. E2E validation, auditor loops, executable specifications and circuit breakers are mitigations. They make drift visible and give it a price; in both projects it has stayed nonetheless. As long as the work is done by actors that have no lasting memory and never see the whole system, drift arises anew at every seam. The review architecture works against a symptom; the cause lies in the shape of today’s tools.

The cost question remains open for me. In both projects, the effort for specification, audit and evidence taken together was larger than the product code, measured in lines. Whether a deliberately smaller specification surface with earlier behavioural tests would be cheaper in the end, I cannot derive from two projects. Only one item of that calculation seems certain to me: the ongoing maintenance of several hundred thousand lines of prose, along with their own drift.

The data

All figures and case examples from the two projects come from their repositories: from Git histories, ticket inventories, operations notes and the machine-readable audit and gate artifacts. The projects are internal undertakings and are deliberately described here only through key figures.

Sources

• How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions, arXiv 2605.29442 (2026), https://arxiv.org/abs/2605.29442
• SpecBench: Evaluating Specification-Level Reasoning for Software Engineering LLM Agents, arXiv 2605.30314 (2026), https://arxiv.org/abs/2605.30314
• Citation Discipline in Spec-Driven Development: A Cross-Model Empirical Study of Output Determinism and Automated Hallucination Detection in LLM-Generated Code, arXiv 2606.30689 (2026), https://arxiv.org/abs/2606.30689
• SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch, arXiv 2607.27167 (2026), https://arxiv.org/abs/2607.27167
• The Spec Growth Engine: Spec-Anchored, Code-Coupled, Drift-Enforced Architecture for AI-Assisted Software Development, arXiv 2606.27045 (2026), https://arxiv.org/abs/2606.27045
• SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents, arXiv 2605.21384 (2026), https://arxiv.org/abs/2605.21384
• Cross-Model LLM Code Review: Should you use Claude to review Codex or vice versa?, arXiv 2607.21656 (2026), https://arxiv.org/abs/2607.21656
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias, arXiv 2606.19544 (2026), https://arxiv.org/abs/2606.19544
• How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks, arXiv 2604.10508 (2026), https://arxiv.org/abs/2604.10508
• The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate, arXiv 2605.00914 (2026), https://arxiv.org/abs/2605.00914
• Atlassian Engineering: Why AI Agents Drift Mid-Task and How a Multi-Agent System Fixes It (23 June 2026), https://www.atlassian.com/blog/development/specialist-agent-orchestration-jira
• Thoughtworks Technology Radar: Spec-driven development (ring “Assess”, as of November 2025), https://www.thoughtworks.com/radar/techniques/spec-driven-development