The Spec-vs-Implementation Check

The numbers describe a strange equilibrium. In Sonar’s 2026 State of Code survey, 96% of developers say they do not fully trust that AI-generated code is functionally correct – yet only 48% always check it before committing, and 61% report AI code that looks correct but is not reliable. AI already accounts for roughly 42% of committed code in the same survey. Distrust without a method becomes resignation.

The failure mode is specific. AI code rarely fails loudly; it fails in the gap between what the requirement meant and what the model understood it to mean. Reviewing the diff cannot catch that gap, because the diff only shows what was built – not what was asked. The missing artifact is the written intent, and every merge without it adds to the debt.

The obvious fix – let the AI test and review its own output – has a structural flaw: both sides of the comparison come from the same assumptions. Tests written by the generating model verify what the model assumed the function should do, which is subtly different from what it should do. Recent research on AI-assisted review draws the same conclusion: a specification is the quality gate precisely because it introduces a reference that is independent of the model’s training distribution – without it, review checks code against itself.

How large the self-assessment gap can get is documented: one team measured 449 AI code reviews and found the system rated itself 98.6% valid while independent validation put it at 69% – a thirty-point gap between confidence and reality. The conclusion is not “never use AI review”; it is: separate the reference from the generator.

Tool-agnostic – it works with Claude Code, Cursor, Copilot, or any agent:

1. Write the Soll (target state) before the run. One goal sentence, the boundaries (files and behavior that must not change), and 3–7 acceptance criteria. Five minutes, plain language.
2. Make every criterion checkable. “Returns a correct Retry-After header on 429 responses” is checkable; “improves rate limiting” is not. If you cannot phrase a yes/no question, sharpen the criterion.
3. Let the tool run unchanged. The method adds nothing between you and the agent – same prompts, same speed.
4. Compare the Ist (actual result) against the Soll. Walk the diff criterion by criterion. Off-scope changes are findings even when they look like improvements – scope creep is where silent regressions hide.
5. Record the outcome. Which criteria passed, what was skipped, what stays open – a few lines stored with the change, so the next person is not re-deriving your check. That record is what an evidence report formalizes.

The whole artifact is small enough to read at a glance:

SOLL (written before the run)
Goal:      Return a correct Retry-After header on 429 responses
Boundary:  api/middleware/* only · rate-limit logic untouched
Criteria:  [1] header present on every 429   [2] value = window rest
           [3] no change to 2xx/4xx paths    [4] unit tests pass

IST (checked after the run)
[1] PASS   [2] PASS   [3] FAIL – 403 handler was also edited (off-scope)
[4] PASS (61 green, 3 new – written before the run)

Verdict:   REJECTED → re-run with boundary reminder · re-check [3]

• Spec theater. A spec nobody compares against is ritual, not verification. The check is the point – the document is just its input.
• Over-specification. Describing the implementation instead of the intent produces specs that age badly and get skipped. Specify what must be true, not how to build it.
• It cannot catch what the spec missed. The method verifies the stated intent. Requirements you did not think of stay unverified – this is why research treats intent formalization as a grand challenge, not a solved problem.
• Skipping the record. An unrecorded check evaporates. The team cannot tell a verified merge from a lucky one a month later.

Everything above works manually with a text file and discipline. Reality Graph systematizes exactly this method: the Soll becomes a structured task with boundaries, the Ist is checked against it after the run, and the outcome lands in a reviewable report – as part of the broader verification loop.