Machine-Checkable Specifications
Last updated: 2026-10-196 min read
A machine-checkable specification records requirements, boundaries, acceptance criteria, and the mechanisms used to evaluate them. Human-readable requirements, schemas, tests, invariants, and acceptance criteria play different roles; validation and verification evidence only support the checks actually performed. “Machine-checkable” does not mean complete, correct, or formally proven.
Contents
Why a prompt is not a specification
Human collaborators can ask follow-up questions and record a decision when requirements are ambiguous. An automated evaluator can only apply the rules, inputs, and observations it has been given. BrainGrid offers practitioner guidance on making those rules explicit; it is not a controlled study or a correctness guarantee. Its examples illustrate the problem , but the appropriate evaluation method remains context-dependent.
Instructions may be retained, but review is clearer when the evaluation contract is a separate, reviewed artifact. A 2026 arXiv preprint on delegation contracts for coding agents proposes four questions for reviewability: what was asked, what the agent was allowed to do, what came back, and what evidence supports it. This is a proposed framework, not proof that the four questions are universally sufficient. A specification can answer the first two and feed the spec-vs-implementation check afterwards.
The four building blocks of a checkable task
- Goal - one sentence. What must be true after the run that was not true before. If the goal needs three sentences, it is probably two tasks.
- Boundaries - the authority envelope. Which files may change, which must not, and which behavior is off-limits. Boundaries are what turn scope creep from a debate into a finding.
- Acceptance criteria - explicit decisions. Each names an evaluation mechanism, input or fixture, expected observation, and decision rule. Subjective outcomes identify the person whose judgment will be recorded.
- Validation plan - which checks count. Tests, types, lint, builds, schemas, invariant checks, and any manual inspection are named up front. A pass is bounded to those checks, their inputs, and the evaluated environment.
Compiling a conversational prompt into this form takes minutes. Reality Graph writes the same four blocks as the contract artifact it checks a run against, so the file outlives the session that produced it. By hand it looks like this:
task-compilation.md
Example - not real run dataPROMPT (what you'd naturally type)
"Rate limiting responses are confusing clients, can you make the
429s more helpful?"
COMPILED SPECIFICATION (what the run is verified against)
Goal: 429 responses carry a correct Retry-After header
Boundary: api/middleware/* only · rate-limit thresholds unchanged
Criteria: [1] Retry-After present on every 429 response
[2] value equals remaining window in seconds (±1s)
[3] empty/malformed client IDs still get a 429, not a 500
[4] 2xx and other 4xx paths byte-identical to before
Validate: unit tests (pre-written) · types · lint · build
manual: header visible behind the CDN (staging)Rules that make criteria checkable
- Turn adjectives into observable decision rules. A threshold needs its unit, workload, environment, measurement method, and tolerance. Otherwise a precise-looking number can be as ambiguous as “fast”.
- Name the unhappy paths. Empty inputs, duplicates, missing permissions, timeouts. Left unstated, the agent invents its own policy for them - and its self-written tests will confirm that invented policy.
- Bind criteria to behavior, not implementation. “Uses a token bucket” ages badly and forbids better solutions; “allows 100 requests per minute per client” is the actual requirement.
- Pick the lightest format that stays decidable. Given-When-Then, schemas, property checks, invariants, examples, and recorded manual inspections can all work. The format makes a procedure explicit; it does not confer correctness.
Limits and typical mistakes
- Over-specifying small tasks. The form can scale down, but risk determines the required depth. A small diff with high consequences can justify substantial boundaries and validation.
- Specifying without checking. A specification nobody compares against is documentation theater. The follow-up comparison is the point of writing it.
- Expecting completeness. Checks cover what the specification and evaluator express. Formalizing intent completely remains an open research problem according to a 2026 arXiv preprint. The method records stated intent; it does not establish its correctness or completeness.
- Letting the model write its own criteria unreviewed. Drafting with AI is fine; accepting the draft unread re-creates the circularity the specification exists to break.
How to write tasks for AI coding tools so the result is checkable?
How to write tasks for AI coding tools so the result is checkable comes down to four parts the task carries before the run starts: a goal in one sentence, boundaries that state what the tool may touch, acceptance criteria written as explicit decisions, and a validation plan that names which checks count. A task that has all four can be compared with its outcome; a task that has only the first is a prompt.
The acceptance criteria do the most work. An adjective such as “fast” or “clean” is not checkable; a decision rule such as a response limit, a named error path or a test that must pass is. Bind the criteria to behavior rather than to an implementation, and pick the lightest format that still lets a reviewer or a script decide yes or no.
Guides on acceptance criteria for AI agents, such as those by BrainGrid, PaellaDoc or the Agentic Skillset project, cover the writing side in more depth, and the agile literature on acceptance criteria applies unchanged. What this page adds is the second half: the written task is the reference a spec-versus-implementation check reads afterwards. A checkable task that nobody compares with the result stays a good intention.
Where Reality Graph fits
Reality Graph can record a run's goal, boundaries, criteria, and validation plan, then capture the configured checks as part of the verification loop, and the outcome lands in an evidence report. That report is evidence about the recorded scope and executed checks, not formal proof or a guarantee of correctness, confidentiality, security, or compliance.
A checkable specification gives you
- A persistent record of what was asked and allowed
- Criteria with a named evaluation mechanism and decision rule
- Unhappy paths decided by you, not invented by the model
- A reference that outlives the prompt and the session
It does not
- Guarantee the intent itself was complete or wise
- Require Given-When-Then or any fixed format
- Pay off without the follow-up comparison
- Turn test or review evidence into formal proof
If these boundaries fit how your team wants to ship:
FAQ
- How do I phrase tasks for AI coding tools so the result is checkable?
- A useful starting structure is a goal, explicit boundaries, acceptance criteria, and a validation plan. It is an example, not a completeness rule: each criterion should name an evaluation mechanism, input or fixture, expected observation, and decision rule; some outcomes still require recorded human judgment.
- What makes an acceptance criterion 'machine-checkable'?
- It has a defined evaluation mechanism and decision rule. Automated checks can evaluate tests, schemas, properties, or invariants; subjective criteria need a named reviewer and recorded judgment. Numerical thresholds also need a unit, workload, environment, and measurement method.
- Do I have to use Given-When-Then?
- No. Given-When-Then is one format. Schemas, property checks, invariants, examples, and recorded manual inspections can also define evaluation. A format does not make a requirement correct or complete; it only makes the chosen decision procedure more explicit.
- Isn't this a lot of overhead for small tasks?
- The form can scale down, but there is no universal time or length rule. Use enough detail to expose authority, risks, and the evaluation method; the appropriate depth depends on the consequence of a wrong change, not only on diff size.
- Why not just write a better prompt?
- A prompt can be retained, but a reviewed specification separates instructions from the criteria used to assess the result. A 2026 arXiv preprint proposes delegation contracts as a reviewability framework; it is research evidence, not proof that one artifact makes agent work correct or auditable.
- What happens to the specification after the run?
- It can become input to spec-versus-implementation checks, tests, validation, and review. Passing those checks is evidence about the stated criteria and evaluated environment; it is not formal proof or a guarantee that the specification was complete or correct.
- What are acceptance criteria for AI agents?
- Explicit decision rules, written before the run, that let a person or a script say yes or no to the agent's result: which behavior must be observable, which inputs must fail how, which checks must pass in which environment. They differ from a prompt in being decidable, and from a test suite in being readable by the person who ordered the work. Criteria the agent writes for itself after the fact do not count.
Keep reading
Sources
- arXiv preprint - Software Delegation Contracts: proposed reviewability framework for AI coding-agent work (2026)
- Addy Osmani - practitioner guidance on writing specs for AI agents (2026)
- BrainGrid - practitioner guidance on acceptance criteria for AI agents (2026)
- arXiv preprint - The Productivity-Reliability Paradox: proposed specification-driven governance (2026)
- arXiv preprint - Intent Formalization: open research challenges for reliable agent coding (2026)