Machine-Checkable Specifications

Human collaborators can ask follow-up questions and record a decision when requirements are ambiguous. An automated evaluator can only apply the rules, inputs, and observations it has been given. BrainGrid offers practitioner guidance on making those rules explicit; it is not a controlled study or a correctness guarantee. Its examples illustrate the problem , but the appropriate evaluation method remains context-dependent.

Instructions may be retained, but review is clearer when the evaluation contract is a separate, reviewed artifact. A 2026 arXiv preprint on delegation contracts for coding agents proposes four questions for reviewability: what was asked, what the agent was allowed to do, what came back, and what evidence supports it. This is a proposed framework, not proof that the four questions are universally sufficient. A specification can answer the first two and feed the spec-vs-implementation check afterwards.

1. Goal - one sentence. What must be true after the run that was not true before. If the goal needs three sentences, it is probably two tasks.
2. Boundaries - the authority envelope. Which files may change, which must not, and which behavior is off-limits. Boundaries are what turn scope creep from a debate into a finding.
3. Acceptance criteria - explicit decisions. Each names an evaluation mechanism, input or fixture, expected observation, and decision rule. Subjective outcomes identify the person whose judgment will be recorded.
4. Validation plan - which checks count. Tests, types, lint, builds, schemas, invariant checks, and any manual inspection are named up front. A pass is bounded to those checks, their inputs, and the evaluated environment.

Compiling a conversational prompt into this form takes minutes. Reality Graph writes the same four blocks as the contract artifact it checks a run against, so the file outlives the session that produced it. By hand it looks like this:

PROMPT (what you'd naturally type)
"Rate limiting responses are confusing clients, can you make the
429s more helpful?"

COMPILED SPECIFICATION (what the run is verified against)
Goal:      429 responses carry a correct Retry-After header
Boundary:  api/middleware/* only · rate-limit thresholds unchanged
Criteria:  [1] Retry-After present on every 429 response
           [2] value equals remaining window in seconds (±1s)
           [3] empty/malformed client IDs still get a 429, not a 500
           [4] 2xx and other 4xx paths byte-identical to before
Validate:  unit tests (pre-written) · types · lint · build
           manual: header visible behind the CDN (staging)

• Turn adjectives into observable decision rules. A threshold needs its unit, workload, environment, measurement method, and tolerance. Otherwise a precise-looking number can be as ambiguous as “fast”.
• Name the unhappy paths. Empty inputs, duplicates, missing permissions, timeouts. Left unstated, the agent invents its own policy for them - and its self-written tests will confirm that invented policy.
• Bind criteria to behavior, not implementation. “Uses a token bucket” ages badly and forbids better solutions; “allows 100 requests per minute per client” is the actual requirement.
• Pick the lightest format that stays decidable. Given-When-Then, schemas, property checks, invariants, examples, and recorded manual inspections can all work. The format makes a procedure explicit; it does not confer correctness.

• Over-specifying small tasks. The form can scale down, but risk determines the required depth. A small diff with high consequences can justify substantial boundaries and validation.
• Specifying without checking. A specification nobody compares against is documentation theater. The follow-up comparison is the point of writing it.
• Expecting completeness. Checks cover what the specification and evaluator express. Formalizing intent completely remains an open research problem according to a 2026 arXiv preprint. The method records stated intent; it does not establish its correctness or completeness.
• Letting the model write its own criteria unreviewed. Drafting with AI is fine; accepting the draft unread re-creates the circularity the specification exists to break.

Reality Graph can record a run's goal, boundaries, criteria, and validation plan, then capture the configured checks as part of the verification loop, and the outcome lands in an evidence report. That report is evidence about the recorded scope and executed checks, not formal proof or a guarantee of correctness, confidentiality, security, or compliance.