FeatureValidation Receipts
Did the AI actually run the tests?
„The tests pass“ is a claim an agent makes about its own work. Reality Graph runs the approved checks itself, as an argv list with no shell and a bounded timeout, and records the exit code it observed. Anything it did not run is recorded as attested, and the two are never merged into one word.
Last updated:
Provenance
Executed or attested, with no third value
The distinction sounds academic until someone reads the record six weeks later and cannot tell whether a result was produced or reported.
| Provenance | How the result was obtained | What it is worth |
|---|---|---|
| Executed | Reality Graph ran the approved command itself and observed the exit code. | The strongest statement available: the command ran, and this is what it returned. |
| Attested | A person typed in the result of a check something else ran. | Useful context, never validated. An attested exit code is a claim, and it is stored as one. |
| Missing or unknown | A required result never arrived, or arrived malformed. | Forced blocking. It is not normalised into a pass. |
validation-receipt.json (excerpt)
Example – illustrative record, not real run data{
"command": ["pytest", "-q"],
"provenance": "executed_by_reality_graph",
"exit_code": 0,
"duration_s": 12.4,
"shell": false,
"timeout_s": 600
}Acceptance criteria
The step that reads your criteria and still does not decide
An optional step can ask a model whether each acceptance criterion is met. It is worth having, and it is worth knowing exactly how little authority it has.
The step is selected explicitly, needs an authenticated Codex CLI, and never sets the outcome. If it did not run, the record says not proven rather than upgrading the result, which is the behaviour that keeps a missing review from looking like a passed one.
Everything that decides the outcome stays deterministic. What the model contributes is a reading, filed next to the evidence, and the verdict is computed without it.
By design
What a green check proves, and what it never could
A pass is a narrow fact, and narrow facts are the only kind worth building on. Selling it as a broad one is how verification tools lose the trust they exist to create.
What you get
- Run approved commands as an argv list, with no shell, a bounded timeout and a filtered environment.
- Re-check the approved specification immediately before execution, so a stale plan-time approval cannot be reused.
- Record the observed exit code, and record separately whether Reality Graph or a person produced it.
- Force a missing, malformed or unknown result to block instead of passing.
What it is not
- Sandbox the command. It constrains which command runs, not what that command may then do, and there is no network isolation.
- Validate an attested result. A typed-in exit code is never checked against anything.
- Measure how strong your tests are. There is no assertion counting, no coverage tooling and no mutation testing.
- Let the semantic review step decide the outcome. It records not proven rather than upgrading a verdict.
- Talk to your CI. There is no pipeline integration, no Git hook and no pull-request check.
A pass means the command exited zero. It does not mean the task was done correctly, and this page will not imply otherwise.
Questions people actually ask
- How do I know Reality Graph ran the command rather than recording a result?
- Because it records which of the two happened. There are exactly two provenance values: Reality Graph executed the command, or a person attested that something else did. A mission can require the first, and no third value exists to blur the difference.
- What stops an approved test command from doing something else entirely?
- Less than people assume, and the honest answer matters here. Commands run as an argv list with no shell, on a bounded timeout, in a filtered environment, and the approved specification is re-checked immediately before execution so a stale approval cannot be reused. That constrains which command runs, not what that command may then do: there is no sandbox and no network isolation.
- What happens when a required check never runs at all?
- It counts as blocking, not as neutral. A missing, malformed or unknown result is forced to block rather than normalised into a pass, because the alternative quietly rewards a check that failed to start.
- Does a green run mean the change is correct?
- No. A pass means the command exited zero. Whether your tests actually cover the behaviour that mattered is a question about your suite, and Reality Graph does not measure the strength of a test: there is no assertion counting, no coverage tooling and no mutation testing in the product.
Ask your last agent run for its exit code
Not the sentence about it, the code itself, and who observed it. That question is the whole feature.