Differential spec analysis

Differential spec analysis is a technique for finding the holes in a specification by exploiting the fact that agents are cheap enough to build the whole thing more than once. Run several independent agent loops against the same brief, let each produce a complete implementation, then compare the results. Wherever they diverge, someone made a decision the spec didn't cover.

The name and the worked example come from Josh Bleecher Snyder, applied to exe.dev's distributed DNS server in claude-is-not-a-compiler; he'd written the technique up separately on commaok.xyz.

The observation that motivates it

Agents ask questions. Bleecher Snyder's loops raised them at every level, from structural approaches down to line-level concerns, and answering those questions is the obvious way to firm up a spec.

The problem is what they don't ask:

It was shocking how many important decisions the agents never asked about but simply made — and made differently.

An agent that hits an ambiguity and resolves it silently produces working code and leaves no trace of the choice. You can't find those by reading one implementation, because in one implementation the decision looks like the only way it could have been. You find them by comparison — divergence is the signal that a decision point existed at all.

The loop

Settle the top-level strategy and architecture with humans first. This is the brief; the technique tests it, it doesn't produce it.
Prompt multiple concurrent agent loops to build the entire system — tests and adversarial review included, not a sketch.
Ask fresh agents to compare the completed implementations and identify interesting deviations. The comparison is itself agent work.
Work through the divergences: experiment, pick, and sometimes revert a pick that generated regret.
Codify each settled decision as terse written guidance.
Throw the implementations away and repeat.

Bleecher Snyder ran the full cycle three times before building a keeper, invoking Brooks ("plan to throw one away; you will, anyhow") and Craig Zerouni ("if you plan to throw one away, you will throw away two"). Total cost was about a week of one person's attention.

What comes out

The artifact is the guidance document, not any of the implementations. By the third pass it was empirically sufficient to steer an agent through the decisions that mattered — goals, architecture, and the occasional low-level specific such as the exact data type for a load-bearing concurrent cache. "Empirically sufficient" is the useful standard here: the spec is done when a fresh agent following it stops diverging in ways you care about.

The canonical divergence from the DNS case: replication catches up by requesting all entries after the last known one, then long-polls for more. Database rollbacks are rare but real, and they violate the append-only assumption the whole scheme rests on. Every agent noticed the hazard; every agent handled it differently. The design that survived tags each row with a randomly generated timeline value, carries the edge's timeline for row N in every "entries since N" request, and treats a mismatch as proof that history was altered — falling back to a full clean re-sync.

Style differences show up too, and are less actionable but not worthless: Claude and Codex both judged Claude's system more elegant and Codex's more thorough.

Why it works

Ambiguity in a spec is invisible from inside any single reading of it. Every other approach to finding gaps — careful review, checklists, asking the implementer to flag unknowns — depends on someone noticing that a choice was available. Differential analysis doesn't require noticing; it makes the gap mechanically observable as a diff. The cost is building N systems instead of one, which is exactly the cost agents collapsed.

This is the same logic as differential testing (run two implementations of a protocol against each other and investigate disagreements), lifted from behavior to design intent.

Limits

Correlated blind spots. Independent loops on the same model family share priors. A decision every agent gets wrong the same way produces no divergence and stays invisible. Bleecher Snyder's use of both Claude and Codex mitigates this partially.
Divergence isn't ranked. The diff tells you a decision existed, not whether it mattered. Sorting signal from style noise is human work, and it's the part that consumed his week.
It needs a real brief. With a vague enough starting point everything diverges and the output is noise rather than a decision list.
It presumes throwaway implementations are affordable. True for a scoped internal service; less obviously true for anything with migrations, data, or users already attached.

Cross-references

claude-is-not-a-compiler — the source and the DNS-server case
vibe-engineering — the practice this technique serves
llm-output-variance — the underlying nondeterminism this exploits rather than fights
specsmaxxing, acceptance-criteria-ids — spec-as-primary-artifact, reached by a different route
log-distributed-llms — the pessimistic reading of multiple agents disagreeing