Twenty-Dollar Kill-Test

Earlier work measured whether parallel AI agents disagree on their own. Mostly they do not — nine agents on the same tickets forked on six of twenty-eight contested design questions. So the obvious move is to stop hoping for disagreement and instruct it: have one agent write three deliberately different plans, then have a second agent with fresh context choose between them.

That has an ugly failure mode. An agent told to produce three distinct paths can produce one real answer and two strawmen, and then a person adjudicates noise wearing the costume of rigour.

So before designing anything around it, we spent $20.28 trying to break it. The rules went into a public repository forty minutes before the first result existed, along with a written promise that surviving would not be counted as success.

Nothing broke. Two things turned up anyway. The agent hired to blend three plans adopted one of them whole — twelve decisions, one source, no edits. Then a second agent, handed the same three plans with no instructions at all, picked the same one.

ELI5 tells it as a kitchen. Technical has the registered rules, the cost table, and the part where our own trap caught us.

Say you run a kitchen where most of the cooking is done by people who are very fast and occasionally very confident about the wrong thing.

You have noticed something. When you give the same dish to three cooks separately, they mostly come back with the same recipe. Not always — where the dish genuinely leaves room, they go different ways. But most of the time you get three copies of one idea, and you have paid three times for it.

So you try something. Instead of hoping they differ, you tell one cook: write me three genuinely different recipes for this dish. Then you hire a fourth cook, who has never seen the first one work, and you say: here are three recipes, take the best parts of each and give me one.

The obvious way this goes wrong

A cook told to produce three different recipes has an easy way out. Write the one recipe you actually believe in, then write two bad ones on either side of it. Now you look thorough. You have given your boss a choice, and the choice was never real.

If that is what happens, the whole idea is worthless, and worse than worthless — it manufactures the feeling of having considered alternatives without any of the work.

So before building a kitchen around this, we spent twenty dollars finding out.

Writing down how you would lose

Here is the part that matters more than the result. Before running anything, we wrote down exactly what would count as the idea failing, and published it where anyone can see the timestamp. Not a description of what we hoped to find. A list of specific things that, if they happened, meant stop.

We also wrote down something less comfortable: that if none of those things happened, we would not get to say the idea works. Only that it survived one small hostile test.

That second promise is the whole trick. Without it, a test you designed yourself always passes, because you get to decide afterwards what passing meant.

What actually came back

The three recipes were real. Genuinely different approaches, differing where the dish left room and agreeing where the kitchen forces your hand — and the cook said out loud which was which. No strawmen.

Then the fourth cook, the one hired to combine the best parts of all three, read them and handed back the second recipe. Unchanged. Every decision in it, twelve out of twelve, taken from that one recipe. Nothing invented, nothing merged.

We thought that might be a fluke of how we had asked. So we ran it again with someone given no instructions whatsoever — here are three recipes, do something. They also handed back the second recipe, whole.

We had built a blender, and it kept handing us a knife.

One run of one dish proves nothing, and we are not allowed to say otherwise. But it is the kind of nothing that changes what you build next. If the second cook is going to pick rather than blend, then the expensive machinery for blending is machinery nobody asked for.

The trap that caught us instead

There was a second dish in the experiment, and it was bait.

We chose something we believed had exactly one correct recipe — the rules are written down elsewhere and there is nothing to decide. If our cook was going to invent fake alternatives, this is where it would have to, because there were none to find.

It did not invent anything. It produced three recipes that never once contradicted the rules, and differed instead on questions the rules genuinely leave open — details we had not noticed were open. Then we asked an uninstructed cook whether there was really anything to decide here, and it said yes, and named the same things.

Our trap had bait in it after all. We had just not read our own dish carefully enough to see it.

That is a finding about the experiment rather than about the cooks, and it is the one most likely to help someone else: before you use a task as a trap, check that it really has only one answer. Ask somebody with no stake, and believe them.

Twenty dollars, about half an hour, one dish that was not what we thought it was, and one result nobody ordered. We still do not know whether the idea works. We know a great deal more about how to find out, and that is what twenty dollars is for.

Prior work measured whether parallel agents disagree spontaneously. They mostly do not: in the first Crossfire study, nine same-model agents on identical briefs forked on six of twenty-eight contested design questions, a base rate of 0.214¹. The divergence that does appear is region-shaped — agents agree where the environment determines the design and fork where the ticket leaves judgement open.

If disagreement is scarce but its location is predictable, the move is to stop sampling for it and instruct it. One agent produces three deliberately distinct paths through a ticket at plan level; a second agent with fresh context measures them against goal outcomes written in advance and composes the plan that goes forward, citing which path each decision came from. The research name is Trident.

The failure mode is obvious enough to be disqualifying. An agent told to produce three distinct paths may manufacture one real answer plus two strawmen, at which point a human is adjudicating noise dressed as rigour, and the process has made the team more confident while making it no more correct.

What follows is the short version. The full method, the instrument validation, the cost accounting and the threats to validity are written up as a technical report⁶, which you can read in the viewer at the top of this page.

The registered rules

Rather than design a study around the idea, we ran a falsification pilot: a small hostile test built to break the idea if it can be broken, with the rules committed publicly before any data existed². Git history carries the ordering — the registration landed at 16:38Z and the results at 17:18Z, forty minutes later³. The registration commit's own sha doubles as the recorded shuffle seed for the handoff, so the ordering of the paths handed to the second agent could not be chosen after the fact.

The pre-commitment, verbatim from the registration:

if no falsifier fires, we will not claim the approach works, will not ship it as a default on this evidence, and will not call any playbook earned. We will say it survived a small hostile test, and name what a real test would require.

That sentence is doing more work than any result below it. A test you designed and can interpret afterwards will pass; naming the disqualifying observations in advance, and pre-committing that survival is not success, is what makes the exercise worth twenty dollars instead of being an expensive way to agree with yourself.

What ran

Six agent invocations plus a smoke probe, on two open tickets from a public repository frozen at one commit and verified unimplemented immediately before dispatch⁴. One task with genuinely open design — reaching a saved terminal session that is not the newest. One intended as a no-latitude control — implementing a contract someone else specifies.

Every check that decided anything was deterministic: JSON-path assertions over question-keyed declarations, regex answer keys, verbatim citation audits. No model graded another model anywhere. The auditor was validated against hand-built controls with known verdicts — a positive, a decorative-citation negative, a degradation case, a malformed case, a fail-closed floor — before it read a single line of real output. That ordering is not a detail; an instrument you tune after seeing your data is not an instrument.

Run	What	Wall	Cost (USD)
smoke	isolation probe	~3 s	0.24
A	generator + baseline (lost to a harness gotcha)	560 s	6.53
A2	generator + baseline, rerun	~510 s	6.53
B	generator + baseline, control task	433 s	4.40
C	reducer + baseline over A2's trio	298 s	2.58
Total		~30 min	20.28

The $6.53 in row two bought nothing and is in the table anyway. The harness deletes every workspace after grading unless an undocumented environment variable is set, which was discovered when a completed run with all assertions passing left no files to audit⁵. The sample is unrecoverable and is preserved unaudited as a record that it existed. A pilot that hides its own waste is not reporting the cost of the method.

Findings

The floor held, so the falsifiers were live. Three schema-valid plans on the open task, each answering all twelve questions, contested on all twelve: a transient first-paint strip with a picker chord, a modal picker matching an existing internal component, and a persistent badge with a drop-down menu. Divergence spent where judgement is open, agreement where the codebase forces it, and the plans stated which was which.

No fabricated divergence — and the trap caught its designers. On the control task the generator produced three paths claiming genuine latitude, with zero contract contradictions across every dictated cell and an unclassified rate of 0.0. Its three-ness lived in exec semantics, parser choice, and three different session-adoption policies. The decisive datum is that the uninstructed baseline independently judged the latitude genuine and named the same dimensions. The control carried more real design freedom than we had modelled, and the flag stands OPEN. The transferable lesson is about experiment design rather than about agents: a no-latitude control must be verified no-latitude — an uninstructed probe agreeing there is nothing to debate — before it can serve as an instrument.

Selection, not synthesis. Given three paths and explicit freedom to combine them per question, the reducer adopted one path whole: twelve adoptions from a single source, zero modifications, zero invented answers, every citation verbatim-honest. The uninstructed baseline reducer, same handoff, no instruction, independently selected the same path whole. Two clean-context agents, one told to harmonise and one told nothing, converged on wholesale selection of the same winner.

The escalation clause found nothing to escalate. Both reducers returned an empty undecidable-on-paper list. The wider design imagines expensive competing implementations spent only where paths cannot be told apart on paper; that clause has now run once and found no work to do.

The registered limitation, verbatim

Carried in every write-up by the terms of the registration:

we do not claim the harmoniser is context-free. It reads three artifacts that contain their own reasoning, and it runs on a stochastic system. What the design removes is structured leakage — the generator's framing about which path deserves to win, the ordering, the provenance. Any residual carry-through is chaos in the substrate, and we name it rather than pretending it away.

What this licenses, and what it does not

No falsifier fired, and per the pre-commitment that is not evidence the approach works. It survived a small hostile test at pilot scale, with one repetition of one task per stage. One repetition is an anecdote everywhere except where a categorical falsifier speaks.

It licenses running the real study on these instruments, with the task-selection repair above and coverage for on-paper undecidability. It does not license any claim that instructed divergence works, that harmonisation adds value over selection, or that the escalation clause earns its place.

What would falsify this

The claim being made is narrow — that this pilot's instruments work and its first datum is selection-shaped — so the ways it fails are narrow too.

The selection result dies if it is an artifact of one trio. Three paths where one is plainly strongest produce wholesale adoption from any competent reducer, and that is a property of the sample rather than of reducers. The test is a trio built so that the best answer to each question sits in a different path: if the reducer still takes one whole, the observation is real; if it starts mixing, this was a sampling accident and the blending machinery earns its place after all.

The instrument claim dies if the auditor's controls were too easy. They were hand-built by the same people who built the auditor, which is the standard way validation flatters itself. The real version is an adversarial set written by someone who wants it to fail, and that has not run.

And the whole exercise is worth less than it looks if the pattern is that a cheap pre-registered test always survives. A method that never kills anything is not a filter, it is a ritual. The number to watch is not this pilot's outcome but how many of the next ten die — and if that number is zero, the falsifiers are being written too kindly.