Trap Sets

Before I build something that makes decisions with Jev, I write down the awkward cases it has to get right: a security hole filed as an ordinary bug report, a complaint about something fixed two releases ago, a rant with nothing in it to act on. I call those traps, and the finished system gets tested against them.

This week we found out what happens when the builder already knows the traps. We gave an AI agent our method, which listed the kinds of trap we had used before, and asked it to design triage for Omarchy’s GitHub issues. It handled 9 of the 10 traps it had been told about. Then we tried it on 10 traps it had never seen, and it handled 5. Designs built without our list handled 7. It had memorised the list.

So the list is gone. Traps now come from a short set of questions borrowed from chemical-plant safety reviews, asked of every part of a bug report, and they are written and locked away by someone who isn’t building the system.

Story walks through it alongside a driving test. Technical has both runs, the procedure, and a 25-item set sealed before any design saw it.

Some of what we build with Jev makes decisions on its own. One system sorts the issues people file against Omarchy, an open-source Linux desktop: is this a bug, could someone act on it as written, does a maintainer need to see it this week.

Most issues are easy to sort. A few are awkward. A security hole arrives filed as an ordinary bug report. Someone complains about a problem that was fixed two releases ago. A furious rant turns up with nothing in it anyone could act on. A person working through the queue would usually catch these. A system might not.

So before we build, we write the awkward cases down, each with the right way to handle it. Each one becomes a trap: we give the finished system the awkward case, and the trap is sprung if the system answers it incorrectly. Testing a system against its traps is a lot like testing a learner driver on a route.

The last series ended with a claim: writing traps first is what helps a system see trouble coming. To test it, we gave a fresh AI agent our revised method, which opened by telling it to write traps and named the kinds of trap we had used before. Of the ten traps those kinds described, the system it designed handled nine, which is a learner passing on the route the instructor drove with them the week before.

That says little about streets they’ve never driven, so we tried the system on ten traps nobody had shown it, written and put away before it finished (a held-out set). It handled five. Two systems designed earlier, without the revised method and never shown its list, handled seven each. Two of the traps it sprang had caught no other system, and both times it had believed what a report said about itself: a report bundling four unrelated problems was sized as one big job, and a furious rant was read as a system that had stopped working and marked for a maintainer’s attention now.

A test centre with a known route stops publishing it, and the list of trap kinds came out of the method. New routes are drawn by a procedure instead of from memory. Engineers who design chemical plants have worked this way for decades, walking the plant drawing one pipe at a time and asking the same short list of questions of each part (guide words): what if nothing flows here, more than should, less, the wrong way, or something other than what belongs? For an issue report, the parts are its title, its steps, the machine it came from, its attachments and its thread, and the same seven questions go to each. Most of the answers are nonsense, and the few that aren’t become traps.

Each hazard gets tested twice: the same junction once with a cyclist and once without, where only the cyclist should change what the learner does (a matched pair). A plain straight road goes in too, where hesitating is the mistake (a positive control), because a learner who brakes at everything fails too.

The route is drawn by someone other than the instructor and sealed until test day. A separate agent wrote a third set of 25 traps this way and sealed it before any system ran, and the four systems we already had then drove it. The best handled 15 of the 25. One kind of hazard was handled by every system every time, so that stretch of road can show a learner getting worse and never one getting better.

Whether designers who draw their own routes this way do better on streets nobody showed them is the next test, run in a smaller town where I know every street myself.

The last series ended by claiming that writing traps first gives a design foresight, and named the run that would test it. That run is below: the design handled 9 of the traps it had been told about and sprang 5 of the 10 it had never seen.

The prediction

Agent Drafting reported a pre-registered comparison in which two agent-drafted designs for Omarchy’s issue triage each handled 6 of 10 traps, and a reference design, whose author wrote the traps before designing, handled 7. It concluded that writing traps first is the step that adds foresight, and named its own falsifier: a fresh agent given the revised method, run under the same protocol, that handles no more traps than before.

The run

The revised method put a step before any question: write the traps, each with its right handling, and design until they pass. It also named the categories the first ten traps covered. A fresh agent built a design with it. Before that agent finished, ten more traps were written and held out, five in categories the revision named and five in categories it never mentioned, and the three earlier designs ran on them first.¹

	Without the method	First method	Reference	Revised method
The first ten, whose categories the revision named	6	6	7	9
Ten held-out traps	7	7	7	5
Held-out, in categories never named	5 of 5	5 of 5	5 of 5	4 of 5

The falsifier fired. The design built with the revised method handled fewer held-out traps than the designs that never saw the revision. Its two misses of its own came from believing the report. It scored a report of several unrelated problems 1.39 on its 0–3 scale for detail, well above its 0.55 floor for asking the reporter for more, and sized the lot as one deep job. It read a rant as holding nothing to act on (0.03) and also as describing an unusable system with a foundational fix (2.98), which put the rant in front of a maintainer under the urgency reading every design was scored on. Every design missed the same three held-out traps: a pasted password, which scored 0.06 on the revised design’s security question; a licensing problem, filed as cosmetic; and an error in the reporter’s own configuration, which only reads as support once you know the failing line was the reporter’s edit.¹

The held-out set was also easier than it was written to be. All three earlier designs handled all five of its traps in unnamed categories, so the bar the new design had to clear was partly set by traps that separated nothing. One designer per condition makes this thin evidence about a method, and the protocol says so. It is still the result.

Why a named trap stops testing

A method that names trap categories turns them into requirements, and a design built from the method meets them. That is useful, and it ends those traps’ life as a test. The revision was the leak: naming the categories let the next design pass those traps without imagining anything new. The same thing happened one level up. The guide-word grid below generated half of the third trap set, and the method now teaches the grid, so for any designer who reads the method that half is no longer unseen.²

Traps by a procedure

The list came out of the method and a procedure went in, borrowed from hazard and operability studies in process engineering, where a team walks a plant drawing part by part and asks a fixed set of guide-word questions of each.³

1. List the parts of an input. For an issue report: title, steps, environment, attachments, thread.
2. Apply each guide word to each part: no, more, less, as well as, part of, reverse, other than. Keep the cells whose right handling a design could plausibly get wrong, and record why each dropped cell was dropped.
3. Write each kept case as a matched pair: a base item and a copy with one change that flips the right handling. A lone item can pass for the wrong reason, and a pair tests whether the design sees the difference.
4. Add positive controls, where caution is the failure. A plain, explicit bug report has to come back confident, and a design that sends everything to a person fails them.
5. When the traps will judge a design, someone other than its designer writes and seals them, with outcome states written so exactly one holds for any result: handled, confidently wrong, or missed.

A sealed set, with baselines first

A separate agent wrote a third set this way: 25 synthetic Omarchy-style issues in nine families. Four families cover what the revised method teaches, four come from the grid, whose 35 cells are recorded with the decision on each, and one is controls. The set was committed and sealed before any design ran on it, and the session that would build the next design received family names, counts and scores, never item text. The agent that wrote it had read the earlier results, a confound the protocol records rather than corrects.⁴

The four existing designs then ran on it three times each, for $0.069 in all, before anyone wrote a bar for the next design to clear. The first run, as handled, confidently wrong and missed:⁴

Of 25	Without the method	First method	Reference	Revised method
Handled	13	12	15	14
Confidently wrong	3	2	9	9
Missed	9	11	1	2
Whole grid families handled, of 4	0	0	0	0

Running the baselines first showed what the set can and cannot measure. One taught family was handled by every design in every run, so it can show a regression and never an improvement. No design handled a whole grid family in the first run. The bar for the next design comes from those numbers rather than from a guess about what a good design should score.

On a small problem

On a small decision the procedure takes a few minutes. This week a daily DMARC aggregate report said every message from a small business domain I look after had failed DMARC since reporting began in July. Guide words over the report’s parts turned up two cases a quick read would have missed. Part of: in the oldest report, 3 of 5 messages left with the relay’s bounce domain as the envelope sender, so SPF could never align for them, and only DKIM could fix them. Other than: the relay provider’s neighbouring cluster publishes an SPF include that looks right and covers none of the servers actually sending, and the first fix drafted for the domain named it. The same pass asked whether any step needed Jev, and none did, because DNS or the reports answered every question exactly.⁵

What would falsify this

There are two claims here. The first, that a design built from a method naming trap categories learns those categories rather than the habit of finding new ones, rests on one designer per condition. Three or more designers per condition, with the method-built designs handling unseen traps as well as the others, would falsify it. The second, that procedure-found traps sealed by someone else fix this, is untested. If designers taught the guide-word procedure handle a sealed set no better than designers without it, the procedure does not transfer either. The next comparison runs at least three designers per condition on a smaller problem, AI-isms in my own writing, where I can label every answer myself.