Agent Drafting

Most of our Jev frames were first drafted by AI coding agents, and they are good at it: a complete state, a question set and a call graph in minutes, with sensible arguments about which questions can share a call and which must wait for the one before. They are also consistent in their blind spots, and a consistent blind spot is the kind that survives review.

We tested how far a written method gets them. Two agents received the same brief for triaging Omarchy’s GitHub issues, one with our method as a set of skills and one without, and one harness scored both designs on the same issues and ten traps, with every rule fixed before the second agent started. The method bought a steadier, better-documented design. It did not buy foresight: each design handled six of the ten traps, and neither asked about a security report. The step that separated designs was writing the traps before any question, so our method now starts there. In this test, neither agent imagined the embarrassing cases on its own.

Story tells it as a kitchen during service. Technical has the comparison, the serial and parallel calls the agents chose, and what they cost.

A good restaurant kitchen keeps a binder. Every dish is in it: what goes on the plate, in what order, how long each part takes. Hand the binder to a capable new cook and by the second night the plates look like the chef’s.

The binder also teaches timing, which is most of what service is. Sides that do not depend on each other get fired together, because making the table wait for each one in turn buys nothing. Dessert waits, because nobody knows whether the table wants dessert until the mains come back. A cook who has learned that difference is doing what an engineer does when deciding which questions to send to Jev in one call and which to hold for a second one (serial and parallel calls): batch whatever is already known, and wait only for the answer that changes what happens next.

We had our frames drafted the way a kitchen trains cooks. AI agents, the kind that write software from a description, drafted the facts each Jev call would see, the questions it would be asked, and the order the calls would run in. They are quick, and they argue timing well. One of them, working on Omarchy’s GitHub issues, found for itself that a second call was worth paying for: when an answer came back uncertain, rereading the whole issue changed about one answer in five.

Then we tested the binder. Two agents got the same order ticket, a brief for sorting Omarchy’s open issues, and one of them also got our binder, a written method in six parts. We decided in advance how both kitchens would be judged, wrote it down, and only then let the second one start.

The cook with the binder turned out the steadier kitchen. Asked to cook the same eighty plates three times over, it changed its answer on half as many as the other. It also explained every choice it made, which is the binder’s whole purpose.

What it did not do was think of the allergy. We had written ten awkward orders before either kitchen opened, the kind that embarrass a restaurant: a message in another language, a complaint about something that was never broken, a report of a security hole, the same order placed twice. Each kitchen handled six of the ten. Neither asked about the security report. The binder made the cooking steadier and left the cook’s imagination where it was.

A third kitchen, run by the chef who wrote the binder, had handled seven. Before designing anything, that chef had written down those ten awkward orders and how each should be handled. So that is now the binder’s first page. Before service, write down the orders that would embarrass the kitchen tonight, and walk each one through before the doors open.

There were other lessons in how the cooks worked. One kept labelling ordinary ingredients as if they were house specials (backticks, in Jev’s terms), a habit from how cooks everywhere write their notes, and a person had to spot it before a check could be added. Another wrote a summary claiming all eight of a certain kind of problem were one thing, and on rereading, six were. The cooks were capable and confident in a steady, repeatable way, and that steadiness is what a tasting before service exists to check.

So the kitchen runs like this now: agents draft, and draft fast, and every draft meets three checks before it ships, the awkward orders it must survive, the numbers it said it would produce before the first run, and people who taste the food.

An agent-drafted frame is plausible by default: complete, internally consistent, well argued. None of that is evidence it decides well, and in our test the drafting agents did not notice what they had left out.

What the agents did

Across this work, AI coding agents drafted state builders, question sets, the written meaning of every answer band, and the call graph; ran the probes; reviewed each other’s designs; and, when people’s labels were slow to arrive, labelled eighty issues to serve as a backstop key. Our method for engineering a Jev decision is written as six skills an agent runs in order: the decision’s shape, the state, what each probability means, the call graph, and evaluation, with a conductor over them.

Serial or parallel

The rule our method teaches for call graphs: batch independent questions into one call; pay for a second call only when a later question needs a fact the first one produces; make every follow-up bring new evidence; and price each split at the real volume. Three designs for Omarchy’s issue queue answered the question differently, two of them drafted with that rule in hand:¹

Design	Calls per issue	Cost per 1,000 issues	The serial step, and why
The reference design	2.98	$0.24	A call that reads the comments, run only when the first call could not settle the kind; the comments are new evidence.
Agent with the method	1.75	$0.40	A second call that rereads the whole issue when an answer is uncertain; its own evaluation found 21 of 96 uncertain answers moved a band on the reread.2
Agent without the method	1.00	$0.18	None. It cuts each issue at 6,000 characters, and lost the explanation that followed a long log.

Split decisions are cheap at this volume. Keeping one sub-decision in its own call costs about ten cents a day at Omarchy’s pace and about ninety dollars a day at twenty decisions a second, an estimate from the median size of the shared state and the published price.³ A pattern I want to test next runs serial at the level of the whole queue instead of the single issue: a broad parallel sweep over every issue with a state built for general questions, then code routes the flagged ones into deeper calls with states built for them.

The comparison

The brief given to the agent already triaging Omarchy’s issues was handed, word for word, to a fresh agent with the method and one extra line telling it to use the skills. Every measure and prediction was committed before the second agent started, both designs’ questions ran through one harness over the same 80 randomly drawn issues and ten adversarial fixtures, and neither design was read until both were finished.⁴

Measure	Without the method	With the method	What was predicted
Traps handled, of 10	6	6	at least 7 against at most 4: failed
Issues whose readings changed across three identical runs	7.5%	3.75%	no prediction
Conformance to the method’s own checklist	10 of 19	19 of 19	favours the method by construction
Cost per 1,000 issues	$0.18	$0.40	under $0.50: held
Agreement with a model’s labels on kind, of 80	77	77	provisional

On the other two labelled fields the comparison depends on how an unanswered field counts. The design with the method leaves more fields unmeasured and is right more often on the ones it answers; counted strictly, it trails on whether a maintainer should look now.⁵ Those labels come from a model, a stand-in until people’s labels arrive, so every figure against them is agreement with a model and runs high.

The traps the designs missed were cases neither author considered: a security report, a stale issue with no thread, someone who had not found a feature that exists. Both designs ranked the security report as an ordinary bug. The one design that handled seven, the reference, was written by an author who wrote the ten traps first. The method now opens with that step.

Where people and checks caught the agents

• Seventeen backticks around command names that point at nothing in the state, in the design drafted with the method. A person noticed; a check that fails any such frame now runs before a frame can.
• A changelog entry claiming all eight contradiction hits in a run were problems belonging to other projects. Re-read issue by issue, six were, and the correction is recorded beside the frozen design.⁶
• The reviewing agent’s sharpest point: the design with the method ranks what reports claim, so a confident but wrong diagnosis would look ready to work, and nothing checks the file a report blames until people label issues.

What would falsify this

The claim is that the method improves steadiness and documentation but not foresight, and that writing the traps first is what adds foresight. A fresh agent given the revised method, run under the same protocol, that handles no more traps than today’s would falsify the second half. People’s labels could still show the method buys accuracy the traps do not measure, which would falsify the first. That run is planned.