Frozen Questions

One way to check a Jev design is to ask people the same questions it answers and see how often they agree. For that to mean anything, the people and the design have to be answering exactly the same question.

We picked 80 of Omarchy’s GitHub issues at random and put them on a public page. For each one, the page asked whether a maintainer should look at it now, rather than later or never. Until people’s answers came in, an AI model answered all 80 to stand in for them, and its notes showed a narrower idea of now: a security problem, someone locked out or losing data, a jammed update, something breaking for lots of people at once. Another agent reviewing the work noticed these were two different questions, so every urgency score we had reported compared answers to one with answers to the other.

I rewrote “now” as a single sentence, gave it a version number, and froze it: the wording stays exactly as written, and any change becomes a new version. Answers to the old wording are kept under their own name and never counted as answers to the new one.

Story walks through it with an exam and its answer key. Technical has both wordings, the frozen sentence, and a prediction with the same kind of loose spot.

Some of what we build with Jev makes decisions on its own. One system reads the issues people file against Omarchy, an open-source Linux desktop, and decides for each one whether a maintainer should look at it this week.

To find out whether it decides well, we ask people the same question about the same issues and compare. Their answers work like the answer key a teacher marks an exam with, and an answer key only works if it was written for the same exam the students sat. For our urgency question, it wasn’t.

We drew 80 issues at random and put them on a public page. For each one, the page asked people whether a maintainer should look at it now, rather than later or never. Until people’s answers came in, an AI model answered all 80 to stand in for them, with a short note beside each answer (a stand-in key). The notes showed a narrower idea of now: a security problem, someone locked out or losing their files, an update that had jammed, a fault hitting many people at once. Plenty of issues a maintainer would happily look at this week don’t fit that.

So the systems were being marked against one exam while people were being handed another, and the scores looked normal, the way marks against the wrong key usually do. A second agent reviewing the comparison put the page and the notes side by side and saw the two questions.

An exam board handles this by printing one paper and fixing its wording before anyone sits it. I wrote the question as a single sentence, gave it a version number, and froze it: the wording stays exactly as written, and any change becomes a new version (a frozen question). I wrote it myself, since I’m the one who acts on the answer: would you put this ahead of ordinary backlog work in the next weekly triage pass, ignoring how many other issues are waiting?

The last clause keeps capacity out of the judgment. How many issues get looked at each week is a separate decision, and someone answering with a long queue in mind says no to things they’d say yes to on a quiet week. A second question went onto the paper as its own item: should someone check this privately before it’s handled in public?

Answers already given to the old wording stay in the record under their own name, and they’re never counted as answers to the new question. Questions sent to Jev work the same way. Jev reads the words it’s given literally, so a reworded question is a new question, and answers to two versions never share a column.

The prediction we wrote before the comparison ran had a loose spot of the same kind. It said the design built with our method would be within five points of the other on accuracy, without saying whether a question the design left unanswered counts as wrong. Counted one way, that design came out six points ahead on urgency; counted the other, nearly nine behind. The report gives both, since picking one after seeing the results would mean writing the key after marking the papers.

Every urgency score in the triage comparison compared answers to two different questions, and nothing in the numbers showed it. The labelling page asked one question, and the stand-in labels the designs were scored against answered a narrower one.

The two wordings

The comparison scores Jev triage designs against labels on 80 of Omarchy’s open issues, drawn by stratified sampling before any design’s answers existed. One labelled field is whether an issue needs a maintainer now. Two wordings of it were in use at once:¹

Where	What “now” meant
The public labelling page	“Should a maintainer look at this now, rather than later or never?”
The model’s stand-in labels, from its notes	A security problem, a lockout or data loss, a wedged update, or a broad current regression

An adversarial review of the comparison found the mismatch on 2026-09-25. By then every design had been scored against the stand-in labels on this field:²

Needs a maintainer now	Without the method	First method	Reference	Revised method
Of answered	61/78	54/64	60/78	53/79
Strict, unanswered counted wrong	76.25%	67.5%	75%	66.25%

Every number in that table compares a design’s reading of urgency with a narrower rule than the page asked people to apply. The revised design said “now” for 32 of the 80 where the stand-in said it for 15. Some of that gap may come from the stand-in’s narrow rule rather than from the design, and until labels to one wording exist, the table cannot say how much.

A second mismatch, in the mapping

Each design has its own outputs, so an adapter maps them onto the labelled fields. The agent that sealed the next trap set found that two designs’ adapters read urgency from severity, so they escalated items the designs themselves had parked, one in an upstream lane and one in a backlog. Some needs-now disagreements therefore belong to the adapter rather than to the design.³ The rule that follows is the same one again: write each design’s adapter before the design runs on anything, mapping its own routes onto the frozen question, and never re-derive a reading from answers already seen.

The frozen sentence

I chose the second version of “now” myself, as the person who acts on it: “Would you put this ahead of ordinary backlog work in the next weekly triage pass, ignoring how many other issues are waiting?” The last clause keeps queue length out of the judgment. How many items a person reviews per pass is a budget, applied in code over stored answers. A look-now cut-off fitted to one project picked 3 of 1,101 issues when the same kind of design ran on another.⁴

A separate safety question was added rather than folded in: “Should someone check this privately before it is handled in public?”, answered yes when a secret or private data is exposed or a way past access controls is shown.

• The page, its export block (headed label-80 v2), its storage key and the label collector all carry version two.
• An answer to the first wording is kept as now_v1 and never scored as “now”.
• Labelling order is frozen in a file: the 39 issues whose urgency is disputed come first, then the other 41, each group shuffled with seed 20260925, with nothing on the page marking the boundary.
• The labeller works blind to every design, and the stand-in’s answers unlock only after they have copied their own.

Freezing the prediction too

The protocol was committed before the second design started, under a rule that nothing above its deviations section is edited; eighteen deviations are recorded, each with its reason.¹ That is what let a failed prediction stay failed: it predicted the method’s design would handle at least 7 of 10 traps and the other at most 4, both handled 6, and the report says so.

One prediction was not specific enough. It said the method’s design would be within five points of the other on accuracy, or better, and never said whether an unanswered field counts as wrong. On needs-now the answer turns on it: 6.2 points ahead on the answers the design gave, 8.75 points behind when an unanswered field counts as wrong. Both are reported, and neither is final until labels to one wording exist.²

The same rule for Jev questions

A Jev question’s id is never sent to the model, so its instructions are the question. Rewording them makes a new version, and answers under two versions do not pool. Every answer our harness stores carries the spec version that produced it, and what each band of each answer means is written before the first call, the practice Triage Map describes, so a run can surprise the spec instead of being described by it afterwards.

What would falsify this

The claim is that the first-version urgency figures compare answers to different questions, and that one owner-chosen, versioned, frozen sentence removes the problem. Two results would falsify it. If people labelling the 80 under the first wording pick out the same “now” set as the stand-in’s narrower rule, the mismatch was real on paper and harmless in practice. If two people answering the second-version sentence disagree about as often as the first wording and the stand-in did, the sentence is not precise enough to serve as a key.