Triage Map

A triage frame worth trusting is written in two halves: the questions Jev answers, and, before any answer exists, what each answer means and how often it should come back. With those expectations on paper, a run over real issues stops being a pile of plausible numbers and becomes a test of the design.

We built one for Omarchy’s GitHub queue: 1,964 open issues, each read by Jev along several narrow dimensions (what kind of item it is, whether someone could act on it as written, how bad and how widespread the problem is) and sorted by plain code into piles: look now, ready for an agent, the fix belongs upstream, ask the reporter, park. The first run, over 200 issues, broke six of our predictions. Reading the issues behind them showed that most reports in this queue name their own cause, so the design needed a question about whether that cause is shown or only claimed, and a pile for fixes that belong to other projects. The whole queue now sits on one map, for 47 cents of Jev calls.

Story walks the map as a mail room. Technical has the dimensions, the six broken predictions, and what the map still gets wrong.

Picture a mail room at the start of a shift: two thousand letters in a heap, a row of bins along the wall marked urgent, can wait, wrong building and needs a reply first, and a sorter whose job is to get every letter into the right bin before the day starts.

Our heap was Omarchy’s GitHub issues, 1,964 of them open on the morning we started. Omarchy is an open-source Linux desktop, and its issue queue gets everything: bug reports, feature requests, people stuck on an install, the occasional message that has nothing to do with the project. The sorter was Jev, a model that answers narrow questions about exactly the facts it is handed.

The obvious way to run a mail room is to ask the sorter one question per letter: does this go in the urgent bin? You get an answer every time, and it always sounds reasonable. You also get no idea why. When the urgent bin overflows, there is nothing to adjust except the sorter’s mood.

So we changed the job. Instead of one big question, the sorter answers several small ones about every letter: what kind of letter is it, can someone act on it as written, how much breaks if it is ignored, how many people it affects. A written rule on the wall, which anyone can read and change, turns those answers into a bin. The sorter judges, and the rule decides.

Then we did one more thing before starting. We wrote down how full each bin should get. Not exactly, but in ranges: maybe a quarter to four-fifths of the letters would describe a problem someone could reproduce; only a sliver would be about another building entirely.

We ran the first two hundred letters, and six of those guesses were wrong.

That sounds like failure. It turned out to be where the design got better. The largest miss was on the scale for how well a letter describes its problem, where nearly four letters in five landed at the very top, the level that means the writer has named the fix. When we opened those letters, the reason was plain. Much of this queue reads as written with the help of AI tools, and those tools almost always name a cause. A scale that nearly every letter tops out on no longer tells letters apart. So the sorter got a new question: did the writer show the cause, or only claim it? About one letter in six turned out to claim a diagnosis without supporting it, and those are the last ones to hand to a helper as a starting point.

The second surprise came from a check we had written for contradictions: a letter that names its fix, yet whose fix looks enormous. It fired eight times, and six of those letters were really addressed to other buildings. The problem sat in the kernel, a driver, or a program Omarchy only packages. So the room got a new bin, for letters to forward upstream, and 213 letters went into it.

By the end of the day the whole heap was sorted, every letter in a bin or marked as one the rule could not place, and the room looked like a map: each letter a dot, grouped by the part of the project it concerns, coloured by its bin. It cost 47 cents.

The map is not finished. The urgent bin holds about six hundred letters, far more than anyone can read in a week, and where that line sits is a person’s call, not the sorter’s. And nobody has yet checked the sorting against people who know the project, which is why eighty letters are on a public page for anyone to label. The written guesses told us where the rules were wrong. Only people can tell us whether the right letters ended up in the right bins.

“Should a maintainer look at this issue now?” asked of Jev once per issue returns a probability that looks decisive and carries the whole decision inside it: impact, reach, whether anyone could act on the report, whether the bug is even the project’s to fix, all weighted where nobody can see or change the weights.

The design

The frames instead ask narrow questions over a state built from each issue: its kind, whether a maintainer could act on it as written, the part of Omarchy it concerns, whether it might be a security problem, how far its damage reaches, how large and whose the fix is, how well the report supports the cause it names, and, for defects only, whether it is a regression, how bad it is and whether there is a workaround. Code adds what it can count exactly, such as how many files in Omarchy’s pinned source tree mention the command an issue names. A queue rule in code combines the stored answers, so changing the rule costs no Jev call.

The questions travel in about three calls per issue: two read every issue (one identifies it, one assesses it), a third adds defect detail and runs only on bugs, and a fourth reads the comments and runs only when the kind of issue is contested. For every question the spec writes down, before any call, what each band of the answer means, what the system does there, and what share of real issues should land in it. Jev reports probabilities rounded to the hundredth, so band edges sit on half-hundredths; a band ending at exactly 0.05 would miss an answer of 0.05.¹

What the first run broke

Version 2 of the design ran over 200 issues, and six expectations failed.² The largest: the reproducibility scale was expected to put between a quarter and four-fifths of issues at “reproducible”, and 79% landed on the level above it, “points at the fix”. A level nearly every issue reaches no longer discriminates. Version 3 stops that scale at reproducible and asks a separate question about how well the report supports the cause it names. Over the full queue, 16% of assessed issues offer no diagnosis or only assert one (6.1% and 9.8% of 1,954).³

An invariant written to catch contradictions made the second change. It flags an issue that names its fix yet whose fix reads as sprawling, and it fired 8 times in the 200. On reading them, six were kernel, driver or packaged-application problems, one was a large feature request, and one concerned a boot entry whose ownership was not checked.² The fix-scope question had merged “a redesign” with “belongs to another project”. Split in two, the second became its own pile: 213 issues whose fix belongs upstream.³

The third change was wording. Lock-screen failures sat mid-scale on the security question, because the question never said whether a lock that fails open counts.

The map

Version 3 ran over all 1,964 open issues in 193 seconds for $0.47, about 24 cents per thousand issues.³

Pile	Issues
Park	742
Look now	605
Ready for an agent	333
Fix belongs upstream	213
Not a work item	10
Ask the reporter	7
No pile: a reading the rule needs was unmeasured	54

81 issues were routed to a person, mostly possible security reports. The 54 without a pile are drawn grey on the map rather than guessed into one. Look now holds 31% of the queue, far more than a maintainer reads in a week; the line is code over stored answers, so moving it costs nothing to try, and where it sits is a maintainer’s decision. Ours is about ten.

Repeat answers are steady: over seventeen fixtures asked three times each, every question’s standard deviation stayed under a hundredth. Across three identical runs of the 80 reserved issues, 5 changed a reading, and the changes came from answers sitting near a band edge or a cut-off in the combination rule.⁴

Where it is still wrong

Every run grades the design against its own expectations. Ten adversarial issues grade it against something else: spam, a non-English regression, a support question written as a bug, a security report, a duplicate pair and five more, each with its correct handling fixed before any design existed. The frozen design handles seven. Two of its three misses are cases it lists as out of scope, a stale report and a duplicate pair; the third is the support question written as a bug, which it sends to look now with confidence, because nothing in the state says the feature exists.⁴

Accuracy is the open measure. Eighty issues drawn at random are on a public labelling page, and Omarchy’s community has been asked to help label them.

What would falsify this

The claim is that expectations written before the run made the run test the design. The evidence so far is that version 3’s new expectations were tested on 200 issues the previous run never saw, and every one held except one rare call that never fired; over the full queue, every expectation held. The claim would be falsified if, once people’s labels exist, a single question asked of each issue matches them as well as the map does, since then the dimensions and their written expectations bought structure without correctness.