Human Labels

One of our Jev systems sorts GitHub issues: what kind of issue is it, could someone act on it, does a maintainer need to see it now. To check it against a person, someone has to sort the same issues by hand. Each hand-sorted answer is a label, like a tag a person ties to the issue, and the system is scored on how often its answers match the tags.

We planned for 80 of Omarchy’s issues, labelled on a public page, without seeing any system’s answers, by me and anyone who knows the project. I asked on the project’s discussion board; no one has replied, and my own pass is still to come. An AI model labelled all 80 as a stand-in, so every accuracy number in that comparison measures agreement with a model, from the same model family that built the systems.

The next test is smaller, in a subject I know: short passages of prose, which I’ll label on whether each sounds machine-written, which pattern it is, and what I’d do about it. I’ll label some twice, days apart; how often I agree with myself is the most any system can be expected to match me.

Story walks through it with a kitchen and its soup. Technical has the key as designed, what arrived instead, and the next test’s three axes.

One of the systems we build with Jev sorts the issues people file against Omarchy, an open-source Linux desktop. For each issue it answers three questions: what kind of issue is it, could someone act on it as written, and should a maintainer look at it now.

To find out whether it answers them the way a person would, a person has to answer the same questions about the same issues, by hand. Each hand-made answer is a label, like a tag a person ties to the issue, and the system is scored on how often its answers match the tags. A kitchen tests a new soup the same way, by asking the diners rather than the chef.

We drew 80 issues at random and put them on a page anyone could use, with no system’s answers showing. I asked on the project’s discussion board for people who know Omarchy to label a few, and no one has replied yet. My own pass is still to come.

An AI model labelled all 80, with a short note on each, so the systems could be scored against something until people’s labels arrive (stand-in labels). We have four versions of the system, built different ways, and three of them agree with the model about the kind of issue 77 times in 80. That is the chef tasting a soup cooked by people trained in the same kitchen: when the two agree, nobody can tell whether the soup is good or they share a habit. All four were drafted by AI models of the same family as the stand-in.

The page keeps the stand-in away from the diners’ verdicts when they come. A person sees the model’s answers only after giving at least one answer of their own, and anything labelled after that is marked and counted separately. Where two people taste the same bowl, how often they agree is reported next to every version’s score, because no system can be expected to agree with people more often than people agree with each other.

Nearly two thousand issues from a project I use but don’t maintain is a lot of soup, and judging whether each one deserves a maintainer this week takes a maintainer’s palate. So the next tasting is a dish I know. I write a lot, and since the summer I’ve kept a list of the phrases I flag in machine-drafted prose, each with my own words for why. The next test asks new systems to read a short passage and say what I’d say about it: would I flag it, which pattern is it, and what would I do about it. The passages will be new ones, since every agent on my machine is told to read that list before it writes, so the list itself can’t be kept from them.

I’ll label about a hundred passages blind, then label twenty of them again a few days later without looking at my first answers. How often I agree with myself is the most any system can be expected to match me, and every system gets read against that. A system that agrees with me more often than I agree with myself is matching something other than my taste, and I’d want to know what.

The triage comparison was built to score each design against people’s labels on 80 of Omarchy’s issues. No people’s labels arrived, so every accuracy figure in it is agreement with a model’s labels, whatever the table’s column says.

The key as designed

The comparison was pre-registered with people’s labels as its accuracy measure. I would label 80 of Omarchy’s open issues, drawn by stratified sampling on label, age and body length with a fixed seed, on three arm-neutral fields: kind (bug, feature, support, docs, other), actionable as written, and needs a maintainer now. Each design’s outputs are mapped onto those fields by an adapter written before its answers exist.¹

The labelling page is public and shows no design’s output.² On 2026-09-24 I asked on Omarchy’s discussion board for people who know the project to label any number of the 80.³ The protocol says how their labels would count: scored separately per labeller and pooled, never merged into mine without saying so, and wherever two people labelled the same issue, their agreement reported beside the designs’ accuracy, since no design can be expected to agree with a rubric more often than people agree with each other.

What arrived

The call for labels had no replies when this was written. I have not done my own pass. Claude Opus 5.5 labelled all 80 with a note on each, and I made those labels the backstop key until people’s exist.⁴ Against them:⁵

Agreement with the model’s labels	Without the method	First method	Reference	Revised method
Kind, of 80	77	77	75	77
Actionable, of answered	58/73	58/66	63/80	63/80
Needs a maintainer now, of answered	61/78	54/64	60/78	53/79

These are a model grading designs drafted by models from the same family. The protocol expected agreement to run high for that reason, and nothing yet measures how high. The needs-now row also compares answers to two different questions, which Frozen Questions covers.

Keeping the stand-in apart

The page carries the model’s 80 answers, hidden until a labeller has copied or exported at least one label of their own. After that, every label they give carries a seen=model mark and is scored apart from labels given blind. The repository’s README prints the model’s answers to two issues as a worked example, so any community label on those two counts as seen, whatever its mark says. Community labels are pooled by provenance: a person, an agent a person checked, an agent, taken from the page, or undeclared.¹

What this buys is narrow: the stand-in cannot leak into people’s labels when they arrive, and every figure against it stays labelled as agreement with a model.

Why it stalled

The problem was too big, and too far from what I know. The queue held 1,964 open issues in a project I use but don’t maintain. The rest of the comparison had the same trouble: one designer per condition, a held-out trap set so easy that every earlier design handled its novel half, and adapter choices sitting between every design and its score. I shelved the comparison and moved the next test to something much smaller.⁶ Labelling is a judgment, and the judgment has to be one the labeller already makes.

The next key

The next comparison asks a design to read one passage of prose with its register (an article, a brief, an email to someone I know, cold outreach, a cover letter, app copy) and say what I would say about it. Every axis is one I can label from the passage alone:⁶

Axis	Values	The question I answer
Flag	yes · no · can’t tell	Would you flag this passage as an AI-ism?
Pattern	one of the named patterns in my ledger · other · none	Which pattern is it?
Fix	must fix · worth fixing · leave it	What would you do with it?

• The three questions are written in their final words and versioned before anyone labels.
• My ledger of flagged patterns cannot be held out, because every agent on my machine is told to read it before writing, so evaluation passages are new: agent-written drafts I have not reviewed, plus some of my own writing as a control against flagging my own voice.
• A tracer of about ten passages comes first, to fix the wording and time a label, then about a hundred.
• I am the only labeller, so the ceiling is my agreement with myself: I re-label twenty passages a few days later, blind to my first answers.
• A passage I never flagged counts as unlabelled, because nobody has said it is clean.
• At least three independently started designers per condition, with and without the method.

It has not started. The protocol, the sealed traps and the baselines come first, and the designers start only after I approve them.

What would falsify this

Two claims. The first is that the comparison’s accuracy figures measure agreement with a model and cannot stand in for people. If my labels on the 80, when they exist, rank the designs the same way and within a few points of the model’s labels, the stand-in was adequate, and the claim is falsified for this problem. The second is that a small set in a domain I know, labelled by me, yields a usable ceiling. If my agreement with myself on the re-labelled twenty is too low to tell any two designs apart, a familiar domain did not give me a key either.