Frame Confidence

Jev returns a confidence with every Choice and Score answer, and TypeSafe defines it precisely: a statistic of the answer’s own probability distribution. It says how settled the answer is inside the frame you built, the state and the questions. It cannot say whether that frame says what you meant, because the frame is all Jev sees.

We measured the gap on hosted Jev. A synthetic report claiming a working feature was broken came back a bug at confidence 0.98; with one sentence of documentation saying the feature exists, still a bug, at 0.77. A frustration rubric with one level moved out of order shifted a score by a whole level and raised confidence to 1.0.

So an answer carries two confidences. Jev supplies the first. The second, whether the measurement was taken the way the engineer intended, is the engineer’s to establish, with meanings written before the call, traps with known answers, and labels from people. And the frame carries its author: in the framing this series uses, a conservative engineer writes direct questions that come back confident and miss the edge case; a cowboy writes looser ones that find the edges and blur the ordinary.

Story tells it with a GPS. Technical has the measurements, the two kinds of engineer, and what would prove this wrong.

A GPS will tell you to turn left in four hundred metres with complete assurance. It is working from its map and the address you typed, and from those two things the route is settled. When two routes come out close, it says that too, and offers both.

That assurance is about the route. It says nothing about whether you typed the right address, or whether a bridge on the route closed last week and nobody told the map. A GPS will drive you, with perfect confidence, to the wrong house.

Jev, the decision model this series is about, reports a confidence with many of its answers, a number between 0 and 1 (its confidence). TypeSafe, the company behind it, is clear about where the number comes from: it is worked out from the answer itself. When one option clearly beats the others, confidence is high. When the options run close, it is low. It is the route’s confidence.

What an engineer wants to know is usually something else: the trip’s confidence. Did the question get asked the way I meant it? Did the facts I handed over contain what the answer should turn on? Jev cannot see any of that, because the address and the map are all it gets (the frame: the state it sees, times the questions it is asked).

We tested the difference three ways.

First, the closed bridge. We handed Jev an issue report from someone who says a desktop’s wallpaper feature is broken, when in fact the feature exists and they had not found it. Asked what kind of report it was, Jev said a bug, with confidence 0.98, as sure as it gets. Nothing on its map said the feature existed. When we added one sentence saying so, it still said bug, but its confidence fell to about 0.77. The more confident answer came from the frame that knew less.

Second, a typo in the address. We asked how frustrated a customer sounded, on a scale from calm to furious, and then asked again with one level of the scale moved out of place, the way a typing slip might move it. For one message, Jev picked the same word, “frustrated”, but the slip moved it a whole level up the scale, so any code reading the number would have treated the customer as angrier than they were. Its confidence went up, to a perfect 1.0. For a stronger message, the slip left the answer split between two far-apart levels, and confidence collapsed to about 0.1. The same kind of mistake got opposite reactions, and neither told us a mistake had been made.

Third, asking again. We asked the same routing questions five times each, and the answers matched to within about a hundredth every time. Asking a GPS twice tells you it is consistent. It does not tell you the address was right.

So where does the trip’s confidence come from? From the driver. Before leaving, write down what arriving should look like, so a strange result stands out. Plan for the closures you can imagine, and check the route against them. And when you get there, look at the house number: have someone who knows the neighbourhood say whether you arrived where you meant to.

Drivers come in two kinds, and so do the engineers who write frames. The careful one types exact addresses for trips they know well, and every day goes smoothly until a road they never thought about closes. The adventurous one types “somewhere by the water” and finds places the careful driver never sees, and some mornings the ordinary commute takes twice as long. Conservative engineers write direct, concrete questions that nail the ordinary case and miss the strange one. Cowboy engineers write looser questions that catch the strange case and blur the ordinary. Both kinds of frame come back with a confidence, and the number cannot tell you which kind of driver was at the wheel.

Routing on confidence, acting when it is high, reviewing when it is middling, stopping when it is low, is the pattern TypeSafe teaches and the one most integrations reach for first. It assumes a high confidence means a right answer. The number is defined more narrowly than that.

What the number is

TypeSafe’s documentation: “confidence is a statistic computed from the probability distribution the answer already gives you,” and for both Choice and Score, “a flatter distribution means lower confidence.” Noul answers carry none.¹ Reading those pages through on a recording for this work, I asked the question this article is about: “is it confident in its measurement, or is it confident in the choice that it’s making?”, with a Choice split 0.44 against 0.43 as the example. The docs answer it plainly. The number is confidence in the outcome, derived from the shape.²

Measured, the shape tracks as documented. Four routing messages, each asked five times of one Choice (billing, technical or sales), including three written to sit between two teams:³

Message	Top two	Confidence
payouts failing for three days	billing 0.99 · technical 0.01	0.99
card declined on a plan upgrade	billing 0.97 · sales 0.03	0.93–0.95
payouts stopped after a bank change; settings page errors	billing 0.90 · technical 0.10	0.83–0.87
charged twice; app crashes on the billing page	billing 0.77 · technical 0.23	0.62–0.68

None of the three produced a tie; the closest split, 0.77 against 0.23, came back at about 0.65. The largest standard deviation of any probability across five repeats was 0.012, so repeating a question measures jitter, which is small, and says nothing about whether the question was the right one.

What it cannot see

A missing fact. The state held a synthetic issue report whose writer says the wallpaper feature is broken; the correct handling, written down before any design existed, is support, because the feature exists. Five repeats per state:

State	bug	support	Confidence
the report alone	0.98	0.01	0.97–0.98
plus one sentence: the feature exists, bound to a shortcut and a menu entry	0.79–0.85	0.14–0.20	0.73–0.80

The more confident of the two came from the state that lacked the deciding fact. Adding the fact lowered confidence without changing the pick: the kind question still let the reporter’s claim decide, so grounding needs a question that points at the fact as well as a state that contains it.

A misordered scale. A frustration Score with levels calm, frustrated, very angry, furious, and the same four with furious moved from last to second. Three repeats per cell:

Message	In order: score · confidence	Misordered: score · confidence
mild	0.01 · 0.99	0–0.01 · 0.99–1
medium	1.03–1.04 · 0.96–0.97	2.00 · 1.00
strong	2.77–2.83 · 0.77–0.83	2.06–2.13 · 0.06–0.13

On the medium message Jev picked “frustrated” both times, correctly, but on the misordered scale that word sits at position 2, so the score code reads moved a whole level, to what the engineer’s own ordering calls very angry, and confidence rose to 1.00. On the strong message the probability split between furious (now position 1) and very angry (position 3), and confidence collapsed. The same fault in the frame raised confidence once and destroyed it once. Confidence follows the shape of the distribution, and the shape cannot know what the engineer intended the positions to mean.

Where the second confidence comes from

Fit to intention has to be measured outside the answer, and in this work it came from three places:

• Meanings written before the call. Every band of every question was given a meaning and an expected share of real issues before the first run. Six expectations broke on the first 200 issues, and reading the issues behind them exposed faults in the frame: a scale nearly every issue topped out on, two ideas merged into one question, a question that never said what counted. Triage Map covers them.
• Traps with known handling. Ten adversarial issues, their correct handling committed before any design. All three triage designs sent the wallpaper report through as a confident bug.⁴
• Labels from people. Eighty issues are on a public labelling page for Omarchy’s community. Until enough arrive, a model’s labels stand in, and every figure scored against them is reported as agreement with a model.

An independent field guide on pairing Jev with LLMs, which circulated the same day, gives the same advice about thresholds: “A threshold copied from an example is an initial placeholder, not evidence that the system is calibrated for your domain.”⁵

Conservative and cowboy

The frame records who wrote it. On the same recording I put it this way: “some people are a little more conservative, maybe they’re less edge-casey, they’re going to ask questions that are more direct and concrete,” while an engineer who is “more of a cowboy, more edge-casey” will “ask questions that are looser.”² The conservative frame comes back confident and repeatable on the ordinary item and is blind to the case its author never imagined. The cowboy frame reaches the edges and pays with ordinary items that spread.

The evidence in hand is one comparison. In the triage test, the design whose author wrote the ten traps before anything else handled seven, including the only security report any design caught. Two designs written without that step handled six each; the steadier of them changed a reading on 3.75% of issues across repeat runs, against 6.25% for the traps-first design.⁴ That fits the framing, and it is one test, not a law.

What would falsify this

The claim is that Jev’s confidence cannot detect a frame that misstates its engineer’s intent. It would be falsified if frames with a known fault of that kind, a missing deciding fact or a misordered scale, reliably came back with lower confidence than their corrected versions. In these probes, two of the three faulty frames came back more confident than their corrected versions, and one less. Small samples, synthetic states and one model version: a larger set could still show a usable signal, and that is the test to run next.