Decision Receipt

Every Jev API call should produce a decision receipt — a record linking the judgment returned to the state and question that produced it. Without that record, cost and accuracy claims about a decision layer are unverifiable, including the claims the system makes about itself.

That's not a hypothetical risk. An evaluation harness measuring Jev's own accuracy printed a cost of $1.44 for one benchmark run. The actual invoiced cost, pulled from the provider's own usage data for the same set of decisions, was $0.0011 — roughly 1,300 times smaller. The harness had conflated a budget reservation, held before a request completes, with the metered charge recorded after it. The discipline used to grade the model's calibration had not been applied to the harness's own printed numbers.

Decision receipts exist precisely to make this kind of error checkable rather than assumed. A harness that grades a model's confidence against outcomes, and never grades its own numbers against the actual invoice, is applying a standard to the model that it doesn't apply to itself — and every unverified number it produces afterward, including accuracy figures, inherits that same risk.

Story tells it through a stack of gas receipts. Technical has both retractions, the real accuracy numbers, and the two ground truths that didn't survive contact.

On a long road trip, it's easy to feel like you know roughly what you're spending. Gas here, food there, a hotel one night — by day three you've got a number in your head, and it feels solid because you were there for all of it.

Then you get home and go through the actual receipts, and the number in your head was wrong, sometimes by a lot, and not evenly. It's wrong more in the direction you were already leaning. If you started the trip worried about money, your gut estimate probably ran high. If you stopped worrying somewhere around day two, it probably ran low. Either way, the feeling wasn't tracking the spending — it was tracking the story you were already telling yourself, and the actual paper in the glovebox is the only thing that can correct it.

I was building something closer to the felt number than the receipt, and I didn't know it until I checked. I'd set up a way to test how good an AI model's judgment actually is — feed it the same decisions over and over, see how often it lands close to right, and print a bill alongside the score so I'd know what all that testing cost. The bill it printed said $1.44 for one run.

That number went straight into a report. It felt like exactly the kind of fact you don't need to double-check — the tool is printing its own receipt, right there in the output. Except a bill printed by the tool that made the purchase is still just a number the tool believes about itself, not a receipt from whoever actually got paid. I went and pulled the real one, straight from the billing provider's own account. $0.001127. Not fourteen percent less. About thirteen hundred times less.

What had actually happened was a mix-up between two numbers that sound like the same thing: what a service reserves against your account the moment it starts working, and what it actually charges once the work is done and the real cost is known. My tool had been printing the reservation and calling it the bill. The direction of the error is the part worth sitting with — it made the option I already doubted look expensive, which matched exactly what I already believed going in. A wrong number that happens to agree with you is the one you're least likely to go check.

The whole reason I'd built this thing was to catch a model being confidently wrong about something and not knowing it. It had just done the identical thing about its own invoice, and the only reason I caught it was the same habit that's supposed to catch everything else: don't trust the number the machine hands you about itself. Go find the actual receipt.

An evaluation harness exists to catch a model being confidently wrong about something it hasn't actually verified. Turning that same suspicion on the harness itself is the part that's easy to skip, because the harness is supposed to be the thing doing the catching.

The bill that was 1,300 times too high

A JevBench-style accuracy run against hosted Jev printed charged=$1.4400 in its own summary line. The real figure, pulled from the billing provider's usage numbers for the same 72 decisions, was $0.001127 — about 1,300 times smaller.¹ The gap wasn't a rounding error; it was two different numbers that share a name in casual conversation. What the harness printed was a reservation — a hold an API places against a budget the moment a request starts, before it knows what the request actually costs. What the invoice records is spend, computed after the fact from the tokens the model actually used. Nothing in the harness distinguished the two, and the error ran in the one direction that would go unnoticed: it made the option already under suspicion look expensive, which matched the working hypothesis going in.

At the actual rate — roughly $1.69/day for about 1.25 requests per second across twenty concurrent agents — cost stopped being the argument for keeping a decision model running locally instead of hosted. Privacy, offline operation and not sending data to a third party remained real arguments. The invoice did not.¹

Two retractions, kept rather than silently fixed

The same evaluation cycle produced two claims that had to be walked back once more evidence arrived, and the record keeps both rather than shipping only the corrected version.

"The prompt must be broken." An earlier finding explained two local decision models both landing at chance as a prompt problem, because they collapsed onto opposite constant answers. Hosted Jev read the identical prompt and scored 0.758. The prompt was fine; the local models were weak at the task.¹

"Both decision models sit at chance." True only of the local ones. Generalizing it to decision models as a category was wrong the moment a third model, evaluated on the same harness, didn't.¹

An independent handbook on Jev engineering practice names the reason this matters past one repo: "average decision accuracy is insufficient," and per-call price "is only one component of completed-task economics" — decision latency, calibration and recovery cost all belong in the same accounting, or a cheap wrong decision ends up costing more than an expensive correct one, downstream where nobody's counting it against the original call.² The measured numbers, once the reservation bug was found and both retractions made:

	JevBench	ECE	Brier	p50 latency
hosted Jev	71/72 (98.6%)	0.046	0.027	197 ms
Open-Jev 2B, local GPU	56/72 (77.8%)	0.061	0.327	173 ms
von 395M, local	47/72 (65.3%)	0.198	0.563	22 ms

71/72 is the same figure TypeSafe has published elsewhere — the second published number this harness independently reproduced, which validates the harness at least as much as it validates the model.¹

The receipt that stopped a bad ground truth from shipping

record.py, the layer every call in this repo writes through, keeps one JSONL line per decision and a deliberate absence rule: a field the backend didn't report is left out of the row entirely, never written as zero or null.

# Only write what exists. `None` stays out of the row entirely, so a reader
# can tell "not reported" from "reported as zero" without a schema to consult.
for key, val in (("value", d.value), ("confidence", d.confidence),
                 ("probabilities", d.probabilities), ("latency_ms", d.latency_ms),
                 ("reason", d.reason)):
    if val is not None:
        row[key] = val

That discipline is what a decision receipt is for in the handbook's own terms: it links a judgment to the state and policy that produced it, so a later audit can tell a model that was uncertain apart from a model that was never asked.² It's also what caught two failed attempts at grading Jev against a real task, before either shipped as a number anyone would go on to quote.

Attempt one scored decisions on whether a finding appeared in more than one project, on the theory that duplication meant it was generated rather than observed. It was discarded before use: one such finding was a real, shared bug — "Fix Safari camera AbortError after 4+ recordings" — appearing in four projects because the bug itself is shared across them, not because an agent invented it four times.¹

Attempt two scored decisions by their source field: commit <sha> read as real, pattern (…) read as boilerplate. It shipped, scored a 40-line classifier at 0.800 against von's 0.500 and the 2B model's 0.525, and was then invalidated after checking the items behind the label by hand: "all of the to-do list items are real and were generated from probably a tie-off or something similar." The source field records how an agent noticed something, not whether the thing it noticed is true — an agent observing that a project genuinely lacks error handling is a true observation regardless of which pass surfaced it. Provenance had been read as validity.¹

Both failures share one shape, worth stating as the actual test for the next ground truth: could this label be measuring how the record was created rather than what's true? If the answer isn't a clean no, it isn't a ground truth yet.¹

The same argument that keeps agent session transcripts

A decision receipt is the same kind of artifact as a coding agent's full session transcript, and the case for keeping either is identical: neither is valuable for what it does in the moment. A transcript isn't read while the agent is working. It becomes valuable afterward, as raw material for noticing which patterns recur often enough to justify turning into a reusable skill, and which single session marked a genuine inflection point in a project rather than routine work. Discard the transcript and that signal is gone before anyone knew to look for it — context engineering downstream of a project depends on the record existing upstream of the question.

A decision receipt earns its keep the same way. In the moment, only the returned value matters to the code that acted on it. Months later, a corpus of receipts is what lets someone ask which contract version actually degraded, which state fields correlate with low confidence, or which class of question this system should stop asking Jev and route to a human instead — questions nobody could pose on the day a single decision was made, and none of them answerable if only the winning value was kept while the state, the distribution and the contract version were thrown away.

What two independent systems agreeing against a label is worth

The clearest evidence that the source-based ground truth was actually broken, rather than merely questionable, came from outside the argument entirely. Hosted Jev and the 40-line classifier — two systems sharing no architecture, no training data and no author — independently disagreed with that label on 12 of 120 sampled items, and agreed with each other. All twelve turned out to be label errors: "Document Safari camera recording limitations" marked as templated boilerplate; "Code cleanup and remove unneeded components" marked as a real finding, backwards from what it should have been.¹ That agreement caps every accuracy number the label produced — the classifier's four-point lead over hosted Jev may be measuring nothing at all, since both were partly graded against a target now known to be wrong in the same twelve places.

What would falsify this

There is still no valid ground truth for whether Jev's answers on the real triage task this repo cares about are actually correct, and the honest move is to say that plainly rather than round up to something that sounds finished. Every number in the table above measures calibration and agreement with a published benchmark, not correctness on this project's own decisions. The two options left are hand-labelling a few hundred items by someone willing to read them, or naming a proxy explicitly for what it's a proxy for, rather than letting it stand in as a fact without saying so. A future version of this article that reports an accuracy figure on the real task without saying which of those two it used should be read the same way the $1.44 line was: plausible, unverified, and worth pulling the actual receipt on before it gets quoted anywhere else.