Call State Economics

Jev supports two calling patterns, and choosing between them is the first engineering decision in any integration. A parallel call answers several independent questions against one state in a single request. A serial call lets one answer produce new state for the next request. Most integrations default to serial calls out of habit — ask one thing, wait, ask the next — even when nothing about the questions depends on each other.

That default is measurably expensive. Splitting twenty independent questions across four sequential calls, instead of one parallel call, costs 2.6 times the money and four times the latency for identical answers, because every additional call re-sends the state and pays its fixed overhead again. A serial call is justified only when a later question needs evidence the current state does not contain yet.

The rule: batch by default, and chain only when a later question genuinely cannot be asked until code has produced new evidence. Applied across every decision point in a production agent, that single distinction is the difference between a decision layer that scales cheaply and one whose bill grows with call count instead of with work actually done.

Story tells it through a doctor's office. Technical has the invoice numbers and the one shape that still earns a second call.

My doctor's office lets you book a fifteen-minute visit or a thirty. Last time in, I had four things to ask about — a headache that wouldn't quit, whether last year's flu shot covers this year, a prescription refill, and a mole I'd been ignoring. None of the four needed the other three answered first, so I booked one visit and asked all four. The doctor answered all four in the same fifteen minutes she'd have spent on one.

That's obvious once you say it out loud, and it isn't how I built the thing I actually build for a living. When I started asking an AI model — Jev, in this case — questions about a business, my first instinct was to ask one thing, get the answer back, then open a brand new request for the next thing. It felt careful. Each question got its own clean slate.

It also meant booking four separate appointments for four things, none of which depended on each other. Every one of those appointments starts the same way: the receptionist re-reads your whole file before the doctor says hello. In Jev's terms, that's the business's name, its address, everything the model needs before it can answer anything at all — sent again, in full, every single time, even though nothing about the business changed between question one and question four.

I wanted to know what that actually costs rather than guess at it, so I ran the same twenty independent questions two ways over one afternoon. Once as a single visit, all twenty asked together. Then as four separate visits, five questions each, with the same background story repeated at the start of every one.

The single visit cost about three one-hundredths of a cent. The four separate visits, asking the exact same twenty things, cost more than two and a half times as much and took about four times as long, waiting on four separate front desks instead of one. The four-visit version didn't get a better answer — it got the identical twenty answers, slower, for more money, because most of what it was paying for on each visit was the receptionist re-reading the file rather than the doctor doing anything new.

So the four real questions about a lead — is this a franchise, can the person actually decide to spend money, how much does the business depend on people finding it online, what could they plausibly pay — go into one visit now. None of them needs to know what the others said. The model reads the file once and answers all four off it, together, the way my doctor did.

There's one kind of appointment that genuinely has to be a second visit, and it's worth being honest about which one. You don't ask "is this infection serious" and "what should we treat it with" in the same fifteen minutes if the second question needs a blood test the first visit had to order. You book the follow-up because there's real new information by the time it happens: the second question literally could not have been answered at the first appointment.

That's the whole difference between asking everything you already know at once and booking a second visit because a real fact showed up in between. The second kind earns its appointment. Most of what gets designed as a chain of questions is really just the same file, asked about four separate times, because that felt like the careful way to do it.

A fan-out and a cascade both look like "ask Jev a few things" from a distance, and the difference between them prices out very differently once you run both.

TypeSafe's own guidance is to batch: "Ask independent questions together" — whether a refund was requested, whether the evidence shows a duplicate charge, whether policy supports a refund, all in one call.¹ An independent handbook on Jev engineering practice — not a TypeSafe document, and its own title page says so² — draws the boundary more sharply: primitives "should not be chained inside the same request when a later question depends on new evidence created by an earlier answer," and sketches the shape of the case where they should:

STATE v7
  → decide: evidence missing
  → search: obtain official source
  → STATE v8
  → decide: evidence sufficient
  → route: fact_check
  → STATE v9

Every arrow that creates new evidence has to be visible somewhere code can see it, or the run reads as one continuous judgment when the graph actually contains a hidden state jump in the middle of it.²

Four questions, one request, in production

A production lead-scoring integration asks Jev four things about each business in a prospecting list — whether it's a franchise, whether one owner could sign off on a new website without a head office, how much new custom depends on the site, what the business could plausibly pay. None of the four needs to know how the others landed, so they travel in one ask_many call:

QUESTIONS = {
    "chain": {"type": "noul", "instructions": "..."},
    "decision_maker": {"type": "noul", "instructions": "..."},
    "web_dependence": {"type": "score", "instructions": "...", "criteria": [...]},
    "budget": {"type": "score", "instructions": "...", "criteria": [...]},
}
answers = client.ask_many(build_state(row), QUESTIONS)

The branching that depends on those answers — block a franchise above a 0.65 cutoff, let Jev veto a regex hit it's confident about, weight budget and web-dependence into a fit score — happens afterward, in a plain function, not as a follow-up question. apply_policy() reads the four stored numbers and computes a rank with no network call at all, which means a weight can change and the entire list can be re-ranked for free, because the judgments themselves never move again.

What the extra call actually costs

To replace instinct with a number, scripts/fanout_cost.py ran the same twenty independent questions against hosted Jev two ways: once as a single request, once split into four requests of five questions each, every one re-sending the same background state.³

WIDTH — one state, one call, N questions
    N   in_tok  out_tok   cost_usd
    1      432       24  0.0000181
    3      472       60  0.0000198
    6      527      117  0.0000221
   12      629      229  0.0000264
   20      768      381  0.0000323

DEPTH — the same 20 questions, split across sequential calls
 calls   in_tok   cost_usd      ms
    1      768   0.0000323     247
    2     1181   0.0000496     423
    4     2007   0.0000843     996

Going from 1 question to 20 in one call adds about 18 input tokens per extra question, because the roughly 410 tokens of background state get paid once no matter how many questions ride along with it. Splitting those same 20 questions into 4 calls instead of 1 costs 2.6 times the input tokens and 4 times the wall clock, for the identical twenty answers — because every one of the four calls pays that ~410-token state cost over again. The script's own comment on the number: "That multiple IS the price of a tree stage."³

In absolute dollars the whole comparison is tiny — $0.0000323 against $0.0000843, a difference of about five thousandths of a cent — which is exactly why the expensive shape is easy to build by accident. Nobody notices a five-thousandths-of-a-cent mistake once. Repeated across every tool call in every agent run, on a machine that keeps twenty agent panes open at a time, it stops being a rounding error.

What would falsify this

The rule — bundle by default, pay for a new call only once new evidence has to exist first — holds as long as the only thing a later question needs from an earlier one is a fact code can compute and hand forward. That's true here: a Decision carries a typed value, a probability distribution and a confidence, never a reasoning trace, so there's nothing to feed a second question except what code already extracted from the first answer. If a real case ever needed the model's own unstructured reasoning to shape the next question — not a fact code could read off the first answer, the reasoning itself — bundling would stop being the free choice it is here and a genuine chain would become unavoidable. Nothing built in this repo has hit that wall yet.