All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Operations · 14

Cost engineering: per-run cost, budget anomalies and model routing

Token counts are telemetry. Cost per run, per step and per tenant, the run that loops and spends ten times the usual, and what routing to a smaller model changes.

Tokens are telemetry
Every model span already records tokens in and tokens out. Cost is that count times a rate.
The run that loops
One run of the simulated day costs 21 times the median, and nothing else about it looks wrong.
Routing, measured
Sending some steps to a smaller model changes the day's total. The records show which steps, and what it saved.

On the simulated day the underwriting agent handles 48 runs. The SLO dashboard of the SLOs and alerts page is green outside the provider outage. The error rate is low. The underwriters have no complaints about the recommendations. The model recommends; a person decides, and every run ended in a controlled state.

At the end of the month the platform engineer reads the model bill, and it is far above the estimate. Nobody can say which runs spent it. The dashboards count requests and errors; none of them counts what a run cost. The answer was in the records all along: every model span carries the number of tokens it read and wrote. Nobody added them up.

This page treats cost as one more signal. It starts from the token counts on a span, turns them into the cost of a step and of a run, attributes the cost to the broker who caused it, finds the run that spent far more than it should, and shows what routing some steps to a smaller model changes. One note before any number appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is. And one more, binding on this page: every cost here is in invented units, called cost units (cu). They are not a price and say nothing about any vendor's price.

1. From a token count to a cost#

Start with the smallest piece. A token is the unit a model reads and writes: a word or part of a word. A model call reads some tokens (the prompt, the retrieved passages, the conversation so far) and writes some tokens (the answer). The model span of GenAI telemetry records both numbers. For the last model step of case UW-006, the worker's log line says it:

2026-09-14 14:07:19 INFO  [agent-worker] write_recommendation finished: 723 tokens in, 111 out (large-model)

A rate turns the counts into cost: so many cost units per thousand tokens read, and a higher number per thousand tokens written. The example invents two model tiers with generic names and two rates each.

TierRead (cu per 1,000 tokens)Written (cu per 1,000 tokens)
small-model1.04.0
large-model10.040.0

These rates are invented for teaching. The shape is common (writing costs more than reading; a larger model costs more than a smaller one), and the numbers are not anyone's.

The cost of one step is one line of arithmetic. This is the function in cost.py:

def step_cost(tokens_in: int, tokens_out: int, model: str, rates: dict[str, dict[str, float]]) -> float:
    return round(tokens_in / 1000 * rates[model]["in"] + tokens_out / 1000 * rates[model]["out"], 6)

For write_recommendation in UW-006: 723 tokens read at 10.0 per thousand is 7.23 cu, and 111 tokens written at 40.0 per thousand is 4.44 cu, so the step costs 11.67 cu. The run had one other model step, plan_steps on the small tier, at 0.826 cu. The per-run cost of run-7f3a is the sum: 12.496 cu.

Plain English. A run's cost is its token counts multiplied by the rates, step by step, then added. If your spans already record tokens and the model tier, you already have everything you need. What is missing is the sum.
Common confusion. The two failed calls to get_roof_assessment in UW-006 cost no tokens: they are tool calls, not model calls. They cost time, not cost units. A run can be slow and cheap, or fast and expensive; the two signals answer different questions.

2. The day, run by run#

Now apply the same arithmetic to every run of the simulated day. The command prints the median run, the threshold for an anomaly, the outlier, the total under different routing rules, and the cost per broker.

cd "$TMPDIR/uw-observe-cost" && /opt/homebrew/bin/python3 run_observe.py --show cost
== cost
rates (invented cost units per 1,000 tokens): {'small-model': {'in': 1.0, 'out': 4.0}, 'large-model': {'in': 10.0, 'out': 40.0}}
median run 12.4335 cu · anomaly threshold 37.3005 cu (3.0 x median)
anomaly run-48f5 UW-502: 265.6070 cu = 21.4 x median · 26485 tokens in, 205 out, 10 model calls
day total by routing rule: as-run 851.1230, all-large 1210.0700, all-small 121.0070, small-except-write 612.8210
by tenant: broker-01 60.3460, broker-02 123.4220, broker-03 74.5920, broker-04 63.0130, broker-05 113.3480, broker-06 416.4020

Read the lines one at a time. The median run costs about 12.4 cu, close to UW-006's 12.496. One run, run-48f5, costs 265.6 cu: 21.4 times the median. Its case is UW-502, this site's own case of an agent that loops. And one broker, broker-06, accounts for 416 cu of the day's 851. All of these numbers are synthetic and illustrative.

Diagram: From a model span to a cost per broker, in invented units

Read left to right. A model span records tokens in and out and the tier. A rate turns that into the cost of one step. The steps of one run add up to the run's cost. Runs carry the broker as tenant, so they add up per broker. The same chain finds the anomaly: run-48f5 sits far above the median. All costs are invented units.

  1. 1 · Model span
    1. Tokens on the span: The span of write_recommendation in UW-006 records how many tokens the model read and wrote.
  2. 2 · Step cost
    1. Times the rate: Tokens per thousand times the tier's invented rate, read and written separately.
  3. 3 · Run cost
    1. Sum of the steps: All model steps of one run added up; tool calls add time, not tokens.
    2. The looping run: Nine next-step decisions with a growing context: 21.4 times the median run.
  4. 4 · Attribution
    1. Per broker: Runs grouped by the tenant that sent them; drill down before acting on a total.

3. The run that loops#

UW-502 is a submission on which the agent keeps searching. It calls search_guidance, asks the model what to do next, searches again, asks again: nine times, until it reaches the example's step limit and ends with an escalate recommendation for the underwriter. That end is a controlled state. Nothing failed. Every span has status ok. On an error dashboard the run is invisible.

The log of the run shows why its cost grows faster than its step count:

2026-09-14 10:41:04 INFO  [agent-worker] decide_next_step finished: 1062 tokens in, 20 out (large-model)
2026-09-14 10:41:05 INFO  [agent-worker] decide_next_step finished: 1512 tokens in, 20 out (large-model)
2026-09-14 10:41:05 INFO  [agent-worker] decide_next_step finished: 1962 tokens in, 20 out (large-model)
2026-09-14 10:41:06 INFO  [agent-worker] decide_next_step finished: 2412 tokens in, 20 out (large-model)

Each call reads everything the run has gathered so far, and each search adds about 450 tokens of passages. So the input of each call is larger than the one before: 1,062 tokens, then 1,512, then 1,962, up to 4,662 on the ninth. The output stays at 20 tokens. The cost of the run is dominated by reading, and reading grows with every loop. Nine loops cost far more than nine times one loop.

This is a cost anomaly: a run whose cost is far outside what runs of its kind usually cost. The example's rule is the simplest one that works on a day: a run costing more than three times the median run is flagged.

def anomalies(costs: list[dict[str, Any]], factor: float = ANOMALY_FACTOR) -> dict[str, Any]:
    """Runs whose cost is above `factor` x the median cost of the day. `costs` rows have run_id and cost."""
    m = median([c["cost"] for c in costs])
    flagged = [{"run_id": c["run_id"], "cost": c["cost"], "multiple": round(c["cost"] / m, 4) if m else None}
               for c in costs if m and c["cost"] > factor * m]
    return {"median": round(m, 6), "factor": factor, "threshold": round(factor * m, 6), "anomalies": flagged}

The median, not the mean, is the anchor on purpose: one looping run would pull the mean up and hide itself. The threshold is 37.3 cu; run-48f5 is the only run above it.

In our insurance agent. The anomaly is not a billing curiosity. A run that loops nine times on retrieval is also a run whose recommendation rests on a search that never converged. The cost signal finds it first; the trace then shows what the agent kept asking, and the underwriter sees an escalation with a reason. Cost and quality point at the same run from two sides.
What this is not. Stopping the loop is not observability. The step limit that ended UW-502 belongs to the harness (see harness.sammuthu.com), and budgets that stop a run belong to reliability (see reliable.sammuthu.com). This page measures, attributes and alerts.

4. Attribution: who spent it#

The month's bill is one number. Cost attribution means splitting it by something a person can act on: per run, per step, per tenant, per feature. In the example every run carries the broker that sent the submission as its tenant, so the per-run costs add up per broker:

Broker (tenant)Cost on the day (cu)
broker-0160.346
broker-02123.422
broker-0374.592
broker-0463.013
broker-05113.348
broker-06416.402

broker-06 stands out, and attribution alone would suggest a heavy customer. Drill one level down and the picture changes: run-48f5 alone is 265.6 of that broker's 416.4 cu. One looping run, not a busy broker. Attribute first, then look at the runs inside the number, or the wrong conversation happens with the wrong customer.

Attribution needs the identifiers of Identifiers on the cost record: run_id, submission_id, the step name and the tenant. Tenant is a bounded label and may sit on a cost metric; run_id belongs on the span or the cost row, never on a metric label (see Advanced).

5. Routing: what a smaller model changes#

Model routing means choosing a model tier per step instead of one model for everything. The example has three routing rules besides the one the day really ran (as-run, in which the plan step uses the small tier and the other steps the large tier).

Routing rulePlan stepsNext-step decisionsWrite the recommendationDay total (cu)
as-run (what the day used)smalllargelarge851.123
all-largelargelargelarge1210.070
all-smallsmallsmallsmall121.007
small-except-writesmallsmalllarge612.821

The re-pricing keeps each step's token counts and changes only the rate. That is a simplification the example states in its code: a different model would write different text, and so different token counts, and possibly a different recommendation. So the table answers one question honestly ("what would these same tokens cost under another rule?") and leaves the real question open: is the recommendation still good when a smaller model writes the decision steps? That is a measurement for eval.sammuthu.com, not a conclusion this page can draw from a cost table.

Control in the labWhat it changesWhat to watch
Rate of each tier (read and written)The cost of every stepThe day total and the median run
Routing ruleWhich tier each step usesThe total, and whether run-48f5 is still an anomaly
Anomaly factorThe threshold above the medianWhich runs are flagged

Try one thing in the lab: switch to small-except-write. The day total falls, and the looping run falls the most, because its cost was nine large-tier decision calls. Routing reduced the damage of the loop. It did not remove the loop.

Before AI. A Java service measured CPU time and database calls per request, and a slow query that ran in a loop was found the same way: a per-request number far above the median, then the trace. The new part is that the unit of cost is tokens, it is visible on every model span, and it grows with the context the agent carries, not with the size of the request.

6. Budgets and alerts#

A budget is an amount of cost units set aside for a period, a tenant or a run. It turns cost from a report into an alert. Two kinds matter, and they answer different questions.

Diagram: Two budgets, two questions

Read each lane on its own. A per-run budget watches one run while it happens and asks whether it is behaving. A period budget watches the month and asks whether spending follows the plan. Both raise alerts; stopping a run is a reliability decision, not an observability one.

  1. Per run
    1. Cost so far: The sum of step costs up to now, checked after each model step.
    2. Run alert: Flag the run, keep its trace by a tail rule, and put it in the review queue.
  2. Per period
    1. Spend rate: How fast the period budget burns compared with an even spend.
    2. Burn alert: Alert when the rate would exhaust the budget early, with a link to attribution.
The same idea for a coding agent. A coding-agent session such as CA-001 (read the instructions, edit, run the tests, fail, edit, pass, open a pull request) is priced the same way: tokens per model step, summed per session. Claude Code, for one, documents an OpenTelemetry export with a claude_code.token.usage metric (with types for input, output and cache tokens) and a claude_code.cost.usage metric, and describes the cost on its request event as "Estimated cost". A session that retries a failing test many times is the coding agent's UW-502. How such a session is run belongs to harness.sammuthu.com; counting it belongs here.
Why loop cost grows faster than loop count

If each loop iteration appends about a passages of tokens to the context, and the first decision call reads c tokens, the k-th call reads about c + (k − 1)a tokens. Over n calls the tokens read sum to about n·c + a·n(n − 1)/2: linear in n from the base context, quadratic from the growth. For UW-502 (c = 1,062, a = 450, n = 9) that is 9 × 1,062 + 450 × 36 = 25,758 tokens read across the nine decision calls, which matches the run's log exactly; with the plan step's 727 tokens the run read 26,485.

Two consequences for design. A step limit bounds n, and so bounds the quadratic term. Summarizing or trimming the context between iterations lowers a, which changes the growth from quadratic to nearly linear. Both are decisions for the harness and context sites; the formula is how you see which one a cost curve calls for.

Prompt caching, where a provider offers it, changes the rate of the repeated prefix rather than the token count. Record cache tokens as their own count (Claude Code's metric has separate cache types), or a cost model built on two counts will overstate the cost of a cached run.

Better anomaly rules than three times the median

The example's rule works on one day of one workload. Production needs more.

  • Per class of run. A run that writes a long recommendation is legitimately more expensive than one that requests information. Compare each run with the median of its own class (case type, route, tenant tier).
  • Robust spread. Use the median absolute deviation, or a high percentile of the last N days, instead of a fixed multiple, so that the threshold follows real variation.
  • Early signals. Cost so far plus steps so far, checked during the run, catches a loop before it finishes. Model-call count is a good proxy and is known earlier than the bill.
  • Flag, then sample. A flagged run is a reason to keep its trace (a tail-sampling rule) and to put it in the review queue of Quality.

Checklist#

In one breath#

Cost is token counts times rates, and the token counts are already on every model span, so the cost of a step, a run and a tenant is a sum away. The median run is the anchor: a run far above it, like the looping UW-502 at 21 times the median, is an anomaly even when every span says ok. Attribute cost before acting on it, measure what routing changes, and leave the question of whether a cheaper model is still good to the evals.

Where this leads#

You can now price a run, find the one that spends ten times too much, and say what routing would change. Every page so far built its own records in its own code. The next page looks at the libraries, standards and products that do this work for you, each read in its own documentation: Tracing tools.