All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Reference · 17

The insurance lab: one timeout, followed through every record

Case UW-006 end to end: the gateway log, the model span, the provider timeout and retry, the checkpoint, the underwriter's decision and the audit event, with the code that produced them.

One run, end to end
Case UW-006, run run-7f3a: from the broker's email to the underwriter's decision 38 minutes later.
Every record
Gateway log, queue, model span, two timeouts, the fallback, the checkpoint, two audit events, a replay and a golden case.
One command
All of it comes from examples/uw-observe/, standard-library Python, run with one command.

Every page of this site took one idea and showed it on a piece of the example. This page puts the pieces back together. It follows a single run, run-7f3a, through every record it leaves, in the order they are written, and stops at each one to say which page explained it and which question it answers. Read it as a review of the site, or as the place to start if you learn best from one complete case.

The case is UW-006. A broker asks the insurer to quote property cover for a masonry office: built 2009, 15,000 square feet, sprinklered, roof 12 years old, wind zone B, no losses. Under the policy fixture this should be a clean case. But at 14:00 on the simulated day the roof-data provider stops answering, and this run arrives at 14:07. The model recommends; a person decides.

An illustrative reconstruction. The insurer, the broker, the case, the rules, the fields, the provider, the times, the token counts and the costs are invented for teaching. The agent is a scripted stand-in, so these records show how the mechanism works, not how good any model is. Nothing here reports the behavior of a real system or model.

0. Run it yourself#

The example needs only Python 3 and no packages. Copy the folder first, because the runner rewrites its out/ folder:

cp -R examples/uw-observe "$TMPDIR/uw-observe-lab" && cd "$TMPDIR/uw-observe-lab" && /opt/homebrew/bin/python3 run_observe.py
Simulated day 2026-09-14 · seed 7 · agent: scripted stand-in (no model called)
runs 48 · log records 661 · metric points 812 · audit events 93 · security log entries 6
run status: denied 1, fallback 5, ok 41, step_limit 1
roof-provider outage 14:00:00-14:45:00: 5 runs inside, get_roof_assessment timeouts 10 (of 54 calls)
run_duration_ms: count 48 · p50 3107 · p95 7346 · buckets [0, 0, 0, 0, 0, 42, 6, 0] (bounds [100, 250, 500, 1000, 2500, 5000, 10000] ms, last = above)
anchor UW-006 · SUB-1006 · run-7f3a · trace 7540bd55d4e55ccb8f04f3d81880717a · arrived 14:07:12 · 7216 ms · request_info · decided request_info at 14:45:19
audit chain: every hash matches (93 events)

wrote out/ (96 trace files, logs.jsonl, metrics.jsonl, audit_ledger.jsonl, security.jsonl, analyses)

python3 run_observe.py --case UW-006 prints everything below for this one run; --show prints the analyses (slo, cost, sampling, drift, replay, promote, redact, cardinality, ledger, labels, ca-001). The folder's README maps every concept of the site to a file and function.

1. 14:07:12, the gateway: a request, a run, a trace#

The broker's email has already become submission SUB-1006. The gateway, the service that receives the request over HTTP, does three things in 17 ms. It assigns a request_id for this HTTP request and a run_id, run-7f3a, for this execution of the workflow. It starts the trace, 7540bd55d4e55ccb8f04f3d81880717a, with a SERVER span. And it puts a message on the queue, with a PRODUCER span and the trace context in a traceparent header.

2026-09-14 14:07:12 INFO  [gateway] POST /submissions 202 accepted SUB-1006 as run-7f3a

Three identifiers already exist, and they must never be confused: the submission (years), the run (this execution) and the request (this HTTP call). Identifiers explains each; Tracing explains the header.

2. 14:07:12, the queue and the worker#

The worker receives the message 165 ms later, reads the traceparent header and continues the same trace with a CONSUMER span. It binds run_id, submission_id, request_id and policy_version 2026.2 to its logger, so every line it writes carries them.

2026-09-14 14:07:12 INFO  [agent-worker] message received from submissions (delivery 1, waited 165 ms)
2026-09-14 14:07:12 INFO  [agent-worker] run started for SUB-1006

If the queue had delivered the message twice, as in case UW-504, there would be two consumer spans and one run; the idempotency_key would keep the second delivery from storing a second recommendation.

3. 14:07:12, the model plans#

The agent run span, underwrite_submission, opens. The first child is a model span, plan_steps, on the small model tier: 726 tokens in, 25 out, finish reason stop, first token after 211 ms, 812 ms in total. The plan: get the roof assessment, search the guidance, find similar decisions, check eligibility.

2026-09-14 14:07:12 INFO  [agent-worker] plan_steps finished: 726 tokens in, 25 out (small-model)

What a model span records, and why, is on GenAI telemetry. Its cost, 0.826 cost units (invented units), is on Cost. Whether its prompt (the broker's email) is stored on the span is the content capture decision of Sensitive content.

4. 14:07:13 to 14:07:17, the provider times out, twice#

The agent calls get_roof_assessment. The roof-data provider, the outside service behind that tool, does not answer. After 2,000 ms the call times out. The tool span ends with status error and attempt_number 1, and carries a span event retry_scheduled with a 500 ms backoff. The second attempt times out the same way.

2026-09-14 14:07:14 ERROR [agent-worker] get_roof_assessment failed: timeout after 2000 ms  attempt_number=1
2026-09-14 14:07:17 ERROR [agent-worker] get_roof_assessment failed: timeout after 2000 ms  attempt_number=2
2026-09-14 14:07:17 WARN  [agent-worker] roof assessment unavailable after 2 attempts; using the controlled fallback request_info

Read the severities. The failed steps are ERROR. The fallback is WARN: the system handled the condition as designed. We do not own the provider's telemetry; we see it only through these spans and lines. These two attempts are why the run will take more than 7 seconds.

The log explorer holds the whole simulated day. Filter by run_id = run-7f3a to see every line of this run, from three services, in time order; then search for "timeout" across the day and see 10 lines from 5 runs instead.

FilterLinesWhat it shows
run_id = run-7f3a15This run, from the gateway, the worker and the cockpit
free text "timeout"10Every timeout of the day, from 5 runs
category tool, severity ERROR10The provider outage in one screen

5. 14:07:17 to 14:07:19, evidence, rules and the recommendation#

The agent continues without the roof assessment. search_guidance returns 5 passages from the policy fixture (the retrieval span lists their ids, scores and trust marks; none is untrusted in this run). find_similar_decisions returns none. check_eligibility runs the deterministic rules of policy version 2026.2 and returns request_info with no rule fired: with the roof age unverified, no rule can decide. The second model span, write_recommendation on the large tier, reads 723 tokens and writes 111: the recommendation.

{"recommendation": "request_info",
 "reasons": ["The roof-data provider did not answer after two attempts, so the roof age on the application (12 years) is not verified.",
             "No recommendation to approve or decline is made without that check.",
             "Next step: ask the provider again later or request an inspection report."],
 "citations": [], "missing": ["roof_assessment"], "stated_confidence": 0.7,
 "requires_human": true, "policy_version": "2026.2"}

This is the agent's account of the run. Every statement in it is backed by a record above: the two error spans, the empty citations, the policy version on every line. That is the difference between a claim and evidence, from the Start page.

6. 14:07:19, the checkpoint and the write#

Two last steps. save_checkpoint records that the run reached "recommendation written", with a hash of the state. A checkpoint is saved execution state from which a run could continue; resuming from it is the subject of memory.sammuthu.com. The span records only that it was written.

Then record_recommendation, the write tool, stores the recommendation for the underwriter with the idempotency key rec-run-7f3a, derived from the run: a second call with the same key stores nothing. And the worker appends the first audit event.

2026-09-14 14:07:19 INFO  [agent-worker] checkpoint saved: recommendation_written
2026-09-14 14:07:19 INFO  [agent-worker] audit event appended: recommendation_recorded (seq 30)
2026-09-14 14:07:19 INFO  [agent-worker] run finished: recommendation request_info written for the underwriter; a person decides

The automated part of the run is over: 7,216 ms, 14 spans in one trace.

SpanKindStarts (ms)Lasts (ms)Status
POST /submissionshttp, SERVER017ok
publish submissionsqueue, PRODUCER123ok
process submissionsqueue, CONSUMER1827034ok
underwrite_submissionagent1827034ok
plan_stepsmodel182812ok
gather_evidenceagent9944890ok
get_roof_assessment attempt 1tool9942000error
get_roof_assessment attempt 2tool34942000error
search_guidanceretrieval5494240ok
find_similar_decisionsretrieval5734150ok
check_eligibilitypolicy_check58844ok
write_recommendationmodel58881310ok
save_checkpointcheckpoint71986ok
record_recommendationtool720412ok

7. 14:10, the alert#

Meanwhile the run counts as bad on two SLIs: it took longer than 7 seconds (latency) and it lacked the roof assessment (evidence). It counts as good on the third: it ended in a controlled state. Together with the other runs inside the outage, it pushes the burn rate over the last hour to 25 and over the last five minutes to 100. At 14:10 the fast-burn rule pages the on-call engineer. SLOs and alerts has the arithmetic; Incidents has the response.

8. 14:45:19, the underwriter decides#

Thirty-eight minutes later the underwriter opens the recommendation in the underwriter cockpit, the web application where underwriters review recommendations and decide. Their decision is a new HTTP request, and so a new trace, 1fb8aad7c68d8c80d7244e4d8780b70e, whose root span carries a span link to the record_recommendation span of the run's trace. Both traces carry run_id run-7f3a. The cockpit appends the second audit event:

seq  30 2026-09-14T14:07:19.216Z recommendation_recorded  actor underwriting-agent {"recommendation": "request_info", "evidence_ids": ["EV-APPL-1006", "EV-GUIDE-1006"]}
seq  75 2026-09-14T14:45:19.256Z human_decision_recorded  actor underwriter        {"decided_by": "underwriter", "decision": "request_info", "override": false, "reason": "Agrees: ask the broker for an inspection report, because the roof-data provider did not answer."}

The evidence of the first event lists the application and the guidance, not the roof assessment, because there was none. The second records who decided (the role), what, whether it overrode the agent, and why. These two events are the answer to the auditor of the Start page, and they are kept for years after the trace has expired.

ControlThe debugging trace of run-7f3aThe audit events of run-7f3a
Sampling at 25 percent, head onlyDropped (its trace id falls outside the kept quarter)Kept: never sampled
Retention after 30 daysTrace gone after 14 days, logs after 30Kept for the illustrative 2,555 days
Redaction (email, money)Prompt maskedUnchanged: no personal data in them
Change the decision of seq 75Not applicableVerification breaks at seq 75

9. After the run: labels, replay and a golden case#

Three more records come from the run after it is over.

A quality label. The online labeller marks the run missing_evidence, because the recommendation lists the roof assessment as missing. Not a failure: a count that rises when a dependency degrades. See Quality.

A replay. The next morning an engineer replays the run from its snapshot. With all six parts captured (tool responses, retrieved passages, policy version, model outputs, clock, seed), it reproduces exactly. With the tool responses missing, the recovered provider answers and the replay diverges at step 1. With nothing captured, the "replay" recommends approve: a different run in a different world. See Replay.

A golden case. The run is promoted: snapshot taken, the broker's email and the building value redacted, the expected result (request_info, roof assessment missing, two attempts) agreed by a person in the role of platform engineer, policy version recorded. It becomes UW-P-7f3a, and the regression dataset moves from 1.4.0 to 1.5.0. See Incidents.

StepWithout it
Snapshot with tool responsesThe case cannot reproduce the timeouts
RedactionThe broker's email leaves production
Expected result, agreedNobody decided what is right
Labelled by a roleNo accountable judgement
Policy versionThe case cannot say which rules applied

Diagram: run-7f3a from 14:07:12 to the next morning

Read left to right in time. The automated run takes 7216 ms and ends with a stored recommendation and an audit event. The alert fires at 14:10. The underwriter decides at 14:45:19 in a second trace. The next day the run is replayed and promoted to a golden case.

  1. 14:07:12 · the run
    1. Gateway and queue: request_id, run_id and the trace start; the message carries traceparent.
    2. Plan, two timeouts, fallback: A model plan, two 2000 ms provider timeouts, guidance search, rules, recommendation.
    3. Audit seq 30: The stored recommendation as an audit event, with evidence ids and idempotency key.
  2. 14:10 · the alert
    1. Fast-burn page: The outage runs burn the budget 25 times too fast over the last hour.
  3. 14:45:19 · the decision
    1. Decision trace: The cockpit's request is a new trace linked to record_recommendation.
    2. Audit seq 75: The underwriter's decision, override false, with the reason.
  4. Next day · after
    1. Replay: The run re-executed from its snapshot in a sandbox.
    2. Golden case: Redacted, expected result agreed, dataset version bumped.

10. Every record of the run, in one table#

TimeRecordKindKeptPage
14:07:12Gateway access linediagnostic log30 daysLog lines
14:07:12Root span, producer span, traceparenttrace14 days, may be sampledTracing
14:07:12Request counter, duration histogrammetricas configuredFour signals
14:07:12 to 14:07:19Model, tool, retrieval, rule, checkpoint spanstrace14 days, may be sampledGenAI telemetry
14:07:14, 14:07:17Two ERROR lines, two error spans, attempt_number 1 and 2log, trace30 / 14 daysIncidents
14:07:19recommendation_recorded, seq 30audit eventyearsAudit logs
14:10Fast-burn pagealertin the alerting systemSLOs and alerts
14:45:19Decision trace with span linktrace14 daysTracing
14:45:19human_decision_recorded, seq 75audit eventyearsAudit logs
Aftermissing_evidence labelquality signalwith the runQuality
Next dayReplay reportdiagnosticshortReplay
Next dayUW-P-7f3a in uw-regression 1.5.0golden casewith the datasetIncidents

Diagram: Where the records of run-7f3a live, and for how long

Each lane is one store with its own promise. Diagnostic records are short-lived and may be sampled or redacted. Audit events are complete and kept for years. Quality labels and golden cases live with the evaluation work. The same run_id joins all of them.

  1. Diagnostic stores
    1. Logs and traces: The 15 log lines and the two traces; may be sampled and redacted.
    2. Metrics: Counters and histograms that include this run among many.
  2. Audit ledger
    1. Seq 30 and 75: The recommendation and the decision, never sampled, never changed.
  3. Quality and evaluation
    1. Label and golden case: The missing_evidence label and the promoted case UW-P-7f3a.
The same idea for a coding agent. The same walk works for the coding-agent session CA-001: a session trace with model and tool spans, a failed run_tests followed by a passing one, a pull request, and a separate CI trace linked to it; its audit record is the review and merge of the pull request. --show ca-001 prints both traces. See harness.sammuthu.com and diff-to-deploy.sammuthu.com.
What this is not. Whether request_info was the right recommendation, and how to grade it across many runs, belongs to eval.sammuthu.com. How to make the provider outage hurt less belongs to reliable.sammuthu.com. This page shows what was recorded, and what each record can prove.
Experiments to run on the example

The runner takes a seed (--seed 11) for a different day; the anchor run keeps its id. Four experiments make the ideas of the site concrete.

  • Break propagation. In pipeline.py, the worker asserts that extract found a valid traceparent on the message. Pass None instead of the header and that assert stops the run: in a real service, without such a check, the worker would start a new trace and the run would split in two without an error.
  • Tamper with the ledger. --show ledger does it for you; try a different seq and field, and watch where verification stops.
  • Change the sampling. In sampling.DEFAULT, switch keep_errors off and run --show sampling: run-7f3a is still kept, now by the slow rule (7,216 ms is above the 7,000 ms threshold). Switch keep_slow off too, and it is gone.
  • Change the rates. In cost.RATES, change the large tier and run --show cost: the median, the threshold and the anomaly multiple all move, and the looping run stays the outlier.

Run the tests after each change: python3 -m unittest in the folder. 68 tests check the facts this page relies on.

Checklist#

In one breath#

run-7f3a arrives at 14:07:12, waits through two 2-second timeouts of the roof-data provider, falls back to request_info, saves a checkpoint, stores its recommendation and appends an audit event, all inside one 7,216 ms trace; the outage it was part of pages the on-call engineer at 14:10; the underwriter agrees at 14:45:19 in a second, linked trace and a second audit event; and the run lives on as a label, a faithful replay and a versioned golden case. Every page of this site is one of those records.

Where this leads#

That is the whole site in one run. The Glossary gathers every term with its example, and About says where the ideas came from, how the facts were checked, and which sister sites take over at each boundary.