Every page of this site took one idea and showed it on a piece of the example. This page puts the pieces back together. It follows a single run, run-7f3a, through every record it leaves, in the order they are written, and stops at each one to say which page explained it and which question it answers. Read it as a review of the site, or as the place to start if you learn best from one complete case.
The case is UW-006. A broker asks the insurer to quote property cover for a masonry office: built 2009, 15,000 square feet, sprinklered, roof 12 years old, wind zone B, no losses. Under the policy fixture this should be a clean case. But at 14:00 on the simulated day the roof-data provider stops answering, and this run arrives at 14:07. The model recommends; a person decides.
An illustrative reconstruction. The insurer, the broker, the case, the rules, the fields, the provider, the times, the token counts and the costs are invented for teaching. The agent is a scripted stand-in, so these records show how the mechanism works, not how good any model is. Nothing here reports the behavior of a real system or model.
0. Run it yourself#
The example needs only Python 3 and no packages. Copy the folder first, because the runner rewrites its out/ folder:
cp -R examples/uw-observe "$TMPDIR/uw-observe-lab" && cd "$TMPDIR/uw-observe-lab" && /opt/homebrew/bin/python3 run_observe.py
Simulated day 2026-09-14 · seed 7 · agent: scripted stand-in (no model called)
runs 48 · log records 661 · metric points 812 · audit events 93 · security log entries 6
run status: denied 1, fallback 5, ok 41, step_limit 1
roof-provider outage 14:00:00-14:45:00: 5 runs inside, get_roof_assessment timeouts 10 (of 54 calls)
run_duration_ms: count 48 · p50 3107 · p95 7346 · buckets [0, 0, 0, 0, 0, 42, 6, 0] (bounds [100, 250, 500, 1000, 2500, 5000, 10000] ms, last = above)
anchor UW-006 · SUB-1006 · run-7f3a · trace 7540bd55d4e55ccb8f04f3d81880717a · arrived 14:07:12 · 7216 ms · request_info · decided request_info at 14:45:19
audit chain: every hash matches (93 events)
wrote out/ (96 trace files, logs.jsonl, metrics.jsonl, audit_ledger.jsonl, security.jsonl, analyses)
python3 run_observe.py --case UW-006 prints everything below for this one run; --show prints the analyses (slo, cost, sampling, drift, replay, promote, redact, cardinality, ledger, labels, ca-001). The folder's README maps every concept of the site to a file and function.
1. 14:07:12, the gateway: a request, a run, a trace#
The broker's email has already become submission SUB-1006. The gateway, the service that receives the request over HTTP, does three things in 17 ms. It assigns a request_id for this HTTP request and a run_id, run-7f3a, for this execution of the workflow. It starts the trace, 7540bd55d4e55ccb8f04f3d81880717a, with a SERVER span. And it puts a message on the queue, with a PRODUCER span and the trace context in a traceparent header.
2026-09-14 14:07:12 INFO [gateway] POST /submissions 202 accepted SUB-1006 as run-7f3a
Three identifiers already exist, and they must never be confused: the submission (years), the run (this execution) and the request (this HTTP call). Identifiers explains each; Tracing explains the header.
2. 14:07:12, the queue and the worker#
The worker receives the message 165 ms later, reads the traceparent header and continues the same trace with a CONSUMER span. It binds run_id, submission_id, request_id and policy_version 2026.2 to its logger, so every line it writes carries them.
2026-09-14 14:07:12 INFO [agent-worker] message received from submissions (delivery 1, waited 165 ms)
2026-09-14 14:07:12 INFO [agent-worker] run started for SUB-1006
If the queue had delivered the message twice, as in case UW-504, there would be two consumer spans and one run; the idempotency_key would keep the second delivery from storing a second recommendation.
3. 14:07:12, the model plans#
The agent run span, underwrite_submission, opens. The first child is a model span, plan_steps, on the small model tier: 726 tokens in, 25 out, finish reason stop, first token after 211 ms, 812 ms in total. The plan: get the roof assessment, search the guidance, find similar decisions, check eligibility.
2026-09-14 14:07:12 INFO [agent-worker] plan_steps finished: 726 tokens in, 25 out (small-model)
What a model span records, and why, is on GenAI telemetry. Its cost, 0.826 cost units (invented units), is on Cost. Whether its prompt (the broker's email) is stored on the span is the content capture decision of Sensitive content.
4. 14:07:13 to 14:07:17, the provider times out, twice#
The agent calls get_roof_assessment. The roof-data provider, the outside service behind that tool, does not answer. After 2,000 ms the call times out. The tool span ends with status error and attempt_number 1, and carries a span event retry_scheduled with a 500 ms backoff. The second attempt times out the same way.
2026-09-14 14:07:14 ERROR [agent-worker] get_roof_assessment failed: timeout after 2000 ms attempt_number=1
2026-09-14 14:07:17 ERROR [agent-worker] get_roof_assessment failed: timeout after 2000 ms attempt_number=2
2026-09-14 14:07:17 WARN [agent-worker] roof assessment unavailable after 2 attempts; using the controlled fallback request_info
Read the severities. The failed steps are ERROR. The fallback is WARN: the system handled the condition as designed. We do not own the provider's telemetry; we see it only through these spans and lines. These two attempts are why the run will take more than 7 seconds.
The log explorer holds the whole simulated day. Filter by run_id = run-7f3a to see every line of this run, from three services, in time order; then search for "timeout" across the day and see 10 lines from 5 runs instead.
| Filter | Lines | What it shows |
|---|---|---|
run_id = run-7f3a | 15 | This run, from the gateway, the worker and the cockpit |
| free text "timeout" | 10 | Every timeout of the day, from 5 runs |
category tool, severity ERROR | 10 | The provider outage in one screen |
5. 14:07:17 to 14:07:19, evidence, rules and the recommendation#
The agent continues without the roof assessment. search_guidance returns 5 passages from the policy fixture (the retrieval span lists their ids, scores and trust marks; none is untrusted in this run). find_similar_decisions returns none. check_eligibility runs the deterministic rules of policy version 2026.2 and returns request_info with no rule fired: with the roof age unverified, no rule can decide. The second model span, write_recommendation on the large tier, reads 723 tokens and writes 111: the recommendation.
{"recommendation": "request_info",
"reasons": ["The roof-data provider did not answer after two attempts, so the roof age on the application (12 years) is not verified.",
"No recommendation to approve or decline is made without that check.",
"Next step: ask the provider again later or request an inspection report."],
"citations": [], "missing": ["roof_assessment"], "stated_confidence": 0.7,
"requires_human": true, "policy_version": "2026.2"}
This is the agent's account of the run. Every statement in it is backed by a record above: the two error spans, the empty citations, the policy version on every line. That is the difference between a claim and evidence, from the Start page.
6. 14:07:19, the checkpoint and the write#
Two last steps. save_checkpoint records that the run reached "recommendation written", with a hash of the state. A checkpoint is saved execution state from which a run could continue; resuming from it is the subject of memory.sammuthu.com. The span records only that it was written.
Then record_recommendation, the write tool, stores the recommendation for the underwriter with the idempotency key rec-run-7f3a, derived from the run: a second call with the same key stores nothing. And the worker appends the first audit event.
2026-09-14 14:07:19 INFO [agent-worker] checkpoint saved: recommendation_written
2026-09-14 14:07:19 INFO [agent-worker] audit event appended: recommendation_recorded (seq 30)
2026-09-14 14:07:19 INFO [agent-worker] run finished: recommendation request_info written for the underwriter; a person decides
The automated part of the run is over: 7,216 ms, 14 spans in one trace.
| Span | Kind | Starts (ms) | Lasts (ms) | Status |
|---|---|---|---|---|
POST /submissions | http, SERVER | 0 | 17 | ok |
publish submissions | queue, PRODUCER | 12 | 3 | ok |
process submissions | queue, CONSUMER | 182 | 7034 | ok |
underwrite_submission | agent | 182 | 7034 | ok |
plan_steps | model | 182 | 812 | ok |
gather_evidence | agent | 994 | 4890 | ok |
get_roof_assessment attempt 1 | tool | 994 | 2000 | error |
get_roof_assessment attempt 2 | tool | 3494 | 2000 | error |
search_guidance | retrieval | 5494 | 240 | ok |
find_similar_decisions | retrieval | 5734 | 150 | ok |
check_eligibility | policy_check | 5884 | 4 | ok |
write_recommendation | model | 5888 | 1310 | ok |
save_checkpoint | checkpoint | 7198 | 6 | ok |
record_recommendation | tool | 7204 | 12 | ok |
7. 14:10, the alert#
Meanwhile the run counts as bad on two SLIs: it took longer than 7 seconds (latency) and it lacked the roof assessment (evidence). It counts as good on the third: it ended in a controlled state. Together with the other runs inside the outage, it pushes the burn rate over the last hour to 25 and over the last five minutes to 100. At 14:10 the fast-burn rule pages the on-call engineer. SLOs and alerts has the arithmetic; Incidents has the response.
8. 14:45:19, the underwriter decides#
Thirty-eight minutes later the underwriter opens the recommendation in the underwriter cockpit, the web application where underwriters review recommendations and decide. Their decision is a new HTTP request, and so a new trace, 1fb8aad7c68d8c80d7244e4d8780b70e, whose root span carries a span link to the record_recommendation span of the run's trace. Both traces carry run_id run-7f3a. The cockpit appends the second audit event:
seq 30 2026-09-14T14:07:19.216Z recommendation_recorded actor underwriting-agent {"recommendation": "request_info", "evidence_ids": ["EV-APPL-1006", "EV-GUIDE-1006"]}
seq 75 2026-09-14T14:45:19.256Z human_decision_recorded actor underwriter {"decided_by": "underwriter", "decision": "request_info", "override": false, "reason": "Agrees: ask the broker for an inspection report, because the roof-data provider did not answer."}
The evidence of the first event lists the application and the guidance, not the roof assessment, because there was none. The second records who decided (the role), what, whether it overrode the agent, and why. These two events are the answer to the auditor of the Start page, and they are kept for years after the trace has expired.
| Control | The debugging trace of run-7f3a | The audit events of run-7f3a |
|---|---|---|
| Sampling at 25 percent, head only | Dropped (its trace id falls outside the kept quarter) | Kept: never sampled |
| Retention after 30 days | Trace gone after 14 days, logs after 30 | Kept for the illustrative 2,555 days |
| Redaction (email, money) | Prompt masked | Unchanged: no personal data in them |
| Change the decision of seq 75 | Not applicable | Verification breaks at seq 75 |
9. After the run: labels, replay and a golden case#
Three more records come from the run after it is over.
A quality label. The online labeller marks the run missing_evidence, because the recommendation lists the roof assessment as missing. Not a failure: a count that rises when a dependency degrades. See Quality.
A replay. The next morning an engineer replays the run from its snapshot. With all six parts captured (tool responses, retrieved passages, policy version, model outputs, clock, seed), it reproduces exactly. With the tool responses missing, the recovered provider answers and the replay diverges at step 1. With nothing captured, the "replay" recommends approve: a different run in a different world. See Replay.
A golden case. The run is promoted: snapshot taken, the broker's email and the building value redacted, the expected result (request_info, roof assessment missing, two attempts) agreed by a person in the role of platform engineer, policy version recorded. It becomes UW-P-7f3a, and the regression dataset moves from 1.4.0 to 1.5.0. See Incidents.
| Step | Without it |
|---|---|
| Snapshot with tool responses | The case cannot reproduce the timeouts |
| Redaction | The broker's email leaves production |
| Expected result, agreed | Nobody decided what is right |
| Labelled by a role | No accountable judgement |
| Policy version | The case cannot say which rules applied |
Diagram: run-7f3a from 14:07:12 to the next morning
Read left to right in time. The automated run takes 7216 ms and ends with a stored recommendation and an audit event. The alert fires at 14:10. The underwriter decides at 14:45:19 in a second trace. The next day the run is replayed and promoted to a golden case.
- 14:07:12 · the run
- Gateway and queue: request_id, run_id and the trace start; the message carries traceparent.
- Plan, two timeouts, fallback: A model plan, two 2000 ms provider timeouts, guidance search, rules, recommendation.
- Audit seq 30: The stored recommendation as an audit event, with evidence ids and idempotency key.
- 14:10 · the alert
- Fast-burn page: The outage runs burn the budget 25 times too fast over the last hour.
- 14:45:19 · the decision
- Decision trace: The cockpit's request is a new trace linked to record_recommendation.
- Audit seq 75: The underwriter's decision, override false, with the reason.
- Next day · after
- Replay: The run re-executed from its snapshot in a sandbox.
- Golden case: Redacted, expected result agreed, dataset version bumped.
10. Every record of the run, in one table#
| Time | Record | Kind | Kept | Page |
|---|---|---|---|---|
| 14:07:12 | Gateway access line | diagnostic log | 30 days | Log lines |
| 14:07:12 | Root span, producer span, traceparent | trace | 14 days, may be sampled | Tracing |
| 14:07:12 | Request counter, duration histogram | metric | as configured | Four signals |
| 14:07:12 to 14:07:19 | Model, tool, retrieval, rule, checkpoint spans | trace | 14 days, may be sampled | GenAI telemetry |
| 14:07:14, 14:07:17 | Two ERROR lines, two error spans, attempt_number 1 and 2 | log, trace | 30 / 14 days | Incidents |
| 14:07:19 | recommendation_recorded, seq 30 | audit event | years | Audit logs |
| 14:10 | Fast-burn page | alert | in the alerting system | SLOs and alerts |
| 14:45:19 | Decision trace with span link | trace | 14 days | Tracing |
| 14:45:19 | human_decision_recorded, seq 75 | audit event | years | Audit logs |
| After | missing_evidence label | quality signal | with the run | Quality |
| Next day | Replay report | diagnostic | short | Replay |
| Next day | UW-P-7f3a in uw-regression 1.5.0 | golden case | with the dataset | Incidents |
Diagram: Where the records of run-7f3a live, and for how long
Each lane is one store with its own promise. Diagnostic records are short-lived and may be sampled or redacted. Audit events are complete and kept for years. Quality labels and golden cases live with the evaluation work. The same run_id joins all of them.
- Diagnostic stores
- Logs and traces: The 15 log lines and the two traces; may be sampled and redacted.
- Metrics: Counters and histograms that include this run among many.
- Audit ledger
- Seq 30 and 75: The recommendation and the decision, never sampled, never changed.
- Quality and evaluation
- Label and golden case: The missing_evidence label and the promoted case UW-P-7f3a.
The same idea for a coding agent. The same walk works for the coding-agent sessionCA-001: a session trace with model and tool spans, a failedrun_testsfollowed by a passing one, a pull request, and a separate CI trace linked to it; its audit record is the review and merge of the pull request.--show ca-001prints both traces. See harness.sammuthu.com and diff-to-deploy.sammuthu.com.
What this is not. Whether request_info was the right recommendation, and how to grade it across many runs, belongs to eval.sammuthu.com. How to make the provider outage hurt less belongs to reliable.sammuthu.com. This page shows what was recorded, and what each record can prove.
Experiments to run on the example
The runner takes a seed (--seed 11) for a different day; the anchor run keeps its id. Four experiments make the ideas of the site concrete.
- Break propagation. In
pipeline.py, the worker asserts thatextractfound a validtraceparenton the message. PassNoneinstead of the header and that assert stops the run: in a real service, without such a check, the worker would start a new trace and the run would split in two without an error. - Tamper with the ledger.
--show ledgerdoes it for you; try a differentseqand field, and watch where verification stops. - Change the sampling. In
sampling.DEFAULT, switchkeep_errorsoff and run--show sampling:run-7f3ais still kept, now by the slow rule (7,216 ms is above the 7,000 ms threshold). Switchkeep_slowoff too, and it is gone. - Change the rates. In
cost.RATES, change the large tier and run--show cost: the median, the threshold and the anomaly multiple all move, and the looping run stays the outlier.
Run the tests after each change: python3 -m unittest in the folder. 68 tests check the facts this page relies on.
Checklist#
- You can name every record
run-7f3aleaves, the store it lives in, and how long it is kept. - Your system can answer the three opening questions of the Start page for one of its own runs: why it was slow, how many runs were affected, who decided.
- The decision lives in an audit event that survives the expiry of every trace.
- One run can be replayed from a snapshot and promoted to a golden case with its evidence trail.
In one breath#
run-7f3a arrives at 14:07:12, waits through two 2-second timeouts of the roof-data provider, falls back to request_info, saves a checkpoint, stores its recommendation and appends an audit event, all inside one 7,216 ms trace; the outage it was part of pages the on-call engineer at 14:10; the underwriter agrees at 14:45:19 in a second, linked trace and a second audit event; and the run lives on as a label, a faithful replay and a versioned golden case. Every page of this site is one of those records.
Where this leads#
That is the whole site in one run. The Glossary gathers every term with its example, and About says where the ideas came from, how the facts were checked, and which sister sites take over at each boundary.