All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Operations · 13

Incident response: from an alert to a root cause to a golden case

An alert fires at 14:10. Reading the trace, finding the cause, and turning the failed run into a versioned case that the eval suite keeps forever.

Metric, logs, trace
The alert says something is wrong. The logs say which runs. One trace says why.
Cause, not symptom
A root cause is the condition that, removed, would have prevented the failure. "The agent was slow" is not one.
Never the same way twice
The failed run becomes a versioned golden case, so every future change is tested against it.

At 14:10 on the simulated day the on-call engineer's phone rings. The alert reads: fast burn on the latency SLI of the underwriting agent, burn rate 25 over the last hour, 100 over the last five minutes. The same rule fires on the evidence SLI. The model recommends; a person decides, and somewhere in the system recommendations are now arriving late and without the roof assessment.

The engineer has the whole toolkit of this site at hand: structured logs, metrics, traces, audit events, replay. The danger in the next ten minutes is not a lack of data. It is reading the wrong record first, chasing a symptom, and ending the incident with nothing learned. This page follows one incident from the page to the postmortem, in the order the records should be read, and ends where the earlier pages could not: with the failed run turned into a test that the system must pass forever. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.

An incident is an unplanned event that degrades a service enough that people must act now. It has a start, a response and an end, and it leaves a record of its own.

1. The order of reading: metric, logs, trace#

Each signal answers one question cheaply. Read them in the order of the questions.

StepSignalQuestionAnswer on the simulated day
1The alert (a metric)Is a promise breaking, and which one?Latency and evidence SLIs burning since about 14:00; controlled SLI unaffected
2Logs, filteredWhich runs, which step, since when?ERROR lines from get_roof_assessment since 14:00, across several runs
3One traceWhy did this run take so long?Two timeouts of 2,000 ms with a 500 ms backoff, then a fallback
4Audit eventsWhat did the underwriters receive and decide?request_info recommendations; decisions arrive later

Step 1, the alert. The page names the SLI, which already narrows the search: latency and evidence are bad, but controlled is at 100 percent. Every run is still ending in a controlled state. That rules out the worst failure (runs crashing or leaving underwriters with nothing) and points at a dependency: evidence is missing, and runs are slow.

Step 2, the logs. The engineer opens the log explorer and filters by severity ERROR and category tool over the last 30 minutes. The explorer below holds the whole simulated day; try the same filter.

FilterWhat it shows during the incident
severity ERROR, category toolEvery get_roof_assessment timeout since 14:00, across the runs inside the outage
run_id = run-7f3aEvery record of one affected run, from every service, in time order
free text "timeout"The same timeouts, mixed with any other line that mentions the word

The pattern is visible in one screen: on the day, 10 of 54 calls to get_roof_assessment timed out, all of them between 14:00 and 14:45, spread over the five runs that arrived in that window. Other tools are fine. These numbers are synthetic.

Step 3, one trace. Pick one affected run and read it whole. The engineer opens run-7f3a (case UW-006):

     0 ms    17 ms  gateway      http         POST /submissions                            ok
   182 ms  7034 ms  agent-worker agent              underwrite_submission                  ok
   182 ms   812 ms  agent-worker model                plan_steps                           ok
   994 ms  2000 ms  agent-worker tool                   get_roof_assessment (attempt 1)    error
  3494 ms  2000 ms  agent-worker tool                   get_roof_assessment (attempt 2)    error
  5494 ms   240 ms  agent-worker retrieval              search_guidance                    ok
  5888 ms  1310 ms  agent-worker model                write_recommendation                 ok
  7198 ms     6 ms  agent-worker checkpoint           save_checkpoint                      ok
  7204 ms    12 ms  agent-worker tool                 record_recommendation                ok

(An excerpt of the waterfall the runner prints for --case UW-006; the full trace has 14 spans.) The time went into two tool spans of exactly 2,000 ms each, the timeout, with 500 ms between them. Everything else is as fast as on any healthy run. Total: 7,216 ms, slightly over the 7-second latency threshold. The recommendation is request_info, with roof_assessment listed as missing.

Step 4, what people received. The audit events of the affected runs show recommendation_recorded with request_info, and later human_decision_recorded: for run-7f3a the underwriter agreed at 14:45:19 and asked the broker for an inspection report. The business impact is clear: five submissions got a slower, less informed answer, and none got a wrong one.

Diagram: From the page at 14:10 to the root cause, in the order of the questions

Read left to right. Each step reads the cheapest record that answers the next question. The alert names the promise; the logs name the runs and the step; one trace shows where the time went; the audit events say what the underwriters received. The cause is then tested against its definition.

  1. 1 · Alert
    1. Fast burn page: The alert names the SLIs that are burning; the controlled SLI is unaffected.
  2. 2 · Logs
    1. ERROR, category tool: A filtered search shows the same tool failing in every affected run.
  3. 3 · One trace
    1. run-7f3a: Two tool spans of 2000 ms with a 500 ms backoff, then a controlled fallback.
  4. 4 · Audit events
    1. What people received: Recorded recommendations and the underwriter's decisions show the business impact.
  5. 5 · Cause
    1. Outside and inside: The provider stopped answering; our timeout and retry settings pushed runs over 7 seconds.
Before AI. This is the same order a Java team uses for a slow endpoint: the latency alert, the error logs filtered by downstream, one distributed trace. The agent adds step 4. With an agent, "what did the users receive" is not obvious from status codes, because a controlled fallback returns success. The audit events say what the underwriter actually saw.

2. Root cause#

A root cause is the condition that, if it had not been present, would have prevented the failure. Each candidate is tested against that definition.

Two causes, one outside, one inside, is a common result. The inside one is the one the team can change. Whether to shorten the timeout, try once instead of twice, or relax the latency promise during a known provider outage is a reliability decision, and it belongs to reliable.sammuthu.com. This page's job was to produce the evidence that makes the decision possible.

Common confusion. The agent's own explanation of the run is not a root cause analysis. The recommendation of run-7f3a says the provider did not answer; that is a claim the agent wrote. The spans with status error and duration 2,000 ms are evidence. A good postmortem cites the evidence, not the agent.
What this is not. Deciding whether the agent's behavior during the incident was acceptable is evaluation, owned by eval.sammuthu.com. Here the records say what happened; the eval site says whether that was good enough.

3. The timeline and the postmortem#

An incident timeline is the list of what happened and what people did, with times, built from the records rather than from memory. For this incident it reads:

Time (simulated)EventSource
14:00:00Roof-data provider stops answeringInferred: the first run after 14:00 logs the first timeout; we cannot see inside the provider
14:07:12run-7f3a arrives for SUB-1006Gateway access log
14:07:14, 14:07:17Attempts 1 and 2 of get_roof_assessment time outWorker ERROR logs, tool spans
14:07:19request_info recorded for the underwriterAudit event recommendation_recorded
14:10:00Fast-burn page on latency and evidenceAlerting
14:20:00Slow-burn rule also firesAlerting
14:45:00Provider answers againInferred: runs after the window get answers again
14:45:19Underwriter decides run-7f3a: agrees, asks the broker for an inspection reportAudit event human_decision_recorded

A postmortem is the written review after the incident: the timeline, the impact, the causes, what went well, what went badly, and the actions, each with an owner. It is blameless by convention: it asks how the system allowed the failure, not who caused it. For this incident, "what went well" has a real entry: the fallback kept every run in a controlled state, and the controlled SLI proved it. "What went badly" too: the latency promise broke for every run during the outage, and an earlier alert on the available SLI at 13:10 paged for a single denied write that was a security matter, not an outage.

4. From a failed run to a golden case#

The last step is the one that makes the next incident smaller. The failed run is turned into a test.

A golden case is a test case whose expected result people agreed on: an input, the expected output, and who agreed. It lives in a versioned dataset that every change to the agent is checked against. Golden cases and their grading belong to eval.sammuthu.com. Promotion is the hand-over this site owns: taking a production run, with its snapshot from Replay, and turning it into such a case.

The run cannot be copied as it is. Four things must happen first:

  1. Take the snapshot, including the tool responses, or the case cannot reproduce what the provider did.
  2. Redact the input: the broker's email and the building value must not leave production in a test fixture.
  3. Set the expected result, agreed by a person with the authority to judge it, and record their role.
  4. Record the policy version the case was decided under.

The example's promote.py refuses to produce a ready case until all four are done; anything missing is listed, and the dataset version does not move. With all four, the runner prints the case:

cd "$TMPDIR/uw-observe-incidents" && /opt/homebrew/bin/python3 run_observe.py --show promote
== promote
{
 "id": "UW-P-7f3a",
 "title": "Promoted from production: UW-006 run run-7f3a",
 "policy_version": "2026.2",
 "input": {
  "source_email": "From: [EMAIL]\nTo: [EMAIL]\nSubject: Quote request SUB-1006\n\nPlease quote property cover for a masonry office, built 2009, 15,000 sq ft, sprinklered. Roof is 12 years old. Wind zone B. No losses.",
  "submission": {
   "submission_id": "SUB-1006",
   "occupancy": "office",
   "construction": "masonry",
   "year_built": 2009,
   "area_sq_ft": 15000,
   "building_value_usd": "[AMOUNT]",
   "sprinklered": true,
   "roof_age_years": 12,
   "wind_zone": "B",
   "prior_losses_5y": 0
  }
 },
 "world": {
  "roof_assessment": {
   "status": "timeout",
   "failing_attempts": 2
  }
 },
 "expected": {
  "recommendation": "request_info",
  "citations": [],
  "missing": [
   "roof_assessment"
  ],
  "requires_human": true,
  "roof_attempts": 2
 },
 "provenance": {
  "labelled_by": "platform engineer",
  "role_is_synthetic": true,
  "date": "2026-09-14",
  "synthetic": true,
  "promoted_from": {
   "run_id": "run-7f3a",
   "trace_id": "7540bd55d4e55ccb8f04f3d81880717a",
   "submission_id": "SUB-1006"
  },
  "redaction": [
   "email",
   "money"
  ]
 }
}
ready True · dataset uw-regression 1.4.0 -> 1.5.0 · missing nothing

Read the case from the bottom up. provenance.promoted_from keeps the run_id, trace_id and submission_id, so anyone can go from the test back to the production evidence. redaction says which rules ran; the email addresses and the building value are now placeholders. expected was set by a person in the role of platform engineer; the role is synthetic, as everything here. world records that the provider failed twice, which is what makes the case reproduce the incident. And the dataset moves from 1.4.0 to 1.5.0: a new case is an addition, a minor version, so a later report can say which dataset version a result was measured against.

The example's dataset already held a hand-written case UW-006. The promoted case is different in kind: it comes from a real (synthetic) production run, with its recorded answers, and it carries the evidence trail. A dataset that grows from production incidents tests the failures that actually happen.

Choice in the labIf left outWhat the lab shows
Snapshot parts (tool responses)The case cannot reproduce the timeout"missing: tool responses" and no version bump
Redaction rulesThe broker's email stays in the input"missing: redaction"
Expected resultNobody agreed what is right"missing: expected result"
Labelled by (a role)No accountable judgement"missing: labelled_by"

Diagram: Promotion: a production run becomes a versioned golden case

Read left to right. The snapshot supplies what the run read. Redaction removes what must not leave production. A person sets the expected result and is recorded by role. Only then is the case added and the dataset version moves. If any step is missing, the case is not ready and the version stays.

  1. 1 · Snapshot
    1. Run and its answers: The input and every outside answer of run-7f3a, from the replay snapshot.
  2. 2 · Redact
    1. Email and money masked: Sensitive values are replaced by placeholders before the case leaves production.
  3. 3 · Agree
    1. Expected result: A person with authority agrees on what should have happened; the role is recorded.
  4. 4 · Version
    1. UW-P-7f3a added: The case joins the regression dataset and the minor version moves.
The same idea for a coding agent. A coding-agent session that broke the build is promoted the same way: the commit, the instruction files, the tool outputs and the failing CI log become a fixture, with the expected outcome agreed by a reviewer. Session CA-001, whose first test run failed before the agent fixed it, is a candidate. How such fixtures gate a pipeline is the subject of diff-to-deploy.sammuthu.com.
Detection time, and where it came from

The provider stopped answering at 14:00. The first page came at 14:10. Where did ten minutes go? The first affected run arrived after 14:00, waited through two timeouts, and finished as a bad event; the alert rules are evaluated every five minutes; the fast rule needs both its windows above the threshold. On a busy service the first bad events arrive within seconds, and detection is bounded by the evaluation interval and the short window. On a quiet one, like the simulated day with a run every half hour or so, detection waits for traffic.

Two improvements are on the observability side. A dependency SLI fed by synthetic probes of the provider detects the outage without waiting for a customer run. And an alert on the worker's client-side error rate for one tool, with a minimum number of calls, pages on the dependency directly, with a runbook that starts at the right place. Both are measurements; whether to fail over, shorten timeouts or pause intake is reliability work.

Rules for promotion at scale

Promoting every failed run floods the dataset with near-duplicates. A few rules keep it useful.

  • Promote by failure class, not by run. Five runs failed the same way during the outage; one case represents them. Tag it with the incident id.
  • Keep the expected result honest. The expected result is what should have happened, not what did happen. For UW-006 they coincide, because the fallback was correct. For a run where the agent was wrong, the expected result is the correction, set by someone with authority.
  • Re-check after policy changes. A golden case is decided under one policy_version. When the fixture changes, cases decided under the old version are reviewed, not silently kept.
  • Respect retention. A promoted case outlives the trace it came from. Redaction before promotion is what makes that acceptable.

Checklist#

In one breath#

An incident is read in the order of its questions: the alert says which promise is breaking, filtered logs say which runs and which step, one trace says why, and the audit events say what the underwriters received. On the simulated day the root cause was outside (the provider stopped answering) and a contributing cause was inside (two 2-second attempts in a 7-second promise). The failed run then becomes UW-P-7f3a, a redacted, versioned golden case that every future change must pass.

Where this leads#

You can now go from a page at 14:10 to a cause, a written timeline, and a test that keeps the failure from coming back unnoticed. One signal was missing from this incident: what the runs cost. A run that loops can be healthy, fast enough and controlled, and still spend twenty times what it should. That is Cost.