At 14:10 on the simulated day the on-call engineer's phone rings. The alert reads: fast burn on the latency SLI of the underwriting agent, burn rate 25 over the last hour, 100 over the last five minutes. The same rule fires on the evidence SLI. The model recommends; a person decides, and somewhere in the system recommendations are now arriving late and without the roof assessment.
The engineer has the whole toolkit of this site at hand: structured logs, metrics, traces, audit events, replay. The danger in the next ten minutes is not a lack of data. It is reading the wrong record first, chasing a symptom, and ending the incident with nothing learned. This page follows one incident from the page to the postmortem, in the order the records should be read, and ends where the earlier pages could not: with the failed run turned into a test that the system must pass forever. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
An incident is an unplanned event that degrades a service enough that people must act now. It has a start, a response and an end, and it leaves a record of its own.
1. The order of reading: metric, logs, trace#
Each signal answers one question cheaply. Read them in the order of the questions.
| Step | Signal | Question | Answer on the simulated day |
|---|---|---|---|
| 1 | The alert (a metric) | Is a promise breaking, and which one? | Latency and evidence SLIs burning since about 14:00; controlled SLI unaffected |
| 2 | Logs, filtered | Which runs, which step, since when? | ERROR lines from get_roof_assessment since 14:00, across several runs |
| 3 | One trace | Why did this run take so long? | Two timeouts of 2,000 ms with a 500 ms backoff, then a fallback |
| 4 | Audit events | What did the underwriters receive and decide? | request_info recommendations; decisions arrive later |
Step 1, the alert. The page names the SLI, which already narrows the search: latency and evidence are bad, but controlled is at 100 percent. Every run is still ending in a controlled state. That rules out the worst failure (runs crashing or leaving underwriters with nothing) and points at a dependency: evidence is missing, and runs are slow.
Step 2, the logs. The engineer opens the log explorer and filters by severity ERROR and category tool over the last 30 minutes. The explorer below holds the whole simulated day; try the same filter.
| Filter | What it shows during the incident |
|---|---|
| severity ERROR, category tool | Every get_roof_assessment timeout since 14:00, across the runs inside the outage |
run_id = run-7f3a | Every record of one affected run, from every service, in time order |
| free text "timeout" | The same timeouts, mixed with any other line that mentions the word |
The pattern is visible in one screen: on the day, 10 of 54 calls to get_roof_assessment timed out, all of them between 14:00 and 14:45, spread over the five runs that arrived in that window. Other tools are fine. These numbers are synthetic.
Step 3, one trace. Pick one affected run and read it whole. The engineer opens run-7f3a (case UW-006):
0 ms 17 ms gateway http POST /submissions ok
182 ms 7034 ms agent-worker agent underwrite_submission ok
182 ms 812 ms agent-worker model plan_steps ok
994 ms 2000 ms agent-worker tool get_roof_assessment (attempt 1) error
3494 ms 2000 ms agent-worker tool get_roof_assessment (attempt 2) error
5494 ms 240 ms agent-worker retrieval search_guidance ok
5888 ms 1310 ms agent-worker model write_recommendation ok
7198 ms 6 ms agent-worker checkpoint save_checkpoint ok
7204 ms 12 ms agent-worker tool record_recommendation ok
(An excerpt of the waterfall the runner prints for --case UW-006; the full trace has 14 spans.) The time went into two tool spans of exactly 2,000 ms each, the timeout, with 500 ms between them. Everything else is as fast as on any healthy run. Total: 7,216 ms, slightly over the 7-second latency threshold. The recommendation is request_info, with roof_assessment listed as missing.
Step 4, what people received. The audit events of the affected runs show recommendation_recorded with request_info, and later human_decision_recorded: for run-7f3a the underwriter agreed at 14:45:19 and asked the broker for an inspection report. The business impact is clear: five submissions got a slower, less informed answer, and none got a wrong one.
Diagram: From the page at 14:10 to the root cause, in the order of the questions
Read left to right. Each step reads the cheapest record that answers the next question. The alert names the promise; the logs name the runs and the step; one trace shows where the time went; the audit events say what the underwriters received. The cause is then tested against its definition.
- 1 · Alert
- Fast burn page: The alert names the SLIs that are burning; the controlled SLI is unaffected.
- 2 · Logs
- ERROR, category tool: A filtered search shows the same tool failing in every affected run.
- 3 · One trace
- run-7f3a: Two tool spans of 2000 ms with a 500 ms backoff, then a controlled fallback.
- 4 · Audit events
- What people received: Recorded recommendations and the underwriter's decisions show the business impact.
- 5 · Cause
- Outside and inside: The provider stopped answering; our timeout and retry settings pushed runs over 7 seconds.
Before AI. This is the same order a Java team uses for a slow endpoint: the latency alert, the error logs filtered by downstream, one distributed trace. The agent adds step 4. With an agent, "what did the users receive" is not obvious from status codes, because a controlled fallback returns success. The audit events say what the underwriter actually saw.
2. Root cause#
A root cause is the condition that, if it had not been present, would have prevented the failure. Each candidate is tested against that definition.
- "The agent was slow." A symptom. Slow because of what?
- "
get_roof_assessmenttimed out." Closer, and still a symptom of something else. - "The roof-data provider did not answer between 14:00 and 14:45." This is the triggering cause. The example cannot see inside the provider: we do not own its telemetry, only the worker's client spans and logs. Its own operator would find the deeper cause.
- "Our timeout and retry settings turned a provider outage into runs over the latency threshold." This is a contributing cause on our side: two attempts of 2,000 ms plus a backoff leave little room in a 7-second promise.
Two causes, one outside, one inside, is a common result. The inside one is the one the team can change. Whether to shorten the timeout, try once instead of twice, or relax the latency promise during a known provider outage is a reliability decision, and it belongs to reliable.sammuthu.com. This page's job was to produce the evidence that makes the decision possible.
Common confusion. The agent's own explanation of the run is not a root cause analysis. The recommendation ofrun-7f3asays the provider did not answer; that is a claim the agent wrote. The spans with statuserrorand duration 2,000 ms are evidence. A good postmortem cites the evidence, not the agent.
What this is not. Deciding whether the agent's behavior during the incident was acceptable is evaluation, owned by eval.sammuthu.com. Here the records say what happened; the eval site says whether that was good enough.
3. The timeline and the postmortem#
An incident timeline is the list of what happened and what people did, with times, built from the records rather than from memory. For this incident it reads:
| Time (simulated) | Event | Source |
|---|---|---|
| 14:00:00 | Roof-data provider stops answering | Inferred: the first run after 14:00 logs the first timeout; we cannot see inside the provider |
| 14:07:12 | run-7f3a arrives for SUB-1006 | Gateway access log |
| 14:07:14, 14:07:17 | Attempts 1 and 2 of get_roof_assessment time out | Worker ERROR logs, tool spans |
| 14:07:19 | request_info recorded for the underwriter | Audit event recommendation_recorded |
| 14:10:00 | Fast-burn page on latency and evidence | Alerting |
| 14:20:00 | Slow-burn rule also fires | Alerting |
| 14:45:00 | Provider answers again | Inferred: runs after the window get answers again |
| 14:45:19 | Underwriter decides run-7f3a: agrees, asks the broker for an inspection report | Audit event human_decision_recorded |
A postmortem is the written review after the incident: the timeline, the impact, the causes, what went well, what went badly, and the actions, each with an owner. It is blameless by convention: it asks how the system allowed the failure, not who caused it. For this incident, "what went well" has a real entry: the fallback kept every run in a controlled state, and the controlled SLI proved it. "What went badly" too: the latency promise broke for every run during the outage, and an earlier alert on the available SLI at 13:10 paged for a single denied write that was a security matter, not an outage.
4. From a failed run to a golden case#
The last step is the one that makes the next incident smaller. The failed run is turned into a test.
A golden case is a test case whose expected result people agreed on: an input, the expected output, and who agreed. It lives in a versioned dataset that every change to the agent is checked against. Golden cases and their grading belong to eval.sammuthu.com. Promotion is the hand-over this site owns: taking a production run, with its snapshot from Replay, and turning it into such a case.
The run cannot be copied as it is. Four things must happen first:
- Take the snapshot, including the tool responses, or the case cannot reproduce what the provider did.
- Redact the input: the broker's email and the building value must not leave production in a test fixture.
- Set the expected result, agreed by a person with the authority to judge it, and record their role.
- Record the policy version the case was decided under.
The example's promote.py refuses to produce a ready case until all four are done; anything missing is listed, and the dataset version does not move. With all four, the runner prints the case:
cd "$TMPDIR/uw-observe-incidents" && /opt/homebrew/bin/python3 run_observe.py --show promote
== promote
{
"id": "UW-P-7f3a",
"title": "Promoted from production: UW-006 run run-7f3a",
"policy_version": "2026.2",
"input": {
"source_email": "From: [EMAIL]\nTo: [EMAIL]\nSubject: Quote request SUB-1006\n\nPlease quote property cover for a masonry office, built 2009, 15,000 sq ft, sprinklered. Roof is 12 years old. Wind zone B. No losses.",
"submission": {
"submission_id": "SUB-1006",
"occupancy": "office",
"construction": "masonry",
"year_built": 2009,
"area_sq_ft": 15000,
"building_value_usd": "[AMOUNT]",
"sprinklered": true,
"roof_age_years": 12,
"wind_zone": "B",
"prior_losses_5y": 0
}
},
"world": {
"roof_assessment": {
"status": "timeout",
"failing_attempts": 2
}
},
"expected": {
"recommendation": "request_info",
"citations": [],
"missing": [
"roof_assessment"
],
"requires_human": true,
"roof_attempts": 2
},
"provenance": {
"labelled_by": "platform engineer",
"role_is_synthetic": true,
"date": "2026-09-14",
"synthetic": true,
"promoted_from": {
"run_id": "run-7f3a",
"trace_id": "7540bd55d4e55ccb8f04f3d81880717a",
"submission_id": "SUB-1006"
},
"redaction": [
"email",
"money"
]
}
}
ready True · dataset uw-regression 1.4.0 -> 1.5.0 · missing nothing
Read the case from the bottom up. provenance.promoted_from keeps the run_id, trace_id and submission_id, so anyone can go from the test back to the production evidence. redaction says which rules ran; the email addresses and the building value are now placeholders. expected was set by a person in the role of platform engineer; the role is synthetic, as everything here. world records that the provider failed twice, which is what makes the case reproduce the incident. And the dataset moves from 1.4.0 to 1.5.0: a new case is an addition, a minor version, so a later report can say which dataset version a result was measured against.
The example's dataset already held a hand-written case UW-006. The promoted case is different in kind: it comes from a real (synthetic) production run, with its recorded answers, and it carries the evidence trail. A dataset that grows from production incidents tests the failures that actually happen.
| Choice in the lab | If left out | What the lab shows |
|---|---|---|
| Snapshot parts (tool responses) | The case cannot reproduce the timeout | "missing: tool responses" and no version bump |
| Redaction rules | The broker's email stays in the input | "missing: redaction" |
| Expected result | Nobody agreed what is right | "missing: expected result" |
| Labelled by (a role) | No accountable judgement | "missing: labelled_by" |
Diagram: Promotion: a production run becomes a versioned golden case
Read left to right. The snapshot supplies what the run read. Redaction removes what must not leave production. A person sets the expected result and is recorded by role. Only then is the case added and the dataset version moves. If any step is missing, the case is not ready and the version stays.
- 1 · Snapshot
- Run and its answers: The input and every outside answer of run-7f3a, from the replay snapshot.
- 2 · Redact
- Email and money masked: Sensitive values are replaced by placeholders before the case leaves production.
- 3 · Agree
- Expected result: A person with authority agrees on what should have happened; the role is recorded.
- 4 · Version
- UW-P-7f3a added: The case joins the regression dataset and the minor version moves.
The same idea for a coding agent. A coding-agent session that broke the build is promoted the same way: the commit, the instruction files, the tool outputs and the failing CI log become a fixture, with the expected outcome agreed by a reviewer. Session CA-001, whose first test run failed before the agent fixed it, is a candidate. How such fixtures gate a pipeline is the subject of diff-to-deploy.sammuthu.com.
Detection time, and where it came from
The provider stopped answering at 14:00. The first page came at 14:10. Where did ten minutes go? The first affected run arrived after 14:00, waited through two timeouts, and finished as a bad event; the alert rules are evaluated every five minutes; the fast rule needs both its windows above the threshold. On a busy service the first bad events arrive within seconds, and detection is bounded by the evaluation interval and the short window. On a quiet one, like the simulated day with a run every half hour or so, detection waits for traffic.
Two improvements are on the observability side. A dependency SLI fed by synthetic probes of the provider detects the outage without waiting for a customer run. And an alert on the worker's client-side error rate for one tool, with a minimum number of calls, pages on the dependency directly, with a runbook that starts at the right place. Both are measurements; whether to fail over, shorten timeouts or pause intake is reliability work.
Rules for promotion at scale
Promoting every failed run floods the dataset with near-duplicates. A few rules keep it useful.
- Promote by failure class, not by run. Five runs failed the same way during the outage; one case represents them. Tag it with the incident id.
- Keep the expected result honest. The expected result is what should have happened, not what did happen. For
UW-006they coincide, because the fallback was correct. For a run where the agent was wrong, the expected result is the correction, set by someone with authority. - Re-check after policy changes. A golden case is decided under one
policy_version. When the fixture changes, cases decided under the old version are reviewed, not silently kept. - Respect retention. A promoted case outlives the trace it came from. Redaction before promotion is what makes that acceptable.
Checklist#
- Each alert names the SLI and links a runbook that starts with the right log filter.
- During an incident, the order is: alert, filtered logs, one whole trace, audit events.
- At least one SLI distinguishes "ended in a controlled state" from "ended well", so that impact is clear.
- The timeline is built from records with times and sources, not from memory.
- The root cause is tested against "if this had not been present, would the failure have been prevented?"
- The postmortem cites spans and logs, not the agent's own explanation.
- A failed run is promoted with its snapshot, redacted, with an agreed expected result, a role and a policy version.
- The dataset version changes with every promotion, and results name the version they were measured on.
In one breath#
An incident is read in the order of its questions: the alert says which promise is breaking, filtered logs say which runs and which step, one trace says why, and the audit events say what the underwriters received. On the simulated day the root cause was outside (the provider stopped answering) and a contributing cause was inside (two 2-second attempts in a 7-second promise). The failed run then becomes UW-P-7f3a, a redacted, versioned golden case that every future change must pass.
Where this leads#
You can now go from a page at 14:10 to a cause, a written timeline, and a test that keeps the failure from coming back unnoticed. One signal was missing from this incident: what the runs cost. A run that loops can be healthy, fast enough and controlled, and still spend twenty times what it should. That is Cost.