The platform team built a dashboard for the underwriting agent on its first day in production. It has twenty-two charts: requests per minute, queue depth, tokens per model, error counts per tool, p50 and p95 of every span name, memory of the worker, retries per hour. It is complete and nobody can read it. On the simulated day the roof-data provider stops answering at 14:00. Six charts move. Which of them means "the underwriters are getting worse service"? Two charts moved at 13:05 as well, for a different reason. Was that worse service?
The question every chart was trying to answer is simpler than the dashboard: are the runs turning out the way we promised, and if not, does a person need to act now? This page builds that answer from the records the earlier pages produced: one number that measures the promise, a target for it, a budget for the misses, and an alert rule that pages a person only when the budget burns fast enough to matter. The model recommends; a person decides, and the promise is about the agent getting a recommendation to that person in a controlled state. One note before any number appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
1. A dashboard answers a question, or it is decoration#
A dashboard is a set of charts on one screen. The useful ones are built backwards: first the question, then the one or two charts that answer it. "Is the agent serving the underwriters?" needs very few charts. "Why is the worker slow?" needs others, and they belong on a second screen that the first one links to.
Before any formula, write the promise in plain English. For the underwriting agent the platform team settles on four sentences:
| Promise (one sentence) | How a run is judged good |
|---|---|
| The recommendation is ready quickly | The run ends within 7 seconds (simulated) of arriving at the gateway |
| The run always ends in a controlled state | A recommendation, a controlled fallback such as request_info, or a stop at the step limit |
| The run got the evidence it needed | The roof-data provider answered |
| The underwriter can open it | A recommendation was stored for the cockpit |
Each row turns a vague wish into a rule that classifies every run as good or bad. That classification is the raw material for everything else on this page. These thresholds are illustrative, chosen for the example.
Plain English. A dashboard that cannot be read in ten seconds during an incident is a report, not a dashboard. Put the promises on the first screen and the diagnostics one click away.
2. SLI, SLO and error budget#
The Google SRE book defines the three words. A service level indicator (SLI) is "a carefully defined quantitative measure of some aspect of the level of service that is provided". In this example every SLI is a ratio: good runs divided by all runs. A service level objective (SLO) is "a target value or range of values for a service level that is measured by an SLI": here, 99 percent of runs good, over 30 days. A service level agreement (SLA) is a contract with consequences; the same book's test is that "if there is no explicit consequence, then you are almost certainly looking at an SLO". This site does not discuss SLAs further.
The error budget follows from the objective. If 99 percent must be good, 1 percent may be bad. The example assumes 48 runs a day, so a 30-day period has 1,440 runs and an error budget of 14.4 bad runs. The SRE workbook says it plainly: "The error budget gives the number of allowed bad events."
The four SLIs over the simulated day:
cd "$TMPDIR/uw-observe-slos" && /opt/homebrew/bin/python3 run_observe.py --show slo
== slo
latency SLI 0.8958 (43/48) · objective 0.99 · 30-day budget 14.4 bad runs · this day spent 34.7% · first alert: fast 14:10:00, slow 14:20:00
controlled SLI 1.0000 (48/48) · objective 0.99 · 30-day budget 14.4 bad runs · this day spent 0.0% · first alert: fast -, slow -
evidence SLI 0.8958 (43/48) · objective 0.99 · 30-day budget 14.4 bad runs · this day spent 34.7% · first alert: fast 14:10:00, slow 14:20:00
available SLI 0.9792 (47/48) · objective 0.99 · 30-day budget 14.4 bad runs · this day spent 6.9% · first alert: fast 13:10:00, slow -
at 14:10:00 fast burn: {'fires': True, 'long_burn': 25.0, 'short_burn': 100.0}
at 14:10:00 slow burn: {'fires': False, 'long_burn': 3.7037, 'short_burn': 50.0}
Read the four lines as four different stories about the same day.
- latency and evidence: five runs fell inside the provider outage between 14:00 and 14:45. Each waited through two timeouts, so each was slow and each lacked the roof assessment. Five bad runs out of a 30-day budget of 14.4 is 34.7 percent of the month's budget, spent in 45 minutes.
- controlled: every run, including those five, ended in a controlled state.
UW-006ended withrequest_infofor the underwriter, which is exactly what the agent is designed to do when evidence is missing. The fallback worked. - available: one run, at 13:05, stored nothing for the underwriter. That is
UW-503, the caller without the write permission: a security log line saysdenied, and no recommendation exists. One bad run is 6.9 percent of the budget.
All of these numbers come from one synthetic day and are illustrative.
In our insurance agent. The SLO set makes a design choice visible. The controlled SLI stayed at 100 percent through the outage because the agent falls back instead of failing. Without that SLI, the dashboard would show the outage as pure failure. With it, the team can say: service was slower and less informed for 45 minutes, and it never left an underwriter without a controlled answer.
What this is not. Choosing the target and deciding what happens when the budget is gone (freeze releases, add a fallback, change a timeout) belongs to reliability engineering: see reliable.sammuthu.com. This page measures and alerts. Timeouts and retries themselves are reliability decisions too; the records here only show what they did.
Diagram: From runs to a page: the chain behind one alert
Read left to right. Each run is judged good or bad by one rule. The share of good runs is the SLI. The SLO sets the target and so the error budget. Burn rate says how fast a window spends that budget. Only a fast burn in two windows pages a person, with a runbook. Numbers are from the synthetic day.
- 1 · Each run
- Good or bad: Each run is classified against one written rule, for example: ended within 7 seconds.
- 2 · SLI
- Share of good runs: The measured ratio of good runs to all runs.
- 3 · SLO and budget
- Target 99 percent: The objective over 30 days; the misses it allows are the error budget.
- 4 · Burn rate
- How fast it burns: Error rate in a window divided by the error rate the objective allows.
- 5 · Alert
- Page with a runbook: Both windows above 14.4: a person is paged, with the first three checks.
3. Burn rate: how fast the budget goes#
A budget of 14.4 bad runs a month does not say when to wake someone. Five bad runs in one afternoon and five bad runs spread over a month spend the same budget, and only the first needs a person now. The measure that separates them is the burn rate: in the workbook's words, "how fast, relative to the SLO, the service consumes the error budget". A burn rate of 1 spends exactly the whole budget over the period. A burn rate of 14.4 spends a month's budget in about two days.
The calculation is the error rate in a window divided by the error rate the SLO allows:
def burn_rate(good: int, total: int, objective: float) -> float:
"""Error rate in a window divided by the error rate the SLO allows. 0 when the window is empty."""
if not total:
return 0.0
return ((total - good) / total) / (1 - objective)
At 14:10, the last hour holds four runs, one of them bad: an error rate of 25 percent, divided by the allowed 1 percent, gives a burn rate of 25. The last five minutes hold one run, and it is bad: a burn rate of 100.
4. Alerts a person can act on#
An alert is a rule that tells a person to act now. The test of an alert is the action: if the person who is paged cannot do anything useful, the alert trains them to ignore alerts. That is alert fatigue, and it is how real incidents get missed.
The rule this example uses comes from the SRE workbook's chapter on alerting on SLOs: a multi-window, multi-burn-rate alert. Two conditions must hold at once: the burn rate over a long window is above a threshold, and the burn rate over a short window (one twelfth of the long one) is above the same threshold. The long window proves the problem is significant; the short window proves it is still happening, so the alert stops soon after the problem does.
| Rule | Long window | Short window | Burn-rate threshold | Action |
|---|---|---|---|---|
| fast burn | 1 hour | 5 minutes | 14.4 | page |
| slow burn | 6 hours | 30 minutes | 6.0 | page |
The workbook gives these two rows for a 99.9 percent SLO, plus a third (burn rate 1 over 3 days, a ticket) that a one-day simulation cannot show. The example keeps the numbers to show the arithmetic, applied to a 99 percent objective; they are not a recommendation for this workload.
At 14:10 the fast rule fires: long burn 25 and short burn 100, both above 14.4. The slow rule does not fire yet: its 6-hour window holds enough good morning runs that the burn there is 3.7. It fires at 14:20. The on-call engineer is paged ten minutes into the outage, and the page says which promise is breaking.
| What to set in the lab | What changes | What to look for |
|---|---|---|
| The SLI (latency, controlled, evidence, available) | Which runs count as bad | The outage moves latency and evidence; only UW-503 moves available |
| The objective (for example 0.99 or 0.999) | The budget and every burn rate | A stricter objective pages earlier and more often |
| The window pair | How much history the rule needs | Shorter windows page faster and flap more |
Every alert should carry a runbook link: a short page that says what the alert means, the first three things to look at, and who to call. For the latency and evidence alerts of the afternoon, the first step is the log explorer filtered to category = tool and severity ERROR; the second is one trace, such as run-7f3a, on Tracing; the third is the roof-data provider's status. Incidents follows that path in full.
Production pitfall. TheavailableSLI fired a fast-burn page at 13:10 for one run. On a day of 48 runs, one bad run in an hour with three runs is a burn rate far above 14.4. The workbook's advice for low-traffic services applies: generate artificial traffic, group small services into one monitored unit, or lower the SLO or lengthen the window if the stakeholders agree. For theUW-503case the right reader is the security team, through the security log, not the on-call engineer through a page.
Before AI. None of this is new. A Spring service behind a load balancer had the same SLIs (latency and availability) and the same alert rules. What the agent adds is the second kind of good: a run can be fast and available and still end without the evidence it needed. That is why theevidenceandcontrolledSLIs exist here, and why Quality adds a third kind next.
Diagram: Two windows must agree before anyone is paged
Read each lane on its own. The long window proves the problem is big enough to matter. The short window proves it is still going on. The fast rule pages within minutes of a sharp outage; the slow rule catches a smaller leak over hours. A single bad run on a quiet service can still trip the fast rule, which is why low-traffic SLIs need a guard.
- Fast burn
- 1 hour above 14.4: Over the last hour the budget burns more than 14.4 times too fast.
- 5 minutes above 14.4: The last five minutes burn too fast as well, so the problem is current.
- Slow burn
- 6 hours above 6: Over six hours the budget burns more than six times too fast.
- 30 minutes above 6: The last half hour confirms the leak continues.
- Quiet traffic
- One bad run: With three runs in an hour, one bad run is a burn rate far above 14.4.
- Add a guard: Require a minimum number of events, add synthetic traffic, or route to a ticket or the security team.
The arithmetic behind the thresholds
With objective o and period P, the budget fraction spent by a window of length w at burn rate b is b × w / P. For the fast rule (b = 14.4, w = 1 hour, P = 30 days = 720 hours) that is 14.4 / 720 = 2 percent of the budget, which is where the workbook's "about 2 percent" comes from. For the slow rule (b = 6, w = 6 hours) it is 36 / 720 = 5 percent.
Burn rate is a ratio of rates, so it is independent of traffic volume, which is why it works across services. Its variance is not: with n events in a window, one bad event moves the burn rate by 1 / (n × (1 − o)). At n = 3 and o = 0.99 that is about 33, which is why a single bad run pages on a quiet service. Minimum-event guards (do not evaluate a window with fewer than k events) are a common addition; they trade detection time for fewer false pages.
The example evaluates the rules every five minutes over the simulated events (slo.evaluate_day). A production system evaluates them in the metrics store from counters of good and total events, which is cheaper and does not depend on keeping every run.
Choosing SLIs for an agent
The SRE book warns that "you shouldn't use every metric you can track in your monitoring system as an SLI" and that a handful is usually enough. For an agent, a useful handful separates three questions.
- Did the system work? Availability and latency, as for any service.
- Did the run end safely? The controlled-state SLI. It is the one that makes fallbacks visible as success rather than failure.
- Did the run have what it needed? An evidence SLI per critical dependency. It catches a degraded dependency even when the agent covers for it.
Quality ("was the recommendation right?") is measured too, but with labels that arrive later and with human review: see Quality. Mixing it into a fast-burn alert is usually a mistake, because the signal arrives hours after the run.
The same idea for a coding agent. For a coding-agent session such as CA-001, the same three questions give three SLIs: the session finished, it ended with passing tests or a clear stop, and the CI run on its pull request succeeded. A pipeline's own SLOs are on diff-to-deploy.sammuthu.com.
Checklist#
- The first screen of the dashboard answers "are we keeping our promises?" and links to diagnostics.
- Each promise is one sentence and one rule that classifies every run as good or bad.
- There is an SLI for a controlled end, so a working fallback is not counted as an outage.
- Each SLO has a period and an error budget in events, written down.
- Alerts use burn rate over a long and a short window, not single failures or raw error counts.
- Every alert has a runbook link with the first three checks.
- Low-traffic SLIs have a guard (minimum events, synthetic traffic, or a ticket instead of a page).
- Security-relevant failures go to the security log's readers, not only to the on-call page.
In one breath#
A dashboard should answer one question first: are we keeping our promises? Write each promise as a rule that marks every run good or bad, measure the share of good runs as an SLI, set an SLO for it, and treat the misses the SLO allows as an error budget. Page a person only when the budget burns fast over a long and a short window at once, as the roof-data outage did at 14:10, and send everything else to a ticket or to the team that can act on it.
Where this leads#
You can now turn records into measured promises and alerts that a person can act on. Every SLI on this page judged runs by time, availability and evidence. None of them asked whether the recommendation was right. A run can be fast, cheap, in a controlled state and still wrong. The next page adds that third kind of signal: Quality.