All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Operations · 10

Dashboards, SLIs, SLOs, error budgets and alerts a person can act on

From a wall of charts to one measured promise. Indicators, objectives, the error budget, burn rate, and the difference between an alert and noise.

One measured promise
An SLI is the share of good runs. An SLO is the target for it. The gap is the error budget.
Burn, not blips
An alert fires when the budget burns too fast in two windows at once, not when one run fails.
14:10 on the simulated day
The roof-data outage pages the on-call engineer ten minutes in. A single denied run earlier also pages. One of those alerts deserved it.

The platform team built a dashboard for the underwriting agent on its first day in production. It has twenty-two charts: requests per minute, queue depth, tokens per model, error counts per tool, p50 and p95 of every span name, memory of the worker, retries per hour. It is complete and nobody can read it. On the simulated day the roof-data provider stops answering at 14:00. Six charts move. Which of them means "the underwriters are getting worse service"? Two charts moved at 13:05 as well, for a different reason. Was that worse service?

The question every chart was trying to answer is simpler than the dashboard: are the runs turning out the way we promised, and if not, does a person need to act now? This page builds that answer from the records the earlier pages produced: one number that measures the promise, a target for it, a budget for the misses, and an alert rule that pages a person only when the budget burns fast enough to matter. The model recommends; a person decides, and the promise is about the agent getting a recommendation to that person in a controlled state. One note before any number appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.

1. A dashboard answers a question, or it is decoration#

A dashboard is a set of charts on one screen. The useful ones are built backwards: first the question, then the one or two charts that answer it. "Is the agent serving the underwriters?" needs very few charts. "Why is the worker slow?" needs others, and they belong on a second screen that the first one links to.

Before any formula, write the promise in plain English. For the underwriting agent the platform team settles on four sentences:

Promise (one sentence)How a run is judged good
The recommendation is ready quicklyThe run ends within 7 seconds (simulated) of arriving at the gateway
The run always ends in a controlled stateA recommendation, a controlled fallback such as request_info, or a stop at the step limit
The run got the evidence it neededThe roof-data provider answered
The underwriter can open itA recommendation was stored for the cockpit

Each row turns a vague wish into a rule that classifies every run as good or bad. That classification is the raw material for everything else on this page. These thresholds are illustrative, chosen for the example.

Plain English. A dashboard that cannot be read in ten seconds during an incident is a report, not a dashboard. Put the promises on the first screen and the diagnostics one click away.

2. SLI, SLO and error budget#

The Google SRE book defines the three words. A service level indicator (SLI) is "a carefully defined quantitative measure of some aspect of the level of service that is provided". In this example every SLI is a ratio: good runs divided by all runs. A service level objective (SLO) is "a target value or range of values for a service level that is measured by an SLI": here, 99 percent of runs good, over 30 days. A service level agreement (SLA) is a contract with consequences; the same book's test is that "if there is no explicit consequence, then you are almost certainly looking at an SLO". This site does not discuss SLAs further.

The error budget follows from the objective. If 99 percent must be good, 1 percent may be bad. The example assumes 48 runs a day, so a 30-day period has 1,440 runs and an error budget of 14.4 bad runs. The SRE workbook says it plainly: "The error budget gives the number of allowed bad events."

The four SLIs over the simulated day:

cd "$TMPDIR/uw-observe-slos" && /opt/homebrew/bin/python3 run_observe.py --show slo
== slo
latency    SLI 0.8958 (43/48) · objective 0.99 · 30-day budget 14.4 bad runs · this day spent 34.7% · first alert: fast 14:10:00, slow 14:20:00
controlled SLI 1.0000 (48/48) · objective 0.99 · 30-day budget 14.4 bad runs · this day spent 0.0% · first alert: fast -, slow -
evidence   SLI 0.8958 (43/48) · objective 0.99 · 30-day budget 14.4 bad runs · this day spent 34.7% · first alert: fast 14:10:00, slow 14:20:00
available  SLI 0.9792 (47/48) · objective 0.99 · 30-day budget 14.4 bad runs · this day spent 6.9% · first alert: fast 13:10:00, slow -
  at 14:10:00 fast burn: {'fires': True, 'long_burn': 25.0, 'short_burn': 100.0}
  at 14:10:00 slow burn: {'fires': False, 'long_burn': 3.7037, 'short_burn': 50.0}

Read the four lines as four different stories about the same day.

All of these numbers come from one synthetic day and are illustrative.

In our insurance agent. The SLO set makes a design choice visible. The controlled SLI stayed at 100 percent through the outage because the agent falls back instead of failing. Without that SLI, the dashboard would show the outage as pure failure. With it, the team can say: service was slower and less informed for 45 minutes, and it never left an underwriter without a controlled answer.
What this is not. Choosing the target and deciding what happens when the budget is gone (freeze releases, add a fallback, change a timeout) belongs to reliability engineering: see reliable.sammuthu.com. This page measures and alerts. Timeouts and retries themselves are reliability decisions too; the records here only show what they did.

Diagram: From runs to a page: the chain behind one alert

Read left to right. Each run is judged good or bad by one rule. The share of good runs is the SLI. The SLO sets the target and so the error budget. Burn rate says how fast a window spends that budget. Only a fast burn in two windows pages a person, with a runbook. Numbers are from the synthetic day.

  1. 1 · Each run
    1. Good or bad: Each run is classified against one written rule, for example: ended within 7 seconds.
  2. 2 · SLI
    1. Share of good runs: The measured ratio of good runs to all runs.
  3. 3 · SLO and budget
    1. Target 99 percent: The objective over 30 days; the misses it allows are the error budget.
  4. 4 · Burn rate
    1. How fast it burns: Error rate in a window divided by the error rate the objective allows.
  5. 5 · Alert
    1. Page with a runbook: Both windows above 14.4: a person is paged, with the first three checks.

3. Burn rate: how fast the budget goes#

A budget of 14.4 bad runs a month does not say when to wake someone. Five bad runs in one afternoon and five bad runs spread over a month spend the same budget, and only the first needs a person now. The measure that separates them is the burn rate: in the workbook's words, "how fast, relative to the SLO, the service consumes the error budget". A burn rate of 1 spends exactly the whole budget over the period. A burn rate of 14.4 spends a month's budget in about two days.

The calculation is the error rate in a window divided by the error rate the SLO allows:

def burn_rate(good: int, total: int, objective: float) -> float:
    """Error rate in a window divided by the error rate the SLO allows. 0 when the window is empty."""
    if not total:
        return 0.0
    return ((total - good) / total) / (1 - objective)

At 14:10, the last hour holds four runs, one of them bad: an error rate of 25 percent, divided by the allowed 1 percent, gives a burn rate of 25. The last five minutes hold one run, and it is bad: a burn rate of 100.

4. Alerts a person can act on#

An alert is a rule that tells a person to act now. The test of an alert is the action: if the person who is paged cannot do anything useful, the alert trains them to ignore alerts. That is alert fatigue, and it is how real incidents get missed.

The rule this example uses comes from the SRE workbook's chapter on alerting on SLOs: a multi-window, multi-burn-rate alert. Two conditions must hold at once: the burn rate over a long window is above a threshold, and the burn rate over a short window (one twelfth of the long one) is above the same threshold. The long window proves the problem is significant; the short window proves it is still happening, so the alert stops soon after the problem does.

RuleLong windowShort windowBurn-rate thresholdAction
fast burn1 hour5 minutes14.4page
slow burn6 hours30 minutes6.0page

The workbook gives these two rows for a 99.9 percent SLO, plus a third (burn rate 1 over 3 days, a ticket) that a one-day simulation cannot show. The example keeps the numbers to show the arithmetic, applied to a 99 percent objective; they are not a recommendation for this workload.

At 14:10 the fast rule fires: long burn 25 and short burn 100, both above 14.4. The slow rule does not fire yet: its 6-hour window holds enough good morning runs that the burn there is 3.7. It fires at 14:20. The on-call engineer is paged ten minutes into the outage, and the page says which promise is breaking.

What to set in the labWhat changesWhat to look for
The SLI (latency, controlled, evidence, available)Which runs count as badThe outage moves latency and evidence; only UW-503 moves available
The objective (for example 0.99 or 0.999)The budget and every burn rateA stricter objective pages earlier and more often
The window pairHow much history the rule needsShorter windows page faster and flap more

Every alert should carry a runbook link: a short page that says what the alert means, the first three things to look at, and who to call. For the latency and evidence alerts of the afternoon, the first step is the log explorer filtered to category = tool and severity ERROR; the second is one trace, such as run-7f3a, on Tracing; the third is the roof-data provider's status. Incidents follows that path in full.

Production pitfall. The available SLI fired a fast-burn page at 13:10 for one run. On a day of 48 runs, one bad run in an hour with three runs is a burn rate far above 14.4. The workbook's advice for low-traffic services applies: generate artificial traffic, group small services into one monitored unit, or lower the SLO or lengthen the window if the stakeholders agree. For the UW-503 case the right reader is the security team, through the security log, not the on-call engineer through a page.
Before AI. None of this is new. A Spring service behind a load balancer had the same SLIs (latency and availability) and the same alert rules. What the agent adds is the second kind of good: a run can be fast and available and still end without the evidence it needed. That is why the evidence and controlled SLIs exist here, and why Quality adds a third kind next.

Diagram: Two windows must agree before anyone is paged

Read each lane on its own. The long window proves the problem is big enough to matter. The short window proves it is still going on. The fast rule pages within minutes of a sharp outage; the slow rule catches a smaller leak over hours. A single bad run on a quiet service can still trip the fast rule, which is why low-traffic SLIs need a guard.

  1. Fast burn
    1. 1 hour above 14.4: Over the last hour the budget burns more than 14.4 times too fast.
    2. 5 minutes above 14.4: The last five minutes burn too fast as well, so the problem is current.
  2. Slow burn
    1. 6 hours above 6: Over six hours the budget burns more than six times too fast.
    2. 30 minutes above 6: The last half hour confirms the leak continues.
  3. Quiet traffic
    1. One bad run: With three runs in an hour, one bad run is a burn rate far above 14.4.
    2. Add a guard: Require a minimum number of events, add synthetic traffic, or route to a ticket or the security team.
The arithmetic behind the thresholds

With objective o and period P, the budget fraction spent by a window of length w at burn rate b is b × w / P. For the fast rule (b = 14.4, w = 1 hour, P = 30 days = 720 hours) that is 14.4 / 720 = 2 percent of the budget, which is where the workbook's "about 2 percent" comes from. For the slow rule (b = 6, w = 6 hours) it is 36 / 720 = 5 percent.

Burn rate is a ratio of rates, so it is independent of traffic volume, which is why it works across services. Its variance is not: with n events in a window, one bad event moves the burn rate by 1 / (n × (1 − o)). At n = 3 and o = 0.99 that is about 33, which is why a single bad run pages on a quiet service. Minimum-event guards (do not evaluate a window with fewer than k events) are a common addition; they trade detection time for fewer false pages.

The example evaluates the rules every five minutes over the simulated events (slo.evaluate_day). A production system evaluates them in the metrics store from counters of good and total events, which is cheaper and does not depend on keeping every run.

Choosing SLIs for an agent

The SRE book warns that "you shouldn't use every metric you can track in your monitoring system as an SLI" and that a handful is usually enough. For an agent, a useful handful separates three questions.

  • Did the system work? Availability and latency, as for any service.
  • Did the run end safely? The controlled-state SLI. It is the one that makes fallbacks visible as success rather than failure.
  • Did the run have what it needed? An evidence SLI per critical dependency. It catches a degraded dependency even when the agent covers for it.

Quality ("was the recommendation right?") is measured too, but with labels that arrive later and with human review: see Quality. Mixing it into a fast-burn alert is usually a mistake, because the signal arrives hours after the run.

The same idea for a coding agent. For a coding-agent session such as CA-001, the same three questions give three SLIs: the session finished, it ended with passing tests or a clear stop, and the CI run on its pull request succeeded. A pipeline's own SLOs are on diff-to-deploy.sammuthu.com.

Checklist#

In one breath#

A dashboard should answer one question first: are we keeping our promises? Write each promise as a rule that marks every run good or bad, measure the share of good runs as an SLI, set an SLO for it, and treat the misses the SLO allows as an error budget. Page a person only when the budget burns fast over a long and a short window at once, as the roof-data outage did at 14:10, and send everything else to a ticket or to the team that can act on it.

Where this leads#

You can now turn records into measured promises and alerts that a person can act on. Every SLI on this page judged runs by time, availability and evidence. None of them asked whether the recommendation was right. A run can be fast, cheap, in a controlled state and still wrong. The next page adds that third kind of signal: Quality.