At 14:07 on a Monday a broker's quote request reaches an insurer. An underwriting agent reads it, asks an outside provider for the roof's age, gets no answer, asks again, gets no answer again, and writes a recommendation for a human underwriter: ask the broker for an inspection report. The model recommends; a person decides. Thirty-eight minutes later the underwriter agrees.
The next morning three people ask about that run. The on-call engineer asks why it took more than seven seconds. The head of underwriting asks how many quotes the provider outage affected. An auditor, months later, asks who decided, on what basis, under which version of the guidelines.
The agent has an answer for all three. Its recommendation says, in fluent sentences, that the provider did not answer and that a person will review. That account is a claim: text the agent wrote about itself. It may be accurate. It cannot be checked against itself, it does not say how long anything took, it says nothing about the other runs of the day, and it was written before the underwriter decided anything.
Observability is the ability to answer questions about what a running system did from the records it left, without changing the system to ask. This site is about those records: what they are, how to make them, how to keep them safe, and what each one lets you prove. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
Three records that are evidence#
Telemetry is the records a running system emits about itself. It comes in a few kinds, called signals, and three of them carry most of the weight. Three sentences hold them apart, and every page of this site uses them in exactly this form:
A log records an event. A metric summarizes measurements. A trace connects the steps of one run.
Here is each of them for the run above, case UW-006, run run-7f3a.
| Record | One example from run-7f3a | The question it answers |
|---|---|---|
| Log | 14:07:14 ERROR get_roof_assessment failed: timeout after 2000 ms, attempt_number 1 | What happened here, at this moment? |
| Metric | Run duration on the simulated day: 48 runs, p50 3,107 ms, p95 7,346 ms | How often, how much, how long, across many runs? |
| Trace | 14 spans from the gateway to the stored recommendation, 7,216 ms, two failed 2,000 ms calls inside | Where did the time go in this one run, and which step called which? |
The on-call engineer's question is answered by the trace. The head of underwriting's by a metric (and the logs that list the affected runs). The auditor's by a fourth kind of record that this site treats with special care: an audit event, a record the business must be able to prove later, kept complete and unaltered for years. A diagnostic record may be dropped; an audit event may not.
Diagram: One run, four kinds of record, four questions
On the left is the agent's own account: a claim. To its right, the records that are evidence, each answering a different question about run-7f3a. The log is a point in time. The metric covers many runs. The trace connects the steps of this one run. The audit event proves the decision for years.
- The agent's account
- A claim: The recommendation's own text about what happened: may be accurate, cannot be checked against itself.
- Log
- Records an event: One event at one moment, with the facts that name it.
- Metric
- Summarizes measurements: Many measurements as one number over time.
- Trace
- Connects the steps: The steps of one run with their lengths and who called whom.
- Audit event
- Proves the decision: Who decided what, on what basis; never sampled, never changed.
Try it. Pick a question in the lab below. It shows which record answers it, the record itself, and why the other kinds cannot.
| Question | The record that answers it | Why the others cannot |
|---|---|---|
Why was run-7f3a slow? | The trace: two get_roof_assessment spans of 2,000 ms each | A log line has no length; a metric has no single run |
| How many runs hit a timeout today? | A metric (or a filtered log search): 10 timeouts in 5 runs | A trace covers one run |
| What happened at 14:07:14? | A log line | A metric has no message; a trace has no free text |
Who decided SUB-1006? | The audit event human_decision_recorded | Logs and traces may be sampled and expire |
Before AI. Think of a flight recorder. A pilot's account after landing is valuable, and nobody treats it as the record. The recorder holds the instrument readings over time (metrics), the cockpit events (logs), and the sequence of the flight (a trace, in effect). An investigation reads the recorder first and the account second. Every backend system built in the last twenty years has its own flight recorder; an agent needs one more than most, because its account of itself is so fluent.
The insurance example in one minute#
One example runs through every page. An insurer, never named, sells commercial property insurance through brokers. A broker's quote request arrives as an email; an intake step turns it into a submission, one JSON record with the building's facts. The underwriting agent reads the submission, asks a roof-data provider for an assessment, searches the insurer's underwriting guidance and similar past decisions, runs the deterministic rules of a versioned policy fixture, and writes a recommendation with reasons and citations. A human underwriter sees it in the underwriter cockpit and decides. The model recommends; a person decides.
The anchor case is UW-006: submission SUB-1006, run run-7f3a, the roof-data provider times out twice, and the agent falls back to request_info. Around it sit the other cases of the series (an obvious decline, a clean approval, a missing roof age, conflicting reports, an injected instruction in a retrieved passage, an underwriter override) and four of this site's own: an ambiguous recommendation (UW-501), a run that loops (UW-502), a caller without permission (UW-503) and a message delivered twice (UW-504).
The code that produces every record on this site is one folder of standard-library Python, examples/uw-observe/. It simulates a production day of 48 runs with a provider outage from 14:00 to 14:45, and writes logs, metrics, traces, audit events and security records for all of them.
Everything in the example is synthetic. The insurer, the cases, the rules, the fields, the provider, the times, the token counts and the costs are invented for teaching. Costs are in invented units, never a price. The agent is a scripted stand-in: no model is called. Nothing here reports the behavior of a real system or model.
Diagram: The running example: one quote request, from the broker's email to the underwriter's decision
Read left to right. The broker's email becomes a submission. The gateway starts the run and puts it on a queue. The underwriting agent gathers evidence, checks the rules and writes a recommendation. The underwriter decides in the cockpit. Each box writes records; the rest of the site follows them.
- 1 · Intake
- Broker's email: A quote request turned into a submission: one JSON record with the building's facts.
- 2 · Gateway and queue
- Run starts: The gateway assigns ids, starts the trace and puts the run on a queue.
- 3 · Underwriting agent
- Evidence, rules, recommendation: Roof assessment, guidance search, similar decisions, deterministic rules, a written recommendation.
- 4 · Underwriter
- A person decides: The underwriter reads the recommendation and decides; the decision is an audit event.
How to read this site#
The pages follow the order in which the questions come up once an agent runs in production. First the records themselves. Then what is new with a model in the loop, and the standard that names it. Then what to do with the records: store them, alert on them, judge quality with them, replay them, respond to incidents and watch cost. A reference part closes the site.
New to observability: read in order#
Foundations. The records and how they join.
Agent telemetry. What a model changes, and what must be protected.
Operations. Using the records.
Reference. Tools, scale, and the whole example.
Experienced with observability: jump to a question#
Each page recaps what it needs in a sentence and links to the full explanation. The rigorous part of each page (formulas with assumptions, edge cases, production trade-offs) is folded into blocks you can open.
| If your question is | Start at |
|---|---|
| What must a model, retrieval or tool span record? | GenAI telemetry, then OpenTelemetry |
| How do I keep the human decision provable for years? | Audit logs |
| Should we capture prompts in traces? | Sensitive content |
| Which backend stores what, and at what cost? | Backends |
| How do I alert on an agent without paging on every failure? | SLOs and alerts |
| How do I see that the agent got worse when nothing is failing? | Quality signals |
| Can I re-run a production run to debug it? | Replay |
| Why did one run cost twenty times the median? | Cost |
| Why did our metrics bill triple after adding a label? | Advanced |
Where this fits#
This site is one of a series on engineering software that contains a model. Each site owns one subject and links to the others where the subjects meet. This one owns evidence of what happened. It does not decide whether a behavior was acceptable (that is evaluation), how to recover from a failure (reliability), or what the agent may do (guardrails); it records what happened so that each of those can be checked.
Diagram: Where this site fits: evidence, between doing and judging
The left lane is the sites about what the agent does and how it is built. The middle is this site: the evidence of what happened. The right lane is the sites that read that evidence to judge or recover. Each site owns its subject and links at the seams; the tiles below link every site.
- Doing
- Instructions, context, memory, tools: How the agent is told, what it sees, what it remembers and what it can call.
- Harness, orchestration, messages: The runtime around the model, what runs next, and how agents talk.
- Seeing (this site)
- Observability Engineering: Logs, metrics, traces, audit events, replay and quality signals: evidence of what happened.
- Judging and recovering
- Evaluation: Whether behavior was acceptable: graders, golden sets, gates.
- Reliability and guardrails: Recovering from failures and enforcing checks.
- Shipping changes: Review and CI for changes made with coding agents.
What this is not. Whether the agent's recommendation was good enough is an evaluation question, answered on eval.sammuthu.com. This site produces the evidence evaluation reads: traces, labels, overrides, and production runs promoted into golden cases.
The same idea for a coding agent. A second, smaller example appears on several pages: a coding-agent session, CA-001, that reads the repository instructions, edits a file, runs the tests (which fail), fixes the code, runs them again (they pass) and opens a pull request for CI. Its records follow the same rules: its account of the session is a claim; its spans, logs and the CI result are evidence. The runtime of such an agent is the subject of harness.sammuthu.com; its path to production is diff-to-deploy.sammuthu.com.
What this site is not#
A map to adapt, not a standard, and not a measurement of any product. The page order and some names are one way of dividing the subject. Products and standards are described from their own documentation, with the date it was read, and none is called best. No price appears anywhere.
In one breath#
An agent's account of what it did is a claim; the records it leaves are evidence. A log records an event, a metric summarizes measurements, a trace connects the steps of one run, and an audit event proves later who decided what. This site builds each of them on one synthetic insurance run, run-7f3a, from a single line on stdout to an audit ledger, a replay and a golden case.
Where this leads#
Start where every system starts: a program printing a line. Log lines shows what turns that line into a record many machines and one engineer can search.