This is a knowledge site about keeping evidence of what agentic software did. It starts from one line printed to a terminal and climbs to structured logs, metrics, traces, audit events, redaction, service level objectives, quality signals, replay and cost. At each step it asks one question: what does this record let me prove about what happened? It is written for engineers and architects who run agents in production, and for experienced backend engineers meeting agents for the first time.
One example runs through every page. An insurer receives a broker's request for a quote. An underwriting agent reads it, checks the roof, looks up the guidance and writes a recommendation. A human underwriter decides. The model recommends; a person decides. The anchor run is case UW-006, in which the roof-data provider times out twice.
What it is and is not#
| It is | It is not |
|---|---|
| A ladder: one idea per step, each word in plain English with a tiny example before it is used | A standard, or the only way to divide the subject |
| One running example, a synthetic insurance agent and one simulated production day of 48 runs | A record of any real insurer, broker, building or decision |
Code that was run: standard-library Python in examples/uw-observe/, with its real output on the pages | A library to install, or a production implementation |
| A demonstration of mechanisms: how a span, a hash chain, a burn rate or a replay works | A measurement of any model or product. The agent is a scripted stand-in |
| Products and standards described from their own documentation, with the date it was read | A ranking or a buying guide. Nothing is called best, and no price is quoted |
| Costs in invented units to show the arithmetic | A statement about what any model or service costs |
An honest boundary, said once on the Start page, once on the insurance lab and here: this is an illustrative reconstruction for teaching. The cases, rules, fields, provider, times, token counts and costs are invented. Nothing here reports the behavior of a real system or model. The rules of the policy fixture (for example, a roof older than 25 years in wind zone C is not eligible) are synthetic and are not insurance practice.
What is original here and what is not#
Most ideas on this site were published by others first. What this site adds is the arrangement, the running example and the code.
| Part | Status |
|---|---|
| Logs, metrics and traces as the three main signals | Common practice; the OpenTelemetry documentation describes the same signals |
| Spans, span kinds, span links, span events, context propagation | From OpenTelemetry and W3C Trace Context, read in their documentation |
traceparent format | W3C Trace Context (Recommendation, November 2021) |
| SLI, SLO, error budget, multi-window burn-rate alerts | From the Google SRE book and workbook; the example uses the workbook's thresholds to show the arithmetic |
| GenAI span names and content capture defaults | From the OpenTelemetry GenAI semantic conventions, which are in Development status |
| AI span kinds and masking variables | From the OpenInference specification |
| The split of diagnostic, audit and security logs by promise and reader | A presentation proposed on this site; the underlying practices are long established |
| One trace per run, with the human decision as a second, linked trace | A design choice of this site, explained on the Tracing page |
| Trust marks on retrieved passages, the online labels, the promotion checklist | Designs proposed on this site, shown in the example code |
The example, its cases UW-501 to UW-504, the simulated day, all numbers | Original and synthetic |
How the facts were checked#
Every statement about a named product, protocol or standard was read in that product's or standard's own documentation on 2026-11-17, and logged with the address and the date in the repository under docs/facts/, one file per page. Where a page did not say something, the text says "the documentation read here does not say". Code for products is labelled "documented shape; not run here"; the only code presented as run is the example's Python, and every output shown was produced by running it.
Some facts changed recently and were read twice before publishing:
- The OpenTelemetry GenAI semantic conventions have moved from the opentelemetry.io pages to their own repository,
open-telemetry/semantic-conventions-genai, and are in Development status. - Several GenAI attribute names are deprecated:
gen_ai.systemin favor ofgen_ai.provider.name, and the prompt and completion token names in favor ofgen_ai.usage.input_tokensandgen_ai.usage.output_tokens. - Datadog's product for LLM applications, formerly LLM Observability, is now called Agent Observability.
Vendor documentation moves fast. Before you rely on a sentence about a product, open the linked page.
Reading list#
The primary sources the pages rely on, grouped by kind. Each was read on 2026-11-17.
Standards and specifications
- OpenTelemetry documentation: What is OpenTelemetry, Signals, Traces, Sampling, Collector, OTLP specification, Metrics data model, Semantic conventions
- OpenTelemetry GenAI semantic conventions repository
- W3C Trace Context
- OpenInference specification
Practice
- Google SRE: Service Level Objectives and Alerting on SLOs
- Prometheus: Metric and label naming
- SLF4J: Manual (MDC and parameterized messages)
Products (described on Backends and Tracing tools)
- Splunk Enterprise, Elastic, Datadog, Grafana with Loki, Tempo and Mimir, Jaeger
- Arize Phoenix, LangSmith
- Claude Code monitoring (for the coding-agent example on Cost)
Sister sites#
This site is one of a series on engineering software that contains a model. Each site owns one subject and links to the others at the seams instead of repeating them.
| Site | Owns |
|---|---|
| prompt.sammuthu.com | Instructions: how they are written, composed, ranked, versioned, defended |
| context.sammuthu.com | The working set the model can see now |
| memory.sammuthu.com | Durable information and saved execution state |
| a2a.sammuthu.com | Messages between agents; the A2A protocol |
| observe.sammuthu.com (this site) | Evidence of what happened: logs, metrics, traces, audit events, replay |
| guardrails.sammuthu.com | Enforceable checks around a probabilistic agent |
| tools.sammuthu.com | Callable capabilities: schema, execution, MCP tools |
| schema.sammuthu.com | Typed output contracts |
| orchestrate.sammuthu.com | The control plane: what runs next |
| reliable.sammuthu.com | Acceptable outcomes despite failures, retries and load |
| harness.sammuthu.com | The runtime around a model call |
| eval.sammuthu.com | Measurement: graders, golden sets, judges, release gates |
| diff-to-deploy.sammuthu.com | Shipping a change with coding agents |
Diagram: The seams: what this site hands to its neighbours
Read left to right. This site produces evidence. Each box on the right is a sister site that reads part of that evidence for its own purpose. Where the subjects meet, a page here says what the concept is not and links the neighbour.
- This site
- Evidence of what happened: Logs, metrics, traces, audit events, quality signals, replay snapshots and promoted cases.
- Neighbours that read it
- Evaluation: Judges whether the behavior was acceptable, from traces and golden cases.
- Reliability: Decides timeouts, retries and recovery from the incident evidence.
- Memory: Owns saved execution state; here only the checkpoint span is recorded.
- Harness: Owns the runtime that emits the telemetry.
The code#
The example lives in the repository under examples/uw-observe/: standard-library Python 3, no packages, no network, no model, no key. python3 run_observe.py simulates the day and writes every record; python3 -m unittest runs its 68 tests. The interactive labs on the pages run in the browser on data generated from the same code, with a test that checks the browser's calculations against Python's. Nothing in the labs is stored: a reload starts fresh.
Author#
Written by Sam Muthu. Corrections and questions about a fact are welcome; each fact log in the repository names the page it came from, so a correction can be checked against the source.