All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
About

About this site and its sources

What this site is, what it is not, where its ideas come from, how the facts were checked, and the sister sites of the series.

This is a knowledge site about keeping evidence of what agentic software did. It starts from one line printed to a terminal and climbs to structured logs, metrics, traces, audit events, redaction, service level objectives, quality signals, replay and cost. At each step it asks one question: what does this record let me prove about what happened? It is written for engineers and architects who run agents in production, and for experienced backend engineers meeting agents for the first time.

One example runs through every page. An insurer receives a broker's request for a quote. An underwriting agent reads it, checks the roof, looks up the guidance and writes a recommendation. A human underwriter decides. The model recommends; a person decides. The anchor run is case UW-006, in which the roof-data provider times out twice.

What it is and is not#

It isIt is not
A ladder: one idea per step, each word in plain English with a tiny example before it is usedA standard, or the only way to divide the subject
One running example, a synthetic insurance agent and one simulated production day of 48 runsA record of any real insurer, broker, building or decision
Code that was run: standard-library Python in examples/uw-observe/, with its real output on the pagesA library to install, or a production implementation
A demonstration of mechanisms: how a span, a hash chain, a burn rate or a replay worksA measurement of any model or product. The agent is a scripted stand-in
Products and standards described from their own documentation, with the date it was readA ranking or a buying guide. Nothing is called best, and no price is quoted
Costs in invented units to show the arithmeticA statement about what any model or service costs

An honest boundary, said once on the Start page, once on the insurance lab and here: this is an illustrative reconstruction for teaching. The cases, rules, fields, provider, times, token counts and costs are invented. Nothing here reports the behavior of a real system or model. The rules of the policy fixture (for example, a roof older than 25 years in wind zone C is not eligible) are synthetic and are not insurance practice.

What is original here and what is not#

Most ideas on this site were published by others first. What this site adds is the arrangement, the running example and the code.

PartStatus
Logs, metrics and traces as the three main signalsCommon practice; the OpenTelemetry documentation describes the same signals
Spans, span kinds, span links, span events, context propagationFrom OpenTelemetry and W3C Trace Context, read in their documentation
traceparent formatW3C Trace Context (Recommendation, November 2021)
SLI, SLO, error budget, multi-window burn-rate alertsFrom the Google SRE book and workbook; the example uses the workbook's thresholds to show the arithmetic
GenAI span names and content capture defaultsFrom the OpenTelemetry GenAI semantic conventions, which are in Development status
AI span kinds and masking variablesFrom the OpenInference specification
The split of diagnostic, audit and security logs by promise and readerA presentation proposed on this site; the underlying practices are long established
One trace per run, with the human decision as a second, linked traceA design choice of this site, explained on the Tracing page
Trust marks on retrieved passages, the online labels, the promotion checklistDesigns proposed on this site, shown in the example code
The example, its cases UW-501 to UW-504, the simulated day, all numbersOriginal and synthetic

How the facts were checked#

Every statement about a named product, protocol or standard was read in that product's or standard's own documentation on 2026-11-17, and logged with the address and the date in the repository under docs/facts/, one file per page. Where a page did not say something, the text says "the documentation read here does not say". Code for products is labelled "documented shape; not run here"; the only code presented as run is the example's Python, and every output shown was produced by running it.

Some facts changed recently and were read twice before publishing:

Vendor documentation moves fast. Before you rely on a sentence about a product, open the linked page.

Reading list#

The primary sources the pages rely on, grouped by kind. Each was read on 2026-11-17.

Standards and specifications

Practice

Products (described on Backends and Tracing tools)

Sister sites#

This site is one of a series on engineering software that contains a model. Each site owns one subject and links to the others at the seams instead of repeating them.

SiteOwns
prompt.sammuthu.comInstructions: how they are written, composed, ranked, versioned, defended
context.sammuthu.comThe working set the model can see now
memory.sammuthu.comDurable information and saved execution state
a2a.sammuthu.comMessages between agents; the A2A protocol
observe.sammuthu.com (this site)Evidence of what happened: logs, metrics, traces, audit events, replay
guardrails.sammuthu.comEnforceable checks around a probabilistic agent
tools.sammuthu.comCallable capabilities: schema, execution, MCP tools
schema.sammuthu.comTyped output contracts
orchestrate.sammuthu.comThe control plane: what runs next
reliable.sammuthu.comAcceptable outcomes despite failures, retries and load
harness.sammuthu.comThe runtime around a model call
eval.sammuthu.comMeasurement: graders, golden sets, judges, release gates
diff-to-deploy.sammuthu.comShipping a change with coding agents

Diagram: The seams: what this site hands to its neighbours

Read left to right. This site produces evidence. Each box on the right is a sister site that reads part of that evidence for its own purpose. Where the subjects meet, a page here says what the concept is not and links the neighbour.

  1. This site
    1. Evidence of what happened: Logs, metrics, traces, audit events, quality signals, replay snapshots and promoted cases.
  2. Neighbours that read it
    1. Evaluation: Judges whether the behavior was acceptable, from traces and golden cases.
    2. Reliability: Decides timeouts, retries and recovery from the incident evidence.
    3. Memory: Owns saved execution state; here only the checkpoint span is recorded.
    4. Harness: Owns the runtime that emits the telemetry.

The code#

The example lives in the repository under examples/uw-observe/: standard-library Python 3, no packages, no network, no model, no key. python3 run_observe.py simulates the day and writes every record; python3 -m unittest runs its 68 tests. The interactive labs on the pages run in the browser on data generated from the same code, with a test that checks the browser's calculations against Python's. Nothing in the labs is stored: a reload starts fresh.

Author#

Written by Sam Muthu. Corrections and questions about a fact are welcome; each fact log in the repository names the page it came from, so a correction can be checked against the source.