All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Foundations · 1

Before AI: stdout, structured logs, severity and aggregation

A program prints a line. What turns that line into something forty machines and one engineer can search: fields, severity, categories and one place to send it all.

A line is a start
A program prints what happened, and when. On one machine that is enough.
Fields, not sentences
A structured log is one JSON object per line, so a machine can filter by run_id instead of guessing from words.
One place to search
Aggregation ships every line from every machine to one index, where 661 records become one query.

The first version of the underwriting agent ran on the platform engineer's laptop. When something went wrong, the program printed a line, and the engineer read it in the terminal:

2026-09-14 14:07:14 ERROR [agent-worker] get_roof_assessment failed: timeout after 2000 ms

That line did its job. It said when, how bad, which program and what happened. The engineer saw it, scrolled up, saw which submission the agent was working on, and understood. The model recommends; a person decides, and on one laptop with one run at a time, a terminal is a perfectly good record of what happened.

Then the agent went to production. The gateway runs on several machines, the worker on a dozen. On the simulated day there are 48 runs, and they write 661 log lines between them. Now the on-call engineer is asked about one submission, SUB-1006. The line above exists, on one of the worker machines, in a file that rotated at noon. Next to it are lines from other runs that interleaved with it. Which "timeout" belongs to which submission? The line does not say.

This page climbs from that single line to records that forty machines and one engineer can search: fields instead of sentences, a severity and a category on every line, timestamps that agree, and one place where all of it is sent. Nothing here is new with AI, and all of it is needed before anything that is. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.

1. A log line and stdout#

A log is a record of one event, with a time: something happened here, at this moment. Each record is a log line (or log record). The simplest place to write one is stdout, the standard output of a process: the stream that a terminal shows, and that a container platform collects. "Print to stdout" is how almost every program starts, and it stays useful in production: a container platform commonly collects each container's stdout, so the program does not need to know where its lines go.

The line above has four parts, separated by spaces and brackets: a timestamp (when it happened), a severity (ERROR), the service in brackets, and a message. A person reads it easily. A program has to guess where each part starts.

2. Why free text fails at forty machines#

Free text fails in three predictable ways once there is more than one machine and more than one run.

  1. The join is missing. The line does not say which run, which submission or which request it belongs to. On a laptop the context was the lines above it. With 48 runs interleaving, the lines above belong to someone else.
  2. The search is a guess. To find the timeouts of one run you search for words. "timeout" finds every timeout of the day, in every run. "SUB-1006" finds only the lines that happened to mention the submission in their text, which most do not.
  3. The format drifts. One developer writes "timed out", another "timeout", a third "deadline exceeded". A search for one word misses the others, and nobody notices.

The example measures the first two on the simulated day. A free-text search for "timeout" across all 661 records returns 10 lines from 5 different runs. Without a field to separate them, the engineer has to read every one to find the two that belong to SUB-1006. These numbers are from the synthetic day.

3. Structured logs: one JSON object per line#

A structured log is a log line whose facts are named fields: one JSON object per line. Here is the same event as the worker writes it:

{"ts": "2026-09-14T14:07:14.994Z", "severity": "ERROR", "category": "tool", "service.name": "agent-worker", "message": "get_roof_assessment failed: timeout after 2000 ms", "request_id": "req-86f59002", "run_id": "run-7f3a", "submission_id": "SUB-1006", "policy_version": "2026.2", "trace_id": "7540bd55d4e55ccb8f04f3d81880717a", "span_id": "a557613ffd8a0c4f", "attempt_number": 1}

Every fact a person had to infer from the free-text line is now a field: the run (run_id), the submission, the attempt, the version of the policy fixture, and the trace and span ids that Tracing uses. The message is still there, for people. The fields are for machines.

The example writes logs with telemetry.Logger. Two functions matter. bind adds identifiers that every following line of the run carries, so that no developer has to remember to add run_id to each line:

def bind(self, **ids: Any) -> None:
    """Ids that every following record carries, until `clear`."""
    self.context.update(ids)

And log writes one record: time, severity, category, service, message, the bound ids, the current span's ids, and any extra fields:

def log(self, severity: str, category: str, service: str, message: str,
        span: dict[str, Any] | None = None, **fields: Any) -> dict[str, Any]:
    """Append one structured log record and return it."""
    assert severity in SEVERITIES and category in CATEGORIES, (severity, category)
    record: dict[str, Any] = {"ts": self.clock.iso(), "severity": severity, "category": category,
                              "service.name": service, "message": message}
    record.update(self.context)
    if span is not None:
        record.update(trace_id=span["trace_id"], span_id=span["span_id"])
    record.update(fields)
    self.records.append(record)
    return record

Notice the assert: a severity or category outside the agreed lists is a bug, caught where the line is written, not discovered during an incident when a filter misses it.

Before AI. A Java service using SLF4J with Logback has the same two parts. The logging context is the MDC (mapped diagnostic context): put run_id into it at the start of a request, and every log line on that thread carries it. A JSON encoder turns each event into one JSON object per line. The SLF4J manual describes MDC support and parameterized messages such as logger.debug("Temperature set to {}. Old value was {}.", t, oldT);. The agent adds no new mechanism here; it adds new fields (run_id, attempt_number, policy_version) and more steps per request.
Common confusion. A structured log is not a trace. Each line is still a point in time. It can carry the trace_id so that it can be joined to the trace, but it does not say how long a step lasted or which step called which. Four signals puts each kind of record in its place.

4. Severity and category#

Severity is the level of a log line: how much attention it deserves. The example uses four:

SeverityMeaningIn run-7f3a
DEBUGDetail for a developer, usually off in productionnone on the simulated day
INFONormal operation worth recording"run started for SUB-1006", "checkpoint saved"
WARNSomething unusual that the system handled"roof assessment unavailable after 2 attempts; using the controlled fallback request_info"
ERRORA step failedthe two get_roof_assessment timeouts

On the simulated day the 661 records split into 633 INFO, 18 WARN and 10 ERROR. The useful property of severity is that it is ordered: "WARN and above" is a filter that removes 95 percent of the lines and keeps everything unusual.

A log category says what kind of step wrote the line. The example's categories are http, queue, agent, model, tool, retrieval, policy, checkpoint, audit and security. Category answers a different question from severity: not "how bad" but "what part of the system". The filter "category tool, severity ERROR" is the first move of the incident on Incidents: it returns exactly the 10 timeouts, from 5 runs.

Production pitfall. Severity inflation. When every handled condition is logged as ERROR, the ERROR filter returns everything and means nothing. The controlled fallback in run-7f3a is WARN, not ERROR, on purpose: the system did what it was designed to do. The timeouts that caused it are ERROR, because a step failed.

5. Timestamps that agree#

A timestamp is the time recorded on a log line. Three rules make timestamps from many machines usable together: write them in UTC, write them in one format with a time zone marker (the example uses ISO 8601 with milliseconds and a Z: 2026-09-14T14:07:14.994Z), and keep the machines' clocks synchronized. A line stamped in local time on one machine and UTC on another puts events in the wrong order, and during an incident the order is the story.

The example uses one simulated clock (telemetry.SimClock) for every process, so its timestamps always agree. Real machines drift; Tracing shows what that does to a trace.

The last step makes the fields useful. Log aggregation means shipping the log lines of every process on every machine to one place, where they are parsed, indexed and searched together. It has four parts, each covered in depth on Backends:

  1. Ship. A small program on each machine (or the container platform) reads stdout or the log files and sends the lines away promptly, so that a machine that disappears takes little with it.
  2. Parse. Each line becomes a record with fields. With structured logs this is trivial: the line is already JSON. With free text it needs patterns, which break when the format drifts.
  3. Index. The store builds lookups over chosen fields so that a filter does not read every record.
  4. Search. A person or an alert asks a question by field and by text.

With the day's 661 records in one place, the opening question becomes one filter. The explorer below holds the simulated day. Compare a free-text search for "timeout" with the field filter run_id = run-7f3a.

QueryRecordsWhat it returns
free text "timeout"10Every timeout of the day, from 5 runs, mixed together
run_id = run-7f3a15Every record of one run from every service (gateway, worker, cockpit), only 2 of which contain the word "timeout"
run_id = run-7f3a and "timeout"2The two failed attempts of the anchor run
category tool, severity ERROR10The dependency outage in one screen

The second row is the point of this page. The field filter returns the gateway's access line, the queue line, the plan, both timeouts, the fallback, the recommendation, the checkpoint, the audit line and the underwriter's decision 38 minutes later: the whole story of one run, including the lines that never mention the word you would have searched for. Free text cannot do that, at any scale.

Diagram: From one printed line to a record forty machines can share

Read left to right. Each step adds one property to the same event: the timeout of attempt 1 in run-7f3a. Free text is readable by people. Fields make it filterable. Severity and category make it sortable by attention and by part of the system. Bound identifiers make it joinable to the rest of its run.

  1. 1 · Free text
    1. A printed line: Time, severity, service and message as one string on stdout.
  2. 2 · Fields
    1. One JSON object: Each fact is a named field a machine can filter on.
  3. 3 · Severity and category
    1. How bad, what part: An ordered severity and a category from an agreed list on every line.
  4. 4 · Identifiers
    1. Bound to the run: Ids added once per run and carried by every line, so lines join across services.

Diagram: Log aggregation: from many machines to one query

Read left to right. Each machine writes its lines to stdout. A shipper sends them away promptly. The store parses and indexes them. One query then returns every line of run-7f3a from the gateway, the worker and the cockpit. The detailed path and the products that implement it are on the Backends page.

  1. 1 · Machines
    1. Gateway: Access lines for every submission and decision request.
    2. Workers: The agent's steps, tool calls, fallbacks and audit appends.
    3. Cockpit: The underwriters' decisions, hours later.
  2. 2 · Ship and parse
    1. Shipper: Reads stdout and sends each line away before the machine can disappear.
  3. 3 · Index and search
    1. One query: Every line of one run from every service, in time order.
What this is not. This page does not decide which records the business must keep for years; a log line for engineers may be sampled and deleted. Records that must be proved later are audit events, on Audit logs.
A log schema, and why to enforce it at the source

Structured logging only pays when the field names are the same everywhere. run_id in the worker and runId in the cockpit are two fields to a search engine, and a filter on one silently misses the other. Three practices keep a schema:

  • One shared logger setup per language, with the bound identifiers and the allowed severities and categories built in, as telemetry.Logger does with its assert.
  • Reserved names for identifiers (run_id, trace_id, submission_id, request_id, attempt_number, policy_version), documented in one place: on this site, Identifiers.
  • A test that parses every log line produced by the test suite and rejects unknown top-level fields or wrong types. A new field then becomes a reviewed change rather than a surprise in production.

Message text should stay stable too. "get_roof_assessment failed: timeout after 2000 ms" with the variable parts also in fields lets people search by text and machines by field. Parameterized messages (the {} placeholders of SLF4J) keep the template constant and the values separate.

How much to log: the cost of every line

Each line is shipped, parsed, indexed, stored and kept. At 661 lines a day this costs nothing; at production volume, logging is often the largest item of an observability budget. Three habits keep it in check: DEBUG off in production unless switched on for one service for a short time; no per-iteration lines inside tight loops (a summary line at the end instead, or a metric, see Four signals); and no large payloads in log lines. The last one matters more for agents than for any earlier system, because the tempting payload is a whole prompt. Prompts belong on spans, under the capture rules of GenAI telemetry, if anywhere.

The same idea for a coding agent. A coding-agent session such as CA-001 writes the same kind of lines: "run_tests failed: 3 tests failed", "open_pull_request ok". Bound with a session id and the repository commit, they can be filtered the same way across many sessions. How a coding agent's harness records its steps is covered on harness.sammuthu.com.

Checklist#

In one breath#

A log line records an event and its time, and on one machine a free-text line on stdout is enough. With many machines and many runs, the line must carry its facts as named fields (one JSON object per line), with a severity, a category, a UTC timestamp and the identifiers of its run, and every line must be shipped to one place where it can be filtered. Then the question "what happened to SUB-1006" is one filter, run_id = run-7f3a, which returns all 15 records of the run, not 10 timeouts from 5 runs.

Where this leads#

You can now write log lines that machines can join and filter. A log line answers "what happened here, at this moment". It does not answer "how often does this happen", "how long do runs take" or "where in this run did the time go". Those are questions for other kinds of record: metrics, traces and events, each described on Four signals.