The first version of the underwriting agent ran on the platform engineer's laptop. When something went wrong, the program printed a line, and the engineer read it in the terminal:
2026-09-14 14:07:14 ERROR [agent-worker] get_roof_assessment failed: timeout after 2000 ms
That line did its job. It said when, how bad, which program and what happened. The engineer saw it, scrolled up, saw which submission the agent was working on, and understood. The model recommends; a person decides, and on one laptop with one run at a time, a terminal is a perfectly good record of what happened.
Then the agent went to production. The gateway runs on several machines, the worker on a dozen. On the simulated day there are 48 runs, and they write 661 log lines between them. Now the on-call engineer is asked about one submission, SUB-1006. The line above exists, on one of the worker machines, in a file that rotated at noon. Next to it are lines from other runs that interleaved with it. Which "timeout" belongs to which submission? The line does not say.
This page climbs from that single line to records that forty machines and one engineer can search: fields instead of sentences, a severity and a category on every line, timestamps that agree, and one place where all of it is sent. Nothing here is new with AI, and all of it is needed before anything that is. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
1. A log line and stdout#
A log is a record of one event, with a time: something happened here, at this moment. Each record is a log line (or log record). The simplest place to write one is stdout, the standard output of a process: the stream that a terminal shows, and that a container platform collects. "Print to stdout" is how almost every program starts, and it stays useful in production: a container platform commonly collects each container's stdout, so the program does not need to know where its lines go.
The line above has four parts, separated by spaces and brackets: a timestamp (when it happened), a severity (ERROR), the service in brackets, and a message. A person reads it easily. A program has to guess where each part starts.
2. Why free text fails at forty machines#
Free text fails in three predictable ways once there is more than one machine and more than one run.
- The join is missing. The line does not say which run, which submission or which request it belongs to. On a laptop the context was the lines above it. With 48 runs interleaving, the lines above belong to someone else.
- The search is a guess. To find the timeouts of one run you search for words. "timeout" finds every timeout of the day, in every run. "SUB-1006" finds only the lines that happened to mention the submission in their text, which most do not.
- The format drifts. One developer writes "timed out", another "timeout", a third "deadline exceeded". A search for one word misses the others, and nobody notices.
The example measures the first two on the simulated day. A free-text search for "timeout" across all 661 records returns 10 lines from 5 different runs. Without a field to separate them, the engineer has to read every one to find the two that belong to SUB-1006. These numbers are from the synthetic day.
3. Structured logs: one JSON object per line#
A structured log is a log line whose facts are named fields: one JSON object per line. Here is the same event as the worker writes it:
{"ts": "2026-09-14T14:07:14.994Z", "severity": "ERROR", "category": "tool", "service.name": "agent-worker", "message": "get_roof_assessment failed: timeout after 2000 ms", "request_id": "req-86f59002", "run_id": "run-7f3a", "submission_id": "SUB-1006", "policy_version": "2026.2", "trace_id": "7540bd55d4e55ccb8f04f3d81880717a", "span_id": "a557613ffd8a0c4f", "attempt_number": 1}
Every fact a person had to infer from the free-text line is now a field: the run (run_id), the submission, the attempt, the version of the policy fixture, and the trace and span ids that Tracing uses. The message is still there, for people. The fields are for machines.
The example writes logs with telemetry.Logger. Two functions matter. bind adds identifiers that every following line of the run carries, so that no developer has to remember to add run_id to each line:
def bind(self, **ids: Any) -> None:
"""Ids that every following record carries, until `clear`."""
self.context.update(ids)
And log writes one record: time, severity, category, service, message, the bound ids, the current span's ids, and any extra fields:
def log(self, severity: str, category: str, service: str, message: str,
span: dict[str, Any] | None = None, **fields: Any) -> dict[str, Any]:
"""Append one structured log record and return it."""
assert severity in SEVERITIES and category in CATEGORIES, (severity, category)
record: dict[str, Any] = {"ts": self.clock.iso(), "severity": severity, "category": category,
"service.name": service, "message": message}
record.update(self.context)
if span is not None:
record.update(trace_id=span["trace_id"], span_id=span["span_id"])
record.update(fields)
self.records.append(record)
return record
Notice the assert: a severity or category outside the agreed lists is a bug, caught where the line is written, not discovered during an incident when a filter misses it.
Before AI. A Java service using SLF4J with Logback has the same two parts. The logging context is the MDC (mapped diagnostic context): putrun_idinto it at the start of a request, and every log line on that thread carries it. A JSON encoder turns each event into one JSON object per line. The SLF4J manual describes MDC support and parameterized messages such aslogger.debug("Temperature set to {}. Old value was {}.", t, oldT);. The agent adds no new mechanism here; it adds new fields (run_id,attempt_number,policy_version) and more steps per request.
Common confusion. A structured log is not a trace. Each line is still a point in time. It can carry the trace_id so that it can be joined to the trace, but it does not say how long a step lasted or which step called which. Four signals puts each kind of record in its place.
4. Severity and category#
Severity is the level of a log line: how much attention it deserves. The example uses four:
| Severity | Meaning | In run-7f3a |
|---|---|---|
| DEBUG | Detail for a developer, usually off in production | none on the simulated day |
| INFO | Normal operation worth recording | "run started for SUB-1006", "checkpoint saved" |
| WARN | Something unusual that the system handled | "roof assessment unavailable after 2 attempts; using the controlled fallback request_info" |
| ERROR | A step failed | the two get_roof_assessment timeouts |
On the simulated day the 661 records split into 633 INFO, 18 WARN and 10 ERROR. The useful property of severity is that it is ordered: "WARN and above" is a filter that removes 95 percent of the lines and keeps everything unusual.
A log category says what kind of step wrote the line. The example's categories are http, queue, agent, model, tool, retrieval, policy, checkpoint, audit and security. Category answers a different question from severity: not "how bad" but "what part of the system". The filter "category tool, severity ERROR" is the first move of the incident on Incidents: it returns exactly the 10 timeouts, from 5 runs.
Production pitfall. Severity inflation. When every handled condition is logged as ERROR, the ERROR filter returns everything and means nothing. The controlled fallback in run-7f3a is WARN, not ERROR, on purpose: the system did what it was designed to do. The timeouts that caused it are ERROR, because a step failed.
5. Timestamps that agree#
A timestamp is the time recorded on a log line. Three rules make timestamps from many machines usable together: write them in UTC, write them in one format with a time zone marker (the example uses ISO 8601 with milliseconds and a Z: 2026-09-14T14:07:14.994Z), and keep the machines' clocks synchronized. A line stamped in local time on one machine and UTC on another puts events in the wrong order, and during an incident the order is the story.
The example uses one simulated clock (telemetry.SimClock) for every process, so its timestamps always agree. Real machines drift; Tracing shows what that does to a trace.
6. Log aggregation: one place to search#
The last step makes the fields useful. Log aggregation means shipping the log lines of every process on every machine to one place, where they are parsed, indexed and searched together. It has four parts, each covered in depth on Backends:
- Ship. A small program on each machine (or the container platform) reads stdout or the log files and sends the lines away promptly, so that a machine that disappears takes little with it.
- Parse. Each line becomes a record with fields. With structured logs this is trivial: the line is already JSON. With free text it needs patterns, which break when the format drifts.
- Index. The store builds lookups over chosen fields so that a filter does not read every record.
- Search. A person or an alert asks a question by field and by text.
With the day's 661 records in one place, the opening question becomes one filter. The explorer below holds the simulated day. Compare a free-text search for "timeout" with the field filter run_id = run-7f3a.
| Query | Records | What it returns |
|---|---|---|
| free text "timeout" | 10 | Every timeout of the day, from 5 runs, mixed together |
run_id = run-7f3a | 15 | Every record of one run from every service (gateway, worker, cockpit), only 2 of which contain the word "timeout" |
run_id = run-7f3a and "timeout" | 2 | The two failed attempts of the anchor run |
category tool, severity ERROR | 10 | The dependency outage in one screen |
The second row is the point of this page. The field filter returns the gateway's access line, the queue line, the plan, both timeouts, the fallback, the recommendation, the checkpoint, the audit line and the underwriter's decision 38 minutes later: the whole story of one run, including the lines that never mention the word you would have searched for. Free text cannot do that, at any scale.
Diagram: From one printed line to a record forty machines can share
Read left to right. Each step adds one property to the same event: the timeout of attempt 1 in run-7f3a. Free text is readable by people. Fields make it filterable. Severity and category make it sortable by attention and by part of the system. Bound identifiers make it joinable to the rest of its run.
- 1 · Free text
- A printed line: Time, severity, service and message as one string on stdout.
- 2 · Fields
- One JSON object: Each fact is a named field a machine can filter on.
- 3 · Severity and category
- How bad, what part: An ordered severity and a category from an agreed list on every line.
- 4 · Identifiers
- Bound to the run: Ids added once per run and carried by every line, so lines join across services.
Diagram: Log aggregation: from many machines to one query
Read left to right. Each machine writes its lines to stdout. A shipper sends them away promptly. The store parses and indexes them. One query then returns every line of run-7f3a from the gateway, the worker and the cockpit. The detailed path and the products that implement it are on the Backends page.
- 1 · Machines
- Gateway: Access lines for every submission and decision request.
- Workers: The agent's steps, tool calls, fallbacks and audit appends.
- Cockpit: The underwriters' decisions, hours later.
- 2 · Ship and parse
- Shipper: Reads stdout and sends each line away before the machine can disappear.
- 3 · Index and search
- One query: Every line of one run from every service, in time order.
What this is not. This page does not decide which records the business must keep for years; a log line for engineers may be sampled and deleted. Records that must be proved later are audit events, on Audit logs.
A log schema, and why to enforce it at the source
Structured logging only pays when the field names are the same everywhere. run_id in the worker and runId in the cockpit are two fields to a search engine, and a filter on one silently misses the other. Three practices keep a schema:
- One shared logger setup per language, with the bound identifiers and the allowed severities and categories built in, as
telemetry.Loggerdoes with itsassert. - Reserved names for identifiers (
run_id,trace_id,submission_id,request_id,attempt_number,policy_version), documented in one place: on this site, Identifiers. - A test that parses every log line produced by the test suite and rejects unknown top-level fields or wrong types. A new field then becomes a reviewed change rather than a surprise in production.
Message text should stay stable too. "get_roof_assessment failed: timeout after 2000 ms" with the variable parts also in fields lets people search by text and machines by field. Parameterized messages (the {} placeholders of SLF4J) keep the template constant and the values separate.
How much to log: the cost of every line
Each line is shipped, parsed, indexed, stored and kept. At 661 lines a day this costs nothing; at production volume, logging is often the largest item of an observability budget. Three habits keep it in check: DEBUG off in production unless switched on for one service for a short time; no per-iteration lines inside tight loops (a summary line at the end instead, or a metric, see Four signals); and no large payloads in log lines. The last one matters more for agents than for any earlier system, because the tempting payload is a whole prompt. Prompts belong on spans, under the capture rules of GenAI telemetry, if anywhere.
The same idea for a coding agent. A coding-agent session such as CA-001 writes the same kind of lines: "run_tests failed: 3 tests failed", "open_pull_request ok". Bound with a session id and the repository commit, they can be filtered the same way across many sessions. How a coding agent's harness records its steps is covered on harness.sammuthu.com.
Checklist#
- Every service writes one JSON object per line to stdout, with the same field names.
- Identifiers (
run_id,submission_id,request_id,trace_id) are bound once per run and carried by every line automatically. - Severity uses a small fixed set, and handled conditions are WARN, not ERROR.
- Every line has a category from a fixed list.
- Timestamps are UTC, ISO 8601 with a zone marker, and clocks are synchronized.
- Lines are shipped off each machine promptly to one aggregated, indexed store.
- A test rejects log lines that do not follow the schema.
In one breath#
A log line records an event and its time, and on one machine a free-text line on stdout is enough. With many machines and many runs, the line must carry its facts as named fields (one JSON object per line), with a severity, a category, a UTC timestamp and the identifiers of its run, and every line must be shipped to one place where it can be filtered. Then the question "what happened to SUB-1006" is one filter, run_id = run-7f3a, which returns all 15 records of the run, not 10 timeouts from 5 runs.
Where this leads#
You can now write log lines that machines can join and filter. A log line answers "what happened here, at this moment". It does not answer "how often does this happen", "how long do runs take" or "where in this run did the time go". Those are questions for other kinds of record: metrics, traces and events, each described on Four signals.