Case UW-006 took 7,216 ms. The log explorer shows every log line of run-7f3a in time order: the gateway accepted it at 14:07:12, the worker logged two ERROR lines for get_roof_assessment, then a WARN for the fallback, then the recommendation. The model recommends; a person decides. From the log lines alone, a careful engineer can guess where the time went. Guess is the right word. Log lines are points in time. They do not say how long each step lasted, which step was waiting for which, or whether the 2 seconds between two lines were spent in the tool, in the queue, or in nothing at all.
Now add the services. The run did not happen in one process. The gateway received the HTTP request and put a message on a queue. A worker picked it up, called a model, called the roof-data provider twice, searched the guidance, called the model again, saved a checkpoint and stored the recommendation. Half an hour later the underwriter cockpit received a decision. Four processes, three of them on different machines, each writing its own records with its own clock.
The question this page answers: how do records from several processes become one picture of one run, with the time of every step and the order in which steps called each other? One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
1. The span: one step, with a start and a length#
A span is the record of one step: its name, when it started, how long it took, whether it failed, and a few named facts about it. Here is one span from run-7f3a, the first call to the roof-data provider, as the example records it (attributes shortened):
| Field | Value | What it says |
|---|---|---|
name | get_roof_assessment | The step |
span_id | a557613ffd8a0c4f | This span's own id: 16 hex characters |
parent_span_id | 0feeb0f297585ee0 | The span that called it (gather_evidence) |
trace_id | 7540bd55d4e55ccb8f04f3d81880717a | The trace it belongs to: 32 hex characters |
start_ms, duration_ms | 994, 2000 | Started 994 ms into the trace, lasted 2,000 ms |
status | error | It failed |
attributes | attempt_number: 1, timeout_ms: 2000, error: timeout | Named facts about this step |
events | retry_scheduled | A fact with a time inside the span |
Two fields carry the structure. The parent span is the span whose step called this one, named by parent_span_id; a span with no parent is the root span of its trace. And every span shares the run's trace_id. Put those together and the spans form a tree: the trace. Four signals defined a trace as the connected record of one run; this is how the connection is stored.
A span event is a fact with a timestamp inside one span. The first get_roof_assessment span carries the event retry_scheduled, with the backoff (500 ms) and the next attempt_number. OpenTelemetry's guidance: use an event when the timestamp matters, an attribute when it does not.
Plain English. A log line says "this happened at 14:07:14". A span says "this step started at 14:07:13, lasted two seconds, failed, and was called by that other step". The span adds length and family.
2. The waterfall: reading a trace#
A waterfall draws the spans of one trace as horizontal bars on a time axis, each indented under its parent. The example prints one with telemetry.waterfall:
cd "$TMPDIR/uw-observe-tracing" && /opt/homebrew/bin/python3 run_observe.py --case UW-006
# trace of the run
trace 7540bd55d4e55ccb8f04f3d81880717a started 2026-09-14T14:07:12.000Z 7216 ms
0 ms 17 ms gateway http POST /submissions ok |# |
12 ms 3 ms gateway queue publish submissions ok |# |
182 ms 7034 ms agent-worker queue process submissions ok | ####################################### |
182 ms 7034 ms agent-worker agent underwrite_submission ok | ####################################### |
182 ms 812 ms agent-worker model plan_steps ok | ##### |
994 ms 4890 ms agent-worker agent gather_evidence ok | ########################### |
994 ms 2000 ms agent-worker tool get_roof_assessment (attempt 1) error | ########### |
3494 ms 2000 ms agent-worker tool get_roof_assessment (attempt 2) error | ########### |
5494 ms 240 ms agent-worker retrieval search_guidance ok | # |
5734 ms 150 ms agent-worker retrieval find_similar_decisions ok | # |
5884 ms 4 ms agent-worker policy_check check_eligibility ok | # |
5888 ms 1310 ms agent-worker model write_recommendation ok | ####### |
7198 ms 6 ms agent-worker checkpoint save_checkpoint ok | #|
7204 ms 12 ms agent-worker tool record_recommendation ok | #|
(This is the trace part of the case view; the same command also prints the logs, the decision trace and the audit events.) Read it in three passes.
- The longest bars.
gather_evidencetakes 4,890 of 7,216 ms, and inside it the twoget_roof_assessmentbars take 2,000 ms each. That is where the time went. - The gaps. Between the end of attempt 1 (2,994 ms) and the start of attempt 2 (3,494 ms) there are 500 ms with no child span: the backoff, which the
retry_scheduledevent names. Between the gateway's publish (ending at 15 ms) and the worker's consume (182 ms) the message waited in the queue; the worker's log line says "waited 165 ms", measured from the moment the gateway finished its request. - The shape. Children run one after another, never in parallel: this agent works step by step. A tree with parallel children would show overlapping bars.
| Span | Service | Kind (example / OpenTelemetry) | Starts at (ms) | Lasts (ms) | Status |
|---|---|---|---|---|---|
POST /submissions | gateway | http / SERVER | 0 | 17 | ok |
publish submissions | gateway | queue / PRODUCER | 12 | 3 | ok |
process submissions | agent-worker | queue / CONSUMER | 182 | 7034 | ok |
underwrite_submission | agent-worker | agent / INTERNAL | 182 | 7034 | ok |
plan_steps | agent-worker | model / CLIENT | 182 | 812 | ok |
get_roof_assessment attempt 1 | agent-worker | tool / INTERNAL | 994 | 2000 | error |
get_roof_assessment attempt 2 | agent-worker | tool / INTERNAL | 3494 | 2000 | error |
write_recommendation | agent-worker | model / CLIENT | 5888 | 1310 | ok |
POST /decisions (second trace) | cockpit | http / SERVER | 38 min later | 40 | ok |
In the lab, switch the case to UW-002 (the clean run) and compare: the same tree without the two red bars. Then open UW-504, where the queue delivered the same message twice: two consumer spans under one producer, one run_id, and only one record_recommendation that stored anything.
In our insurance agent. Only the trace shows caseUW-008. The agent reached the right recommendation without callingsearch_guidance. Its output is identical to a careful run; its log lines differ by one missing line that nobody would notice. In the waterfall, the missing bar is obvious.
3. Span kinds: who called whom, and how#
A span kind says what role a span plays in a call between processes. OpenTelemetry defines five, and the example uses all of them:
| Span kind | Meaning (OpenTelemetry) | In run-7f3a |
|---|---|---|
| SERVER | A synchronous incoming remote call | The gateway's POST /submissions; the cockpit's POST /decisions |
| CLIENT | A synchronous outgoing remote call | The two model calls, plan_steps and write_recommendation |
| PRODUCER | Creates a job processed later, such as a message on a queue | publish submissions at the gateway |
| CONSUMER | Processes a job created by a producer, possibly long after | process submissions in the worker |
| INTERNAL | Does not cross a process boundary (the default) | The agent's own steps, rule checks, the checkpoint |
The example keeps a second field, kind, for its own step categories (model, tool, retrieval, policy_check, checkpoint, human). The two answer different questions: the OpenTelemetry kind says how the call crossed a boundary; the example's kind says what sort of agent step it was. GenAI telemetry is about the second.
Common confusion. The tool spans for get_roof_assessment are INTERNAL, following the OpenTelemetry GenAI convention for an executed tool (see OpenTelemetry). The HTTP request to the roof-data provider inside that tool would be a CLIENT child span. The example does not model it: we do not own the provider's telemetry, and the tool span already carries the timeout.
4. Context propagation: how four processes write one tree#
The gateway starts the trace: it creates the trace id and the root span. The worker is a different process, started by a message, possibly on another machine. How does the worker know which trace its spans belong to and which span is their parent?
The gateway tells it. Context propagation means passing the trace id and the current span id along with every call or message, so that the receiver can continue the same trace. On HTTP and on most queues the context travels in a header. The W3C Trace Context standard defines that header: traceparent, which in its own words "identifies the incoming request in a tracing system". Its format is four fields separated by dashes: a version, the 32-hex trace id, the 16-hex id of the caller's span, and flags, of which the lowest bit says whether the caller sampled the trace.
For run-7f3a the gateway's producer span writes this header on the queue message:
traceparent: 00-7540bd55d4e55ccb8f04f3d81880717a-28d1d73a6ccd1b1b-01
The worker reads it, and its consumer span takes 28d1d73a6ccd1b1b (the producer span) as its parent. The code is two small functions in telemetry.py:
def inject(span: dict[str, Any], sampled: bool = True) -> str:
"""The W3C `traceparent` header value for a span: version-traceid-parentid-flags."""
return f"00-{span['trace_id']}-{span['span_id']}-{'01' if sampled else '00'}"
def extract(header: str | None) -> dict[str, Any] | None:
"""Parse a `traceparent` value. Returns None when it is missing or invalid (the receiver then starts a new trace)."""
m = TRACEPARENT.match(header or "")
if not m or m.group(1) == "ff" or set(m.group(2)) == {"0"} or set(m.group(3)) == {"0"}:
return None
return {"trace_id": m.group(2), "span_id": m.group(3), "sampled": bool(int(m.group(4), 16) & 1)}
Notice what extract does with a missing or broken header: it returns nothing, and the receiver starts a new trace. That is the most common tracing failure in practice. One service in the chain that does not forward the header splits the run into two unrelated traces, and nothing errors.
Diagram: How the trace context travels from the gateway to the worker
Read left to right. The gateway starts the trace and its producer span writes a traceparent header on the queue message. The worker reads the header and its consumer span takes the producer span as parent. Every later span in the worker inherits the same trace id. If any hop drops the header, the receiver starts a new trace and the run splits in two.
- 1 · Gateway
- POST /submissions: Receives the broker's submission, creates the trace id and the root span.
- publish submissions: Puts the message on the queue and writes the trace context into a header.
- 2 · Message header
- traceparent: The W3C header that carries the trace id and the caller's span id with the message.
- 3 · Worker
- process submissions: Reads the header and continues the same trace with the producer span as parent.
- underwrite_submission: All later spans of the run share the trace id and nest under the consumer.
Before AI. A Kafka producer in a Spring service with tracing instrumentation adds the same header to the record, and the consumer continues the trace. A thread pool or an @Async method that does not copy the context is the in-process version of the broken chain. The agent adds new places where context gets lost: a tool implemented as a separate process, a framework that runs tool calls on its own threads, an outside API that cannot carry your header.
5. Span links: the decision that arrives later#
The underwriter decides on run-7f3a at 14:45:19, 38 minutes after the run ended. That decision is a new HTTP request to the cockpit. Should its spans join the run's trace?
On this site, no: one trace per run for the automated part, and the decision is a second trace. A trace that stays open for 38 minutes, or for days when a decision waits over a weekend, breaks tail sampling, inflates durations and confuses every latency chart. Instead, the decision trace's root span carries a span link: a reference to a span in another trace, meaning "this was caused by that". OpenTelemetry describes links as existing "so that you can associate one span with one or more spans, implying a causal relationship".
decision trace 1fb8aad7c68d8c80d7244e4d8780b70e
POST /decisions (cockpit, SERVER, 40 ms)
link -> trace 7540bd55d4e55ccb8f04f3d81880717a, span d0b81cb6c43351ad (record_recommendation)
link.reason: decision on this recommendation
underwriter_decision (cockpit, INTERNAL, 25 ms): decision request_info, override false
The link points at the exact span that stored the recommendation. And both traces carry run_id run-7f3a, so a search by run finds both. In the lab above, follow the link from the decision trace back to the run.
The example also uses a link inside the run: the consumer span links to the producer span with the reason "message creation context". With one delivery that is redundant (the producer is also the parent). With a queue that batches messages, one consumer span processes many messages from many traces and can have only one parent; links are how it points to all of them.
Diagram: Two traces for one run, joined by a span link and by run_id
The automated part of run-7f3a is one trace that ends after 7216 ms. The underwriter's decision 38 minutes later is a new request and a new trace. Its root span carries a span link to the span that stored the recommendation. Both traces carry run_id, so a search by run finds both.
- Trace of the run
- Gateway to worker: One trace per run for the automated part, from the gateway to the stored recommendation.
- record_recommendation: The span that stored the recommendation for the underwriter.
- Trace of the decision
- POST /decisions: The cockpit's own request, its own trace, with a span link back to the recommendation.
- underwriter_decision: The person's decision, recorded as a span and as an audit event.
What this is not. Saving the state of a run so that it can continue later is a checkpoint, and resuming from it belongs to memory.sammuthu.com. The span named save_checkpoint records that a checkpoint was written and its state hash; the trace does not hold the state.
The same idea for a coding agent. SessionCA-001is one trace: read the repository instructions, plan, edit, run the tests (9.4 seconds, error), read the failure and fix, edit, run the tests again (pass), open a pull request. The CI pipeline that runs on the pull request is a separate trace in a separate system, linked to the session'sopen_pull_requestspan, exactly like the underwriter's decision. Switch the waterfall lab toCA-001to see both. CI itself is covered on diff-to-deploy.sammuthu.com.
Clock skew: when a child starts before its parent
Each process stamps its spans with its own clock. If the worker's clock runs 300 ms behind the gateway's, the consumer span can appear to start before the producer span ended, or even before it started. Backends handle this in different ways: some shift a child so it starts after its parent, some show it as recorded and leave the reader to notice.
The example avoids the problem by design: one simulated clock for every process, and merge_spans shifts each process's spans by its origin. Real systems reduce skew with time synchronization and read cross-process gaps with care: the gap of about 165 ms between publish and consume in run-7f3a is queue wait plus any skew between two machines. Durations measured within one process are reliable; gaps measured across processes are estimates.
One trace per run, or one per attempt: the choice this site made
A run could be modelled as one trace per attempt (each retry of the whole workflow starts a fresh trace) or one trace per run (retries of a step are child spans with attempt_number). This site uses one trace per run for the automated part, with step retries as sibling spans, because the question during an incident is "what happened to this run", and the answer should be one waterfall. The cost: a run that is retried from scratch after a crash would need a new trace and a link, which the example does not model.
The decision trace is separate for the reasons in section 5: human latency is unbounded, and traces should end. The join keys are the span link (exact) and run_id (searchable). Neither alone is enough: links are not indexed by every backend, and run_id does not say which span caused the decision.
Checklist#
- Every process continues the incoming trace context and forwards it on every outgoing call and message.
- A test sends a request through every hop and asserts one trace id from the gateway to the last span.
- Spans carry a span kind that matches how they cross process boundaries.
- Step retries are sibling spans with
attempt_number, and the wait between them is a span event or visible as a gap. - Work that arrives much later (the underwriter's decision, CI) is a new trace with a span link and a shared
run_id. - Long cross-process gaps are read as estimates because of clock skew.
- The waterfall of a healthy run is known, so that a missing bar (as in
UW-008) is noticed.
In one breath#
A span records one step with its start, length, status and parent; spans that share a trace id form a tree, and a waterfall draws it so that the two 2-second timeouts of run-7f3a are visible at a glance. The traceparent header carries the trace id and the caller's span id across HTTP and the queue, so four processes write one tree, and a forgotten header silently splits it. Work that arrives much later, like the underwriter's decision, becomes its own trace, joined by a span link and by run_id.
Where this leads#
You can now follow one run across processes and read where its time went. Most of the spans in this trace are not HTTP calls: they are model calls, retrievals and tool calls, and each needs its own facts recorded (tokens, documents, arguments, what the model was told). What an agent's spans must record, and what they must not, is the subject of GenAI telemetry.