All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Foundations · 4

Distributed tracing: one span tree across HTTP, queues, tools and models

A run crosses a gateway, a queue, a worker, a tool and a model. How the trace context travels with it, and how to read the waterfall that results.

One run, one tree
A trace is a tree of spans: each step records when it started, how long it took, and which step called it.
The id travels
A header called traceparent carries the trace id across HTTP and the queue, so four processes write one tree.
Links for later
The underwriter decides 38 minutes later, in a new trace. A span link joins it to the run.

Case UW-006 took 7,216 ms. The log explorer shows every log line of run-7f3a in time order: the gateway accepted it at 14:07:12, the worker logged two ERROR lines for get_roof_assessment, then a WARN for the fallback, then the recommendation. The model recommends; a person decides. From the log lines alone, a careful engineer can guess where the time went. Guess is the right word. Log lines are points in time. They do not say how long each step lasted, which step was waiting for which, or whether the 2 seconds between two lines were spent in the tool, in the queue, or in nothing at all.

Now add the services. The run did not happen in one process. The gateway received the HTTP request and put a message on a queue. A worker picked it up, called a model, called the roof-data provider twice, searched the guidance, called the model again, saved a checkpoint and stored the recommendation. Half an hour later the underwriter cockpit received a decision. Four processes, three of them on different machines, each writing its own records with its own clock.

The question this page answers: how do records from several processes become one picture of one run, with the time of every step and the order in which steps called each other? One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.

1. The span: one step, with a start and a length#

A span is the record of one step: its name, when it started, how long it took, whether it failed, and a few named facts about it. Here is one span from run-7f3a, the first call to the roof-data provider, as the example records it (attributes shortened):

FieldValueWhat it says
nameget_roof_assessmentThe step
span_ida557613ffd8a0c4fThis span's own id: 16 hex characters
parent_span_id0feeb0f297585ee0The span that called it (gather_evidence)
trace_id7540bd55d4e55ccb8f04f3d81880717aThe trace it belongs to: 32 hex characters
start_ms, duration_ms994, 2000Started 994 ms into the trace, lasted 2,000 ms
statuserrorIt failed
attributesattempt_number: 1, timeout_ms: 2000, error: timeoutNamed facts about this step
eventsretry_scheduledA fact with a time inside the span

Two fields carry the structure. The parent span is the span whose step called this one, named by parent_span_id; a span with no parent is the root span of its trace. And every span shares the run's trace_id. Put those together and the spans form a tree: the trace. Four signals defined a trace as the connected record of one run; this is how the connection is stored.

A span event is a fact with a timestamp inside one span. The first get_roof_assessment span carries the event retry_scheduled, with the backoff (500 ms) and the next attempt_number. OpenTelemetry's guidance: use an event when the timestamp matters, an attribute when it does not.

Plain English. A log line says "this happened at 14:07:14". A span says "this step started at 14:07:13, lasted two seconds, failed, and was called by that other step". The span adds length and family.

2. The waterfall: reading a trace#

A waterfall draws the spans of one trace as horizontal bars on a time axis, each indented under its parent. The example prints one with telemetry.waterfall:

cd "$TMPDIR/uw-observe-tracing" && /opt/homebrew/bin/python3 run_observe.py --case UW-006
# trace of the run
trace 7540bd55d4e55ccb8f04f3d81880717a  started 2026-09-14T14:07:12.000Z  7216 ms
     0 ms    17 ms  gateway      http         POST /submissions                            ok      |#                                        |
    12 ms     3 ms  gateway      queue          publish submissions                        ok      |#                                        |
   182 ms  7034 ms  agent-worker queue            process submissions                      ok      | ####################################### |
   182 ms  7034 ms  agent-worker agent              underwrite_submission                  ok      | ####################################### |
   182 ms   812 ms  agent-worker model                plan_steps                           ok      | #####                                   |
   994 ms  4890 ms  agent-worker agent                gather_evidence                      ok      |      ###########################        |
   994 ms  2000 ms  agent-worker tool                   get_roof_assessment (attempt 1)    error   |      ###########                        |
  3494 ms  2000 ms  agent-worker tool                   get_roof_assessment (attempt 2)    error   |                   ###########           |
  5494 ms   240 ms  agent-worker retrieval              search_guidance                    ok      |                              #          |
  5734 ms   150 ms  agent-worker retrieval              find_similar_decisions             ok      |                                #        |
  5884 ms     4 ms  agent-worker policy_check         check_eligibility                    ok      |                                 #       |
  5888 ms  1310 ms  agent-worker model                write_recommendation                 ok      |                                 ####### |
  7198 ms     6 ms  agent-worker checkpoint           save_checkpoint                      ok      |                                        #|
  7204 ms    12 ms  agent-worker tool                 record_recommendation                ok      |                                        #|

(This is the trace part of the case view; the same command also prints the logs, the decision trace and the audit events.) Read it in three passes.

  1. The longest bars. gather_evidence takes 4,890 of 7,216 ms, and inside it the two get_roof_assessment bars take 2,000 ms each. That is where the time went.
  2. The gaps. Between the end of attempt 1 (2,994 ms) and the start of attempt 2 (3,494 ms) there are 500 ms with no child span: the backoff, which the retry_scheduled event names. Between the gateway's publish (ending at 15 ms) and the worker's consume (182 ms) the message waited in the queue; the worker's log line says "waited 165 ms", measured from the moment the gateway finished its request.
  3. The shape. Children run one after another, never in parallel: this agent works step by step. A tree with parallel children would show overlapping bars.
SpanServiceKind (example / OpenTelemetry)Starts at (ms)Lasts (ms)Status
POST /submissionsgatewayhttp / SERVER017ok
publish submissionsgatewayqueue / PRODUCER123ok
process submissionsagent-workerqueue / CONSUMER1827034ok
underwrite_submissionagent-workeragent / INTERNAL1827034ok
plan_stepsagent-workermodel / CLIENT182812ok
get_roof_assessment attempt 1agent-workertool / INTERNAL9942000error
get_roof_assessment attempt 2agent-workertool / INTERNAL34942000error
write_recommendationagent-workermodel / CLIENT58881310ok
POST /decisions (second trace)cockpithttp / SERVER38 min later40ok

In the lab, switch the case to UW-002 (the clean run) and compare: the same tree without the two red bars. Then open UW-504, where the queue delivered the same message twice: two consumer spans under one producer, one run_id, and only one record_recommendation that stored anything.

In our insurance agent. Only the trace shows case UW-008. The agent reached the right recommendation without calling search_guidance. Its output is identical to a careful run; its log lines differ by one missing line that nobody would notice. In the waterfall, the missing bar is obvious.

3. Span kinds: who called whom, and how#

A span kind says what role a span plays in a call between processes. OpenTelemetry defines five, and the example uses all of them:

Span kindMeaning (OpenTelemetry)In run-7f3a
SERVERA synchronous incoming remote callThe gateway's POST /submissions; the cockpit's POST /decisions
CLIENTA synchronous outgoing remote callThe two model calls, plan_steps and write_recommendation
PRODUCERCreates a job processed later, such as a message on a queuepublish submissions at the gateway
CONSUMERProcesses a job created by a producer, possibly long afterprocess submissions in the worker
INTERNALDoes not cross a process boundary (the default)The agent's own steps, rule checks, the checkpoint

The example keeps a second field, kind, for its own step categories (model, tool, retrieval, policy_check, checkpoint, human). The two answer different questions: the OpenTelemetry kind says how the call crossed a boundary; the example's kind says what sort of agent step it was. GenAI telemetry is about the second.

Common confusion. The tool spans for get_roof_assessment are INTERNAL, following the OpenTelemetry GenAI convention for an executed tool (see OpenTelemetry). The HTTP request to the roof-data provider inside that tool would be a CLIENT child span. The example does not model it: we do not own the provider's telemetry, and the tool span already carries the timeout.

4. Context propagation: how four processes write one tree#

The gateway starts the trace: it creates the trace id and the root span. The worker is a different process, started by a message, possibly on another machine. How does the worker know which trace its spans belong to and which span is their parent?

The gateway tells it. Context propagation means passing the trace id and the current span id along with every call or message, so that the receiver can continue the same trace. On HTTP and on most queues the context travels in a header. The W3C Trace Context standard defines that header: traceparent, which in its own words "identifies the incoming request in a tracing system". Its format is four fields separated by dashes: a version, the 32-hex trace id, the 16-hex id of the caller's span, and flags, of which the lowest bit says whether the caller sampled the trace.

For run-7f3a the gateway's producer span writes this header on the queue message:

traceparent: 00-7540bd55d4e55ccb8f04f3d81880717a-28d1d73a6ccd1b1b-01

The worker reads it, and its consumer span takes 28d1d73a6ccd1b1b (the producer span) as its parent. The code is two small functions in telemetry.py:

def inject(span: dict[str, Any], sampled: bool = True) -> str:
    """The W3C `traceparent` header value for a span: version-traceid-parentid-flags."""
    return f"00-{span['trace_id']}-{span['span_id']}-{'01' if sampled else '00'}"


def extract(header: str | None) -> dict[str, Any] | None:
    """Parse a `traceparent` value. Returns None when it is missing or invalid (the receiver then starts a new trace)."""
    m = TRACEPARENT.match(header or "")
    if not m or m.group(1) == "ff" or set(m.group(2)) == {"0"} or set(m.group(3)) == {"0"}:
        return None
    return {"trace_id": m.group(2), "span_id": m.group(3), "sampled": bool(int(m.group(4), 16) & 1)}

Notice what extract does with a missing or broken header: it returns nothing, and the receiver starts a new trace. That is the most common tracing failure in practice. One service in the chain that does not forward the header splits the run into two unrelated traces, and nothing errors.

Diagram: How the trace context travels from the gateway to the worker

Read left to right. The gateway starts the trace and its producer span writes a traceparent header on the queue message. The worker reads the header and its consumer span takes the producer span as parent. Every later span in the worker inherits the same trace id. If any hop drops the header, the receiver starts a new trace and the run splits in two.

  1. 1 · Gateway
    1. POST /submissions: Receives the broker's submission, creates the trace id and the root span.
    2. publish submissions: Puts the message on the queue and writes the trace context into a header.
  2. 2 · Message header
    1. traceparent: The W3C header that carries the trace id and the caller's span id with the message.
  3. 3 · Worker
    1. process submissions: Reads the header and continues the same trace with the producer span as parent.
    2. underwrite_submission: All later spans of the run share the trace id and nest under the consumer.
Before AI. A Kafka producer in a Spring service with tracing instrumentation adds the same header to the record, and the consumer continues the trace. A thread pool or an @Async method that does not copy the context is the in-process version of the broken chain. The agent adds new places where context gets lost: a tool implemented as a separate process, a framework that runs tool calls on its own threads, an outside API that cannot carry your header.

The underwriter decides on run-7f3a at 14:45:19, 38 minutes after the run ended. That decision is a new HTTP request to the cockpit. Should its spans join the run's trace?

On this site, no: one trace per run for the automated part, and the decision is a second trace. A trace that stays open for 38 minutes, or for days when a decision waits over a weekend, breaks tail sampling, inflates durations and confuses every latency chart. Instead, the decision trace's root span carries a span link: a reference to a span in another trace, meaning "this was caused by that". OpenTelemetry describes links as existing "so that you can associate one span with one or more spans, implying a causal relationship".

decision trace 1fb8aad7c68d8c80d7244e4d8780b70e
  POST /decisions (cockpit, SERVER, 40 ms)
    link -> trace 7540bd55d4e55ccb8f04f3d81880717a, span d0b81cb6c43351ad (record_recommendation)
           link.reason: decision on this recommendation
    underwriter_decision (cockpit, INTERNAL, 25 ms): decision request_info, override false

The link points at the exact span that stored the recommendation. And both traces carry run_id run-7f3a, so a search by run finds both. In the lab above, follow the link from the decision trace back to the run.

The example also uses a link inside the run: the consumer span links to the producer span with the reason "message creation context". With one delivery that is redundant (the producer is also the parent). With a queue that batches messages, one consumer span processes many messages from many traces and can have only one parent; links are how it points to all of them.

Diagram: Two traces for one run, joined by a span link and by run_id

The automated part of run-7f3a is one trace that ends after 7216 ms. The underwriter's decision 38 minutes later is a new request and a new trace. Its root span carries a span link to the span that stored the recommendation. Both traces carry run_id, so a search by run finds both.

  1. Trace of the run
    1. Gateway to worker: One trace per run for the automated part, from the gateway to the stored recommendation.
    2. record_recommendation: The span that stored the recommendation for the underwriter.
  2. Trace of the decision
    1. POST /decisions: The cockpit's own request, its own trace, with a span link back to the recommendation.
    2. underwriter_decision: The person's decision, recorded as a span and as an audit event.
What this is not. Saving the state of a run so that it can continue later is a checkpoint, and resuming from it belongs to memory.sammuthu.com. The span named save_checkpoint records that a checkpoint was written and its state hash; the trace does not hold the state.
The same idea for a coding agent. Session CA-001 is one trace: read the repository instructions, plan, edit, run the tests (9.4 seconds, error), read the failure and fix, edit, run the tests again (pass), open a pull request. The CI pipeline that runs on the pull request is a separate trace in a separate system, linked to the session's open_pull_request span, exactly like the underwriter's decision. Switch the waterfall lab to CA-001 to see both. CI itself is covered on diff-to-deploy.sammuthu.com.
Clock skew: when a child starts before its parent

Each process stamps its spans with its own clock. If the worker's clock runs 300 ms behind the gateway's, the consumer span can appear to start before the producer span ended, or even before it started. Backends handle this in different ways: some shift a child so it starts after its parent, some show it as recorded and leave the reader to notice.

The example avoids the problem by design: one simulated clock for every process, and merge_spans shifts each process's spans by its origin. Real systems reduce skew with time synchronization and read cross-process gaps with care: the gap of about 165 ms between publish and consume in run-7f3a is queue wait plus any skew between two machines. Durations measured within one process are reliable; gaps measured across processes are estimates.

One trace per run, or one per attempt: the choice this site made

A run could be modelled as one trace per attempt (each retry of the whole workflow starts a fresh trace) or one trace per run (retries of a step are child spans with attempt_number). This site uses one trace per run for the automated part, with step retries as sibling spans, because the question during an incident is "what happened to this run", and the answer should be one waterfall. The cost: a run that is retried from scratch after a crash would need a new trace and a link, which the example does not model.

The decision trace is separate for the reasons in section 5: human latency is unbounded, and traces should end. The join keys are the span link (exact) and run_id (searchable). Neither alone is enough: links are not indexed by every backend, and run_id does not say which span caused the decision.

Checklist#

In one breath#

A span records one step with its start, length, status and parent; spans that share a trace id form a tree, and a waterfall draws it so that the two 2-second timeouts of run-7f3a are visible at a glance. The traceparent header carries the trace id and the caller's span id across HTTP and the queue, so four processes write one tree, and a forgotten header silently splits it. Work that arrives much later, like the underwriter's decision, becomes its own trace, joined by a span link and by run_id.

Where this leads#

You can now follow one run across processes and read where its time went. Most of the spans in this trace are not HTTP calls: they are model calls, retrievals and tool calls, and each needs its own facts recorded (tokens, documents, arguments, what the model was told). What an agent's spans must record, and what they must not, is the subject of GenAI telemetry.