All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Foundations · 3

Correlation identifiers: the ids that join the records

request_id, run_id, trace_id, span_id, submission_id, attempt_number and session id: what each one names, how long it lives, and what breaks when two are mixed up.

One id, one thing
A submission, a request, a run, a trace, a step, a try: each has its own id, and none stands in for another.
Lifetimes differ
A request_id lives 17 ms. A run_id lives until the underwriter decides. A submission_id lives for years.
Mix two, lose a run
Join by the wrong id and you get two runs that look like a duplicate, or one run with its decision missing.

The morning after the simulated day, the compliance team asks a plain question: show us everything the system did with the quote request SUB-1004. The model recommends; a person decides, and they want to see both halves.

The on-call engineer filters every record by submission_id = SUB-1004. The filter returns 26 log lines, two recommendation_recorded audit events (seq 12 and seq 37) and two human_decision_recorded events (seq 57 and seq 85). Two recommendations for one quote request. Both say escalate, both cite R-ROOF-03, both rest on the same evidence. It looks like the agent wrote the same recommendation twice: a duplicate write, perhaps a retry that should have been idempotent.

It is not a duplicate. SUB-1004 was processed twice: once at 09:58:40 (case UW-004) and again at 16:02:40 (case UW-010, the same input run again). Two complete runs, each with its own gateway request, its own trace, its own recommendation and its own decision by the underwriter. The 26 lines are 13 lines from each, interleaved by time. Nothing in the records was wrong. The engineer joined them with an identifier that names the quote request, while the question "did the agent write twice?" is about runs.

This page is about choosing the right join key. Every record in the example carries several identifiers. Each names a different thing and lives for a different length of time. Knowing which is which is the difference between "two runs" and "a bug", and between "the whole run" and "the run without its decision". One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.

1. What an identifier is for#

Records are written by different services at different moments. The gateway writes a line when the request arrives, the worker writes a dozen while the agent works, and the cockpit writes two when the underwriter decides 38 minutes later. To read them as one story, each record must carry a value that says what it belongs to.

A correlation id is such a value: an identifier copied onto every record that belongs together, so the records can be joined later by an exact match. A tiny example: all 15 log lines of the anchor run carry run_id: run-7f3a. The filter run_id = run-7f3a returns those 15 and nothing else, from three services.

The word "correlation id" names a role, not one field. The example has several such fields because there are several things to join by. The first thing to ask of any identifier is: what does one value of it name, and how long does that thing live?

2. The identifiers of one run#

Here is every identifier of the anchor run run-7f3a (case UW-006), with its value as the example writes it. Read the "Lives for" column with care: it is the reason the ids cannot replace each other.

IdNamesValue in run-7f3aCreated byLives for
submission_idThe quote request, the business objectSUB-1006Intake, before the agentYears
request_idOne HTTP request at the gateway or the cockpitgateway req-86f59002; cockpit req-9ad562e4The service that receives the requestOne request (17 ms at the gateway)
run_idOne execution of the workflow for a submissionrun-7f3aThe gateway, when it accepts the submissionFrom the gateway to the human decision (38 min here)
trace_idOne trace: 32 lowercase hex charactersrun 7540bd55d4e55ccb8f04f3d81880717a; decision 1fb8aad7c68d8c80d7244e4d8780b70eThe first service of each traceOne trace (7,216 ms; 40 ms)
span_idOne step: 16 lowercase hex charactersfirst roof lookup a557613ffd8a0c4fThe process that runs the stepOne step
attempt_numberWhich try of the same step, inside one run1 and 2 on the two get_roof_assessment spansThe worker, per stepOne step
session_id and roleThe underwriter's cockpit session; a role, never a namesess-e29f1f, role underwriterThe cockpit at sign-inOne sign-in
policy_versionThe version of the policy fixture the run used2026.2The run, from the policy fixtureUntil the next version
evidence_idsThe evidence a recommendation rests onEV-APPL-1006, EV-GUIDE-1006 (no roof evidence: both lookups failed)The agent, per piece of evidenceYears
idempotency_keyOne write of the recommendationrec-run-7f3aThe worker, derived from run_idOne write

Three things in this table are design decisions of this site, and the rest of the page depends on them. A submission can have several runs, so submission_id and run_id are different fields. A run has one trace for its automated part, and the underwriter's decision is a second trace, so run_id and trace_id are different fields. And attempt_number counts tries of one step inside a run, so a retry never creates a new run.

The example mints the ids of a run in pipeline.py. The gateway's request and the run's trace come from ids_for:

def ids_for(run_id: str, seed: int) -> dict[str, str]:
    """Deterministic ids for one run: one request at the gateway, one trace for the automated part."""
    return {"request_id": "req-" + hex_id(f"req:{run_id}:{seed}", 8),
            "trace_id": hex_id(f"trace:{run_id}:{seed}", 32)}

The underwriter's decision, in decide, gets a new trace and a new request of its own, and a session:

# three lines from pipeline.py:decide (indentation removed)
trace_id = hex_id(f"trace:decision:{run_id}:{ctx.seed}", 32)
request_id = "req-" + hex_id(f"req:decision:{run_id}:{ctx.seed}", 8)
session_id = "sess-" + hex_id(f"sess:{run_id}", 6)

The ids are derived from a hash so that the example is repeatable with seed 7; a production system would generate random ids. And the example gives every decision its own session id, a simplification: a real cockpit session covers many decisions by one person. What a reader should take from the code is the shape. The decision shares run_id with the run and nothing else: not the trace, not the request.

Diagram: The identifiers of run-7f3a, from the longest-lived to the shortest

Read left to right, from what lives longest to what lives shortest. The submission outlives every run of it. The run lives from the gateway to the underwriter's decision and has two traces: the automated part and the decision. Inside a trace each step is a span, and a retried step has an attempt number. The HTTP requests are the shortest of all. Each id names one box and nothing else.

  1. 1 · Years
    1. submission_id: The business object; it can have several runs and keeps its evidence.
  2. 2 · One run
    1. run_id: One execution of the workflow; the only id on every record of the run.
    2. idempotency_key: Makes a repeated write of the recommendation store nothing.
  3. 3 · Traces
    1. Run trace: The automated part: gateway, queue, worker, tools and model.
    2. Decision trace: The underwriter's request, joined by a span link and by run_id.
  4. 4 · Steps and tries
    1. span_id: One step of one trace, 16 hex characters.
    2. attempt_number: Which try of the same step; a retry never makes a new run.
  5. 5 · Milliseconds
    1. request_id: One request at the gateway or the cockpit; ends when it is answered.
Plain English. Think of a parcel. The order number (the submission) lasts as long as the customer's account. Each attempt to ship it (a run) gets its own tracking number. Each scan at a depot (a request) is a moment. If a parcel is lost and shipped again, the order has two tracking numbers, and nobody calls that a duplicate order.

3. What one id selects, and what it misses#

The surest way to understand an identifier is to filter by it and look at what comes back. The lab below holds the records of each named case. Pick an identifier, and see which of the run's log lines, traces, audit events and metrics that one value selects.

Identifier = valueLog linesTracesAudit eventsMisses
run_id = run-7f3a15 of 152 of 22 of 2nothing
trace_id = 7540bd55… (run)13 of 151 of 21 of 2the decision: 2 cockpit lines, its trace, seq 75
submission_id = SUB-100615 of 151 of 22 of 2the decision trace (it carries no submission id)
gateway request_id = req-86f5900213 of 150 of 2the decision lines and both audit events
cockpit request_id = req-9ad562e4, or session_id = sess-e29f1f2 of 150 of 2everything before the decision
any idmetrics: 0metrics carry no ids, on purpose

Two rows deserve a second look. submission_id looks perfect for SUB-1006, because that submission had one run. It is perfect only by luck, as SUB-1004 showed in the opening. And no identifier selects any metric. That is deliberate: a metric with a run_id label would grow one series per run, the subject of Advanced. Metrics lead to runs through their time window, not through an id.

In the lab, switch the case to UW-010 and pick submission_id: the value SUB-1004 selects the lines of two runs. Then switch to UW-504 and pick the gateway request_id.

4. One submission, two runs: UW-010#

Back to the opening. The submission SUB-1004 arrived twice on the simulated day. Case UW-004 ran it at 09:58:40; case UW-010 ran the same input again at 16:02:40. On eval.sammuthu.com, UW-010 is a case run twenty times to measure variation; here it is one more run of the same submission, which is all this page needs.

Run 1 (UW-004)Run 2 (UW-010)
submission_idSUB-1004SUB-1004
run_idrun-8bd7run-534e
gateway request_idreq-b6f479aereq-a7b7ab26
trace_iddead7486…64e2f675…
Duration3,305 ms3,055 ms
Recommendation (audit seq)escalate (seq 12)escalate (seq 37)
Decision (audit seq)escalate, no override (seq 57)escalate, no override (seq 85)
evidence_idsEV-APPL-1004, EV-ROOF-0410, EV-GUIDE-1004the same

Read the last row. The evidence ids are identical, because evidence belongs to the submission (the application, the roof record, the guidance), not to the run. That is correct: two runs looked at the same evidence. It is also why evidence_ids cannot tell runs apart.

Now follow what breaks when submission_id is used as if it named a run:

On the simulated day this happens twice: SUB-1001 also has two runs (cases UW-001 and UW-008). The day has 48 runs and 46 distinct submission ids.

Common confusion. A run is not a session and not a trial. A session belongs to a person signed in to the cockpit. A trial is an eval word: one of several runs of the same case, made on purpose to measure variation. A run is one execution of the workflow for a submission, whatever caused it.

5. One run, two deliveries: UW-504#

The opposite mistake happens when one run is split. In case UW-504 (this site's own case), the queue delivers the same message twice: the worker finished the run, but its acknowledgement to the queue was lost, so after 30 seconds the queue delivered the message again. The command below prints the run's log lines; this excerpt is the first part of its output, the logs.

cd "$TMPDIR/uw-observe-correlation" && /opt/homebrew/bin/python3 run_observe.py --case UW-504
UW-504 · SUB-1504 · run-f610 · status ok · recommendation approve

# logs (filter run_id = run-f610)
2026-09-14 15:20:00 INFO  [gateway] POST /submissions 202 accepted SUB-1504 as run-f610
2026-09-14 15:20:00 INFO  [agent-worker] message received from submissions (delivery 1, waited 235 ms)
2026-09-14 15:20:00 INFO  [agent-worker] run started for SUB-1504
2026-09-14 15:20:01 INFO  [agent-worker] plan_steps finished: 726 tokens in, 25 out (small-model)
2026-09-14 15:20:01 INFO  [agent-worker] get_roof_assessment ok  attempt_number=1
2026-09-14 15:20:01 INFO  [agent-worker] search_guidance returned 5 passages
2026-09-14 15:20:01 INFO  [agent-worker] check_eligibility: approve (rules: none)
2026-09-14 15:20:03 INFO  [agent-worker] write_recommendation finished: 714 tokens in, 93 out (large-model)
2026-09-14 15:20:03 INFO  [agent-worker] checkpoint saved: recommendation_written
2026-09-14 15:20:03 INFO  [agent-worker] audit event appended: recommendation_recorded (seq 35)
2026-09-14 15:20:03 INFO  [agent-worker] run finished: recommendation approve written for the underwriter; a person decides
2026-09-14 15:20:33 INFO  [agent-worker] message received from submissions (delivery 2, waited 33218 ms)
2026-09-14 15:20:33 WARN  [agent-worker] redelivery of a message already processed for run-f610
2026-09-14 15:20:33 INFO  [agent-worker] run started for SUB-1504
2026-09-14 15:20:34 INFO  [agent-worker] plan_steps finished: 726 tokens in, 25 out (small-model)
2026-09-14 15:20:34 INFO  [agent-worker] get_roof_assessment ok  attempt_number=1
2026-09-14 15:20:34 INFO  [agent-worker] search_guidance returned 5 passages
2026-09-14 15:20:34 INFO  [agent-worker] check_eligibility: approve (rules: none)
2026-09-14 15:20:36 INFO  [agent-worker] write_recommendation finished: 714 tokens in, 93 out (large-model)
2026-09-14 15:20:36 INFO  [agent-worker] checkpoint saved: recommendation_written
2026-09-14 15:20:36 INFO  [agent-worker] record_recommendation: duplicate call, nothing stored (same idempotency_key)
2026-09-14 15:20:36 INFO  [agent-worker] run finished: recommendation approve written for the underwriter; a person decides
2026-09-14 16:17:03 INFO  [cockpit] audit event appended: human_decision_recorded (seq 80); decision approve, override false
2026-09-14 16:17:03 INFO  [cockpit] POST /decisions 200

What a successful run proves: the second delivery reused the run's identifiers, did all the work again, and stored nothing new. What it does not prove: that a real queue redelivers in this way; the redelivery is scripted in the case file.

Each identifier tells a different part of this story:

The duplicate was not free. The second delivery made two more model calls (751 and 807 tokens again). The idempotency key protects the stored recommendation; it does not prevent the repeated work. Detecting the redelivery before the agent starts, from the message id or the run's checkpoint, is a reliability design, outside this page.

What this is not. Making retries and redeliveries safe (timeouts, backoff, idempotent writes, deduplication) is the subject of reliable.sammuthu.com. This page is about the identifiers that let you see that a redelivery happened, and count it once.

6. One run, two traces: the decision#

The last conflation is the subtlest, because the two ids look alike: both are long hex strings, and in most systems one request is one trace. On this site the automated part of a run is one trace, and the underwriter's decision, 38 minutes later, is a new request and a second trace. The reason is on Tracing: a trace that stays open while a person reads breaks every latency chart.

So trace_id names less than the run. The lab's second row shows the cost: filtering by the run's trace_id returns 13 of the 15 log lines, one of the two traces and one of the two audit events. It misses exactly the part the compliance team cares about most, the decision (seq 75). Two things join the traces: a span link from the decision trace's root span to the record_recommendation span of the run (d0b81cb6c43351ad), and run_id, which both carry. The link is exact; run_id is searchable. Use run_id to find; follow the link to see which step the decision answered.

The three cases of this page side by side: in each, one id is too wide or too narrow for the question, and run_id is the one that fits.

Diagram: Three ways to join by the wrong id

Each column is one case from the simulated day. In UW-010 the submission id is too wide: it joins two runs and makes them look like a duplicate. In UW-504 the request id is too narrow, and the attempt number repeats across two deliveries. In UW-006 the run's trace id misses the decision trace. In all three, run_id selects exactly the run.

  1. UW-010 · one submission, two runs
    1. submission_id SUB-1004: Selects 26 lines and two recommendations from two runs, which look like a duplicate write.
    2. run_id: Splits them into two complete runs of 13 lines each.
  2. UW-504 · one run, two deliveries
    1. request_id req-047ea6c8: The gateway request ended in 19 ms; it misses the decision an hour later.
    2. run_id and idempotency_key: Both deliveries share them, so the second write stores nothing.
  3. UW-006 · one run, two traces
    1. trace_id of the run: Selects the automated part and misses the underwriter's decision.
    2. run_id and a span link: Both traces carry run_id; the decision links to the span that stored the recommendation.

7. The ids that describe, not join#

Two fields in the table are carried on records for a different reason. They do not join records of one run; they say under what conditions the run happened.

policy_version names the version of the policy fixture the run used. On the simulated day every one of the 661 log lines carries 2026.2. The field matters at the edge: a run that started before a version change must keep the version it started with, and the past approval of case UW-007, decided under 2025.4, is evidence about the old version, not a rule of the new one. Without the field, the drift in override rate after 2026-07-01 on Quality signals could not be split by version.

evidence_ids name what a recommendation rests on, in the EV-ROOF-0412 style. They outlive the run, because the evidence belongs to the submission. In run-7f3a the list has no roof evidence at all: both lookups timed out, so there was nothing to rest on, and the recommendation names the gap in its missing list instead. An empty place in evidence_ids is itself a fact to observe.

Before AI. A Spring service has carried correlation ids for years, in two places. Inside one process, the logging context: SLF4J's MDC (mapped diagnostic context) holds key-value pairs for the current thread, and the log encoder adds them to every line. Across a Kafka topic, record headers: a ProducerRecord carries headers, and the consumer reads them from the ConsumerRecord. The shape below is assembled from the SLF4J and Kafka API documentation; documented shape, not run here.
// producer side: the gateway, after it accepts SUB-1006
ProducerRecord<String, String> record = new ProducerRecord<>("submissions", submissionId, payload);
record.headers().add("run_id", runId.getBytes(StandardCharsets.UTF_8));
producer.send(record);

// consumer side: the worker
Header h = consumed.headers().lastHeader("run_id");
String runId = new String(h.value(), StandardCharsets.UTF_8);
MDC.put("run_id", runId);
try {
    log.info("run started for {}", consumed.key());
} finally {
    MDC.remove("run_id");
}

The example's Logger.bind is the MDC, and the message header that carries traceparent is the Kafka header. What the agent adds is new ids with new lifetimes (run_id outliving its request, attempt_number per step, policy_version, evidence_ids) and new places where they get lost: a thread pool that does not copy the MDC, a tool that runs in another process, a model provider that cannot carry your header at all.

Designing identifiers: who mints them, how they travel, and how they leak

Mint each id once, where its thing begins. The gateway mints run_id when it accepts a submission, not the worker, so that the gateway's own access line already carries it and a redelivery cannot mint a second one. The cockpit mints its own request_id and session. An id minted in two places will eventually disagree.

Pick the format by its job. Trace and span ids follow W3C Trace Context: the trace id is a 16-byte array written as 32 hex characters, the parent id an 8-byte array written as 16, and all zeros is invalid for both. Business ids (SUB-1006, run-7f3a) are for people as well as machines: short, prefixed, typed. Production ids need enough randomness that two services never mint the same value; the example's short hashed ids are repeatable for teaching and would collide at scale.

Carry ids explicitly across every hop. Over HTTP and queues, traceparent carries the trace context. Other ids need their own carrier. OpenTelemetry's baggage is a key-value store propagated alongside the context; its documentation, read today, says baggage values are not added to spans, metrics or logs unless you add them, and warns that "Sensitive Baggage items can be shared with unintended resources, like third-party APIs." A run_id in baggage would reach the roof-data provider. Decide per id whether it may leave your boundary; a submission id that encodes a customer may not.

Ids are data too. A session_id tied to one underwriter is personal data in some jurisdictions; this site records the role on audit events and the session only on diagnostic records. Retention follows the record, not the id.

Testing that every record of a run can be joined

An id that is missing from one record breaks the join silently: the filter returns fewer lines and nothing errors. The cure is a test, run on the records a test suite produces. In the example, every one of the 661 log lines of the day carries request_id, run_id, trace_id, span_id, submission_id and policy_version; attempt_number appears on the 54 lines of roof lookups, delivery_count on the 50 consumer lines, idempotency_key on 48, session_id on the 92 cockpit lines. Those counts are the assertions:

  1. Every log line and every span of a run carries run_id, or carries a trace_id whose root span carries it.
  2. Every audit event carries run_id, submission_id, trace_id, span_id and policy_version.
  3. Every trace of a run that has a decision is joined to the decision trace by a span link to the record_recommendation span.
  4. Every retried step carries attempt_number, and the values within one execution are 1, 2, ... without gaps.
  5. No metric carries a label from the list of run-scoped ids.
  6. A run with two deliveries has one run_id, one idempotency_key and one stored recommendation (case UW-504).

The example's unit tests check several of these on its own records (test_observe.py: distinct ids, attempt_number inside one run, the linked decision trace, two deliveries with one recommendation); the list above is the general design.

The same idea for a coding agent. Session CA-001 has the same layering: the repository and branch (lasting like a submission), the session id (one execution, like a run), one trace per session, an attempt count on the repeated run_tests step, and the pull request number that joins the session to the CI trace, which is a separate trace linked from open_pull_request. CI itself is on diff-to-deploy.sammuthu.com.

Checklist#

In one breath#

Every record carries several identifiers, and each names one thing with its own lifetime: the quote request (submission_id, years), the HTTP request (request_id, milliseconds), the run (run_id, until the human decision), the trace and the span, the try of a step (attempt_number), the underwriter's session. Join by the one that names what the question is about. Joining by submission_id made the two runs of SUB-1004 look like a duplicate write, and joining by the run's trace_id lost the decision of run-7f3a; run_id finds the whole run, and the idempotency key derived from it kept the double delivery of UW-504 to one stored recommendation.

Where this leads#

You can now pick the identifier that answers a question and say what each one misses. One of them, trace_id, carries more than a name: it holds a tree of steps with their timing and parents, and it has to travel from the gateway through the queue to the worker without being dropped. How spans are built, how traceparent carries the trace across processes, and how the span link reaches the decision trace is the subject of Tracing.