The morning after the simulated day, the compliance team asks a plain question: show us everything the system did with the quote request SUB-1004. The model recommends; a person decides, and they want to see both halves.
The on-call engineer filters every record by submission_id = SUB-1004. The filter returns 26 log lines, two recommendation_recorded audit events (seq 12 and seq 37) and two human_decision_recorded events (seq 57 and seq 85). Two recommendations for one quote request. Both say escalate, both cite R-ROOF-03, both rest on the same evidence. It looks like the agent wrote the same recommendation twice: a duplicate write, perhaps a retry that should have been idempotent.
It is not a duplicate. SUB-1004 was processed twice: once at 09:58:40 (case UW-004) and again at 16:02:40 (case UW-010, the same input run again). Two complete runs, each with its own gateway request, its own trace, its own recommendation and its own decision by the underwriter. The 26 lines are 13 lines from each, interleaved by time. Nothing in the records was wrong. The engineer joined them with an identifier that names the quote request, while the question "did the agent write twice?" is about runs.
This page is about choosing the right join key. Every record in the example carries several identifiers. Each names a different thing and lives for a different length of time. Knowing which is which is the difference between "two runs" and "a bug", and between "the whole run" and "the run without its decision". One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
1. What an identifier is for#
Records are written by different services at different moments. The gateway writes a line when the request arrives, the worker writes a dozen while the agent works, and the cockpit writes two when the underwriter decides 38 minutes later. To read them as one story, each record must carry a value that says what it belongs to.
A correlation id is such a value: an identifier copied onto every record that belongs together, so the records can be joined later by an exact match. A tiny example: all 15 log lines of the anchor run carry run_id: run-7f3a. The filter run_id = run-7f3a returns those 15 and nothing else, from three services.
The word "correlation id" names a role, not one field. The example has several such fields because there are several things to join by. The first thing to ask of any identifier is: what does one value of it name, and how long does that thing live?
2. The identifiers of one run#
Here is every identifier of the anchor run run-7f3a (case UW-006), with its value as the example writes it. Read the "Lives for" column with care: it is the reason the ids cannot replace each other.
| Id | Names | Value in run-7f3a | Created by | Lives for |
|---|---|---|---|---|
submission_id | The quote request, the business object | SUB-1006 | Intake, before the agent | Years |
request_id | One HTTP request at the gateway or the cockpit | gateway req-86f59002; cockpit req-9ad562e4 | The service that receives the request | One request (17 ms at the gateway) |
run_id | One execution of the workflow for a submission | run-7f3a | The gateway, when it accepts the submission | From the gateway to the human decision (38 min here) |
trace_id | One trace: 32 lowercase hex characters | run 7540bd55d4e55ccb8f04f3d81880717a; decision 1fb8aad7c68d8c80d7244e4d8780b70e | The first service of each trace | One trace (7,216 ms; 40 ms) |
span_id | One step: 16 lowercase hex characters | first roof lookup a557613ffd8a0c4f | The process that runs the step | One step |
attempt_number | Which try of the same step, inside one run | 1 and 2 on the two get_roof_assessment spans | The worker, per step | One step |
session_id and role | The underwriter's cockpit session; a role, never a name | sess-e29f1f, role underwriter | The cockpit at sign-in | One sign-in |
policy_version | The version of the policy fixture the run used | 2026.2 | The run, from the policy fixture | Until the next version |
evidence_ids | The evidence a recommendation rests on | EV-APPL-1006, EV-GUIDE-1006 (no roof evidence: both lookups failed) | The agent, per piece of evidence | Years |
idempotency_key | One write of the recommendation | rec-run-7f3a | The worker, derived from run_id | One write |
Three things in this table are design decisions of this site, and the rest of the page depends on them. A submission can have several runs, so submission_id and run_id are different fields. A run has one trace for its automated part, and the underwriter's decision is a second trace, so run_id and trace_id are different fields. And attempt_number counts tries of one step inside a run, so a retry never creates a new run.
The example mints the ids of a run in pipeline.py. The gateway's request and the run's trace come from ids_for:
def ids_for(run_id: str, seed: int) -> dict[str, str]:
"""Deterministic ids for one run: one request at the gateway, one trace for the automated part."""
return {"request_id": "req-" + hex_id(f"req:{run_id}:{seed}", 8),
"trace_id": hex_id(f"trace:{run_id}:{seed}", 32)}
The underwriter's decision, in decide, gets a new trace and a new request of its own, and a session:
# three lines from pipeline.py:decide (indentation removed)
trace_id = hex_id(f"trace:decision:{run_id}:{ctx.seed}", 32)
request_id = "req-" + hex_id(f"req:decision:{run_id}:{ctx.seed}", 8)
session_id = "sess-" + hex_id(f"sess:{run_id}", 6)
The ids are derived from a hash so that the example is repeatable with seed 7; a production system would generate random ids. And the example gives every decision its own session id, a simplification: a real cockpit session covers many decisions by one person. What a reader should take from the code is the shape. The decision shares run_id with the run and nothing else: not the trace, not the request.
Diagram: The identifiers of run-7f3a, from the longest-lived to the shortest
Read left to right, from what lives longest to what lives shortest. The submission outlives every run of it. The run lives from the gateway to the underwriter's decision and has two traces: the automated part and the decision. Inside a trace each step is a span, and a retried step has an attempt number. The HTTP requests are the shortest of all. Each id names one box and nothing else.
- 1 · Years
- submission_id: The business object; it can have several runs and keeps its evidence.
- 2 · One run
- run_id: One execution of the workflow; the only id on every record of the run.
- idempotency_key: Makes a repeated write of the recommendation store nothing.
- 3 · Traces
- Run trace: The automated part: gateway, queue, worker, tools and model.
- Decision trace: The underwriter's request, joined by a span link and by run_id.
- 4 · Steps and tries
- span_id: One step of one trace, 16 hex characters.
- attempt_number: Which try of the same step; a retry never makes a new run.
- 5 · Milliseconds
- request_id: One request at the gateway or the cockpit; ends when it is answered.
Plain English. Think of a parcel. The order number (the submission) lasts as long as the customer's account. Each attempt to ship it (a run) gets its own tracking number. Each scan at a depot (a request) is a moment. If a parcel is lost and shipped again, the order has two tracking numbers, and nobody calls that a duplicate order.
3. What one id selects, and what it misses#
The surest way to understand an identifier is to filter by it and look at what comes back. The lab below holds the records of each named case. Pick an identifier, and see which of the run's log lines, traces, audit events and metrics that one value selects.
| Identifier = value | Log lines | Traces | Audit events | Misses |
|---|---|---|---|---|
run_id = run-7f3a | 15 of 15 | 2 of 2 | 2 of 2 | nothing |
trace_id = 7540bd55… (run) | 13 of 15 | 1 of 2 | 1 of 2 | the decision: 2 cockpit lines, its trace, seq 75 |
submission_id = SUB-1006 | 15 of 15 | 1 of 2 | 2 of 2 | the decision trace (it carries no submission id) |
gateway request_id = req-86f59002 | 13 of 15 | 0 of 2 | the decision lines and both audit events | |
cockpit request_id = req-9ad562e4, or session_id = sess-e29f1f | 2 of 15 | 0 of 2 | everything before the decision | |
| any id | metrics: 0 | metrics carry no ids, on purpose |
Two rows deserve a second look. submission_id looks perfect for SUB-1006, because that submission had one run. It is perfect only by luck, as SUB-1004 showed in the opening. And no identifier selects any metric. That is deliberate: a metric with a run_id label would grow one series per run, the subject of Advanced. Metrics lead to runs through their time window, not through an id.
In the lab, switch the case to UW-010 and pick submission_id: the value SUB-1004 selects the lines of two runs. Then switch to UW-504 and pick the gateway request_id.
4. One submission, two runs: UW-010#
Back to the opening. The submission SUB-1004 arrived twice on the simulated day. Case UW-004 ran it at 09:58:40; case UW-010 ran the same input again at 16:02:40. On eval.sammuthu.com, UW-010 is a case run twenty times to measure variation; here it is one more run of the same submission, which is all this page needs.
Run 1 (UW-004) | Run 2 (UW-010) | |
|---|---|---|
submission_id | SUB-1004 | SUB-1004 |
run_id | run-8bd7 | run-534e |
gateway request_id | req-b6f479ae | req-a7b7ab26 |
trace_id | dead7486… | 64e2f675… |
| Duration | 3,305 ms | 3,055 ms |
| Recommendation (audit seq) | escalate (seq 12) | escalate (seq 37) |
| Decision (audit seq) | escalate, no override (seq 57) | escalate, no override (seq 85) |
evidence_ids | EV-APPL-1004, EV-ROOF-0410, EV-GUIDE-1004 | the same |
Read the last row. The evidence ids are identical, because evidence belongs to the submission (the application, the roof record, the guidance), not to the run. That is correct: two runs looked at the same evidence. It is also why evidence_ids cannot tell runs apart.
Now follow what breaks when submission_id is used as if it named a run:
- The count is wrong. "Recommendations per submission" says 2 where the business expected 1, and an alert for duplicate writes fires on a legitimate second run.
- The duration is wrong. The first line is at 09:58:40 and the last at 18:08:43. A "time from first to last record" computed per submission reports eight hours for two runs of about three seconds each.
- The idempotency is wrong, in the other direction. If the key that makes
record_recommendationsafe to repeat were derived from the submission (rec-SUB-1004), the second run's recommendation would be silently dropped as a duplicate, and the underwriter would decide on the stale one. The example derives it from the run:key = f"rec-{run_id}"inagent_sim.py:_record.
On the simulated day this happens twice: SUB-1001 also has two runs (cases UW-001 and UW-008). The day has 48 runs and 46 distinct submission ids.
Common confusion. A run is not a session and not a trial. A session belongs to a person signed in to the cockpit. A trial is an eval word: one of several runs of the same case, made on purpose to measure variation. A run is one execution of the workflow for a submission, whatever caused it.
5. One run, two deliveries: UW-504#
The opposite mistake happens when one run is split. In case UW-504 (this site's own case), the queue delivers the same message twice: the worker finished the run, but its acknowledgement to the queue was lost, so after 30 seconds the queue delivered the message again. The command below prints the run's log lines; this excerpt is the first part of its output, the logs.
cd "$TMPDIR/uw-observe-correlation" && /opt/homebrew/bin/python3 run_observe.py --case UW-504
UW-504 · SUB-1504 · run-f610 · status ok · recommendation approve
# logs (filter run_id = run-f610)
2026-09-14 15:20:00 INFO [gateway] POST /submissions 202 accepted SUB-1504 as run-f610
2026-09-14 15:20:00 INFO [agent-worker] message received from submissions (delivery 1, waited 235 ms)
2026-09-14 15:20:00 INFO [agent-worker] run started for SUB-1504
2026-09-14 15:20:01 INFO [agent-worker] plan_steps finished: 726 tokens in, 25 out (small-model)
2026-09-14 15:20:01 INFO [agent-worker] get_roof_assessment ok attempt_number=1
2026-09-14 15:20:01 INFO [agent-worker] search_guidance returned 5 passages
2026-09-14 15:20:01 INFO [agent-worker] check_eligibility: approve (rules: none)
2026-09-14 15:20:03 INFO [agent-worker] write_recommendation finished: 714 tokens in, 93 out (large-model)
2026-09-14 15:20:03 INFO [agent-worker] checkpoint saved: recommendation_written
2026-09-14 15:20:03 INFO [agent-worker] audit event appended: recommendation_recorded (seq 35)
2026-09-14 15:20:03 INFO [agent-worker] run finished: recommendation approve written for the underwriter; a person decides
2026-09-14 15:20:33 INFO [agent-worker] message received from submissions (delivery 2, waited 33218 ms)
2026-09-14 15:20:33 WARN [agent-worker] redelivery of a message already processed for run-f610
2026-09-14 15:20:33 INFO [agent-worker] run started for SUB-1504
2026-09-14 15:20:34 INFO [agent-worker] plan_steps finished: 726 tokens in, 25 out (small-model)
2026-09-14 15:20:34 INFO [agent-worker] get_roof_assessment ok attempt_number=1
2026-09-14 15:20:34 INFO [agent-worker] search_guidance returned 5 passages
2026-09-14 15:20:34 INFO [agent-worker] check_eligibility: approve (rules: none)
2026-09-14 15:20:36 INFO [agent-worker] write_recommendation finished: 714 tokens in, 93 out (large-model)
2026-09-14 15:20:36 INFO [agent-worker] checkpoint saved: recommendation_written
2026-09-14 15:20:36 INFO [agent-worker] record_recommendation: duplicate call, nothing stored (same idempotency_key)
2026-09-14 15:20:36 INFO [agent-worker] run finished: recommendation approve written for the underwriter; a person decides
2026-09-14 16:17:03 INFO [cockpit] audit event appended: human_decision_recorded (seq 80); decision approve, override false
2026-09-14 16:17:03 INFO [cockpit] POST /decisions 200
What a successful run proves: the second delivery reused the run's identifiers, did all the work again, and stored nothing new. What it does not prove: that a real queue redelivers in this way; the redelivery is scripted in the case file.
Each identifier tells a different part of this story:
run_idis the same on both deliveries (run-f610), because the message carries the run, and a redelivery is not a new run. One trace holds both: 24 spans over 36,144 ms, with two consumer spans (delivery_count1 and 2) under the one producer span.idempotency_keyis the same too (rec-run-f610), so the secondrecord_recommendationstored nothing. The ledger has onerecommendation_recordedevent (seq 35), not two.attempt_numberis 1 on both roof lookups. It counts tries of one step within one execution, and each delivery executed the step once. The fielddelivery_counton the consumer span and its log lines tells the executions apart. Reading twoattempt_number=1lines as "the step was tried once" would be wrong.request_idis notrun_id. The gateway's requestreq-047ea6c8was answered with 202 after 19 ms. The run went on for 36 seconds and its decision came an hour later in another request. Filtering by the gateway'srequest_idreturns 22 of the run's 24 log lines and misses the decision. Ifrequest_idhad been used as the run's key, a second delivery would carry the run correctly (the message carries it), but any second HTTP request about the same work, such as the decision, would look like a different run.
The duplicate was not free. The second delivery made two more model calls (751 and 807 tokens again). The idempotency key protects the stored recommendation; it does not prevent the repeated work. Detecting the redelivery before the agent starts, from the message id or the run's checkpoint, is a reliability design, outside this page.
What this is not. Making retries and redeliveries safe (timeouts, backoff, idempotent writes, deduplication) is the subject of reliable.sammuthu.com. This page is about the identifiers that let you see that a redelivery happened, and count it once.
6. One run, two traces: the decision#
The last conflation is the subtlest, because the two ids look alike: both are long hex strings, and in most systems one request is one trace. On this site the automated part of a run is one trace, and the underwriter's decision, 38 minutes later, is a new request and a second trace. The reason is on Tracing: a trace that stays open while a person reads breaks every latency chart.
So trace_id names less than the run. The lab's second row shows the cost: filtering by the run's trace_id returns 13 of the 15 log lines, one of the two traces and one of the two audit events. It misses exactly the part the compliance team cares about most, the decision (seq 75). Two things join the traces: a span link from the decision trace's root span to the record_recommendation span of the run (d0b81cb6c43351ad), and run_id, which both carry. The link is exact; run_id is searchable. Use run_id to find; follow the link to see which step the decision answered.
The three cases of this page side by side: in each, one id is too wide or too narrow for the question, and run_id is the one that fits.
Diagram: Three ways to join by the wrong id
Each column is one case from the simulated day. In UW-010 the submission id is too wide: it joins two runs and makes them look like a duplicate. In UW-504 the request id is too narrow, and the attempt number repeats across two deliveries. In UW-006 the run's trace id misses the decision trace. In all three, run_id selects exactly the run.
- UW-010 · one submission, two runs
- submission_id SUB-1004: Selects 26 lines and two recommendations from two runs, which look like a duplicate write.
- run_id: Splits them into two complete runs of 13 lines each.
- UW-504 · one run, two deliveries
- request_id req-047ea6c8: The gateway request ended in 19 ms; it misses the decision an hour later.
- run_id and idempotency_key: Both deliveries share them, so the second write stores nothing.
- UW-006 · one run, two traces
- trace_id of the run: Selects the automated part and misses the underwriter's decision.
- run_id and a span link: Both traces carry run_id; the decision links to the span that stored the recommendation.
7. The ids that describe, not join#
Two fields in the table are carried on records for a different reason. They do not join records of one run; they say under what conditions the run happened.
policy_version names the version of the policy fixture the run used. On the simulated day every one of the 661 log lines carries 2026.2. The field matters at the edge: a run that started before a version change must keep the version it started with, and the past approval of case UW-007, decided under 2025.4, is evidence about the old version, not a rule of the new one. Without the field, the drift in override rate after 2026-07-01 on Quality signals could not be split by version.
evidence_ids name what a recommendation rests on, in the EV-ROOF-0412 style. They outlive the run, because the evidence belongs to the submission. In run-7f3a the list has no roof evidence at all: both lookups timed out, so there was nothing to rest on, and the recommendation names the gap in its missing list instead. An empty place in evidence_ids is itself a fact to observe.
Before AI. A Spring service has carried correlation ids for years, in two places. Inside one process, the logging context: SLF4J's MDC (mapped diagnostic context) holds key-value pairs for the current thread, and the log encoder adds them to every line. Across a Kafka topic, record headers: aProducerRecordcarries headers, and the consumer reads them from theConsumerRecord. The shape below is assembled from the SLF4J and Kafka API documentation; documented shape, not run here.
// producer side: the gateway, after it accepts SUB-1006
ProducerRecord<String, String> record = new ProducerRecord<>("submissions", submissionId, payload);
record.headers().add("run_id", runId.getBytes(StandardCharsets.UTF_8));
producer.send(record);
// consumer side: the worker
Header h = consumed.headers().lastHeader("run_id");
String runId = new String(h.value(), StandardCharsets.UTF_8);
MDC.put("run_id", runId);
try {
log.info("run started for {}", consumed.key());
} finally {
MDC.remove("run_id");
}
The example's Logger.bind is the MDC, and the message header that carries traceparent is the Kafka header. What the agent adds is new ids with new lifetimes (run_id outliving its request, attempt_number per step, policy_version, evidence_ids) and new places where they get lost: a thread pool that does not copy the MDC, a tool that runs in another process, a model provider that cannot carry your header at all.
Designing identifiers: who mints them, how they travel, and how they leak
Mint each id once, where its thing begins. The gateway mints run_id when it accepts a submission, not the worker, so that the gateway's own access line already carries it and a redelivery cannot mint a second one. The cockpit mints its own request_id and session. An id minted in two places will eventually disagree.
Pick the format by its job. Trace and span ids follow W3C Trace Context: the trace id is a 16-byte array written as 32 hex characters, the parent id an 8-byte array written as 16, and all zeros is invalid for both. Business ids (SUB-1006, run-7f3a) are for people as well as machines: short, prefixed, typed. Production ids need enough randomness that two services never mint the same value; the example's short hashed ids are repeatable for teaching and would collide at scale.
Carry ids explicitly across every hop. Over HTTP and queues, traceparent carries the trace context. Other ids need their own carrier. OpenTelemetry's baggage is a key-value store propagated alongside the context; its documentation, read today, says baggage values are not added to spans, metrics or logs unless you add them, and warns that "Sensitive Baggage items can be shared with unintended resources, like third-party APIs." A run_id in baggage would reach the roof-data provider. Decide per id whether it may leave your boundary; a submission id that encodes a customer may not.
Ids are data too. A session_id tied to one underwriter is personal data in some jurisdictions; this site records the role on audit events and the session only on diagnostic records. Retention follows the record, not the id.
Testing that every record of a run can be joined
An id that is missing from one record breaks the join silently: the filter returns fewer lines and nothing errors. The cure is a test, run on the records a test suite produces. In the example, every one of the 661 log lines of the day carries request_id, run_id, trace_id, span_id, submission_id and policy_version; attempt_number appears on the 54 lines of roof lookups, delivery_count on the 50 consumer lines, idempotency_key on 48, session_id on the 92 cockpit lines. Those counts are the assertions:
- Every log line and every span of a run carries
run_id, or carries atrace_idwhose root span carries it. - Every audit event carries
run_id,submission_id,trace_id,span_idandpolicy_version. - Every trace of a run that has a decision is joined to the decision trace by a span link to the
record_recommendationspan. - Every retried step carries
attempt_number, and the values within one execution are 1, 2, ... without gaps. - No metric carries a label from the list of run-scoped ids.
- A run with two deliveries has one
run_id, oneidempotency_keyand one stored recommendation (caseUW-504).
The example's unit tests check several of these on its own records (test_observe.py: distinct ids, attempt_number inside one run, the linked decision trace, two deliveries with one recommendation); the list above is the general design.
The same idea for a coding agent. SessionCA-001has the same layering: the repository and branch (lasting like a submission), the session id (one execution, like a run), one trace per session, an attempt count on the repeatedrun_testsstep, and the pull request number that joins the session to the CI trace, which is a separate trace linked fromopen_pull_request. CI itself is on diff-to-deploy.sammuthu.com.
Checklist#
- Each identifier is documented once: what one value names, who mints it, and how long it lives.
run_idis minted where the run begins and is carried by every log line, span and audit event of the run, across every service.- A submission can have several runs, and nothing (counts, durations, idempotency keys) assumes one run per submission.
- The idempotency key of a write is derived from the run, not from the submission or the HTTP request.
attempt_numbercounts tries of one step inside one execution, and a redelivery is recorded separately (delivery_countor equivalent).- Work that arrives later, such as the underwriter's decision, carries
run_idand a span link back to the run. - No run-scoped id is a metric label.
- A test asserts that every record of a test run can be found by
run_id.
In one breath#
Every record carries several identifiers, and each names one thing with its own lifetime: the quote request (submission_id, years), the HTTP request (request_id, milliseconds), the run (run_id, until the human decision), the trace and the span, the try of a step (attempt_number), the underwriter's session. Join by the one that names what the question is about. Joining by submission_id made the two runs of SUB-1004 look like a duplicate write, and joining by the run's trace_id lost the decision of run-7f3a; run_id finds the whole run, and the idempotency key derived from it kept the double delivery of UW-504 to one stored recommendation.
Where this leads#
You can now pick the identifier that answers a question and say what each one misses. One of them, trace_id, carries more than a name: it holds a tree of steps with their timing and parents, and it has to travel from the gateway through the queue to the worker without being dropped. How spans are built, how traceparent carries the trace across processes, and how the span link reaches the decision trace is the subject of Tracing.