All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Operations · 12

Reproducibility and replay: fixtures, snapshots and their limits

A trace shows what happened. A replay runs it again. What has to be captured for a second run to mean anything, and what can never be reproduced.

Shows versus re-runs
A trace shows what happened. A replay runs it again from what was captured, and compares.
Six parts to capture
Tool responses, retrieved passages, policy version, model outputs, clock and seed. Miss one and the replay drifts.
What it proves
A faithful replay proves your code does the same thing with the same inputs. It never proves the model would.

The morning after the simulated day, an engineer is asked to explain run-7f3a. That is case UW-006: the roof-data provider timed out twice, and the agent wrote a request_info recommendation for the underwriter. The model recommends; a person decides. The trace is clear about what happened. The engineer wants more than that. She has changed the retry logic overnight and wants to know whether her change would have handled that run differently. The obvious move is to run the agent again on submission SUB-1006 and watch.

She does. The new run recommends approve.

Nothing is wrong with her change. The roof-data provider recovered overnight and answered the second time around. A 12-year-old roof in wind zone B is a clean case, so approve is the right recommendation for the world of this morning. It tells her nothing about the world of 14:07 yesterday. She did not re-run that run. She ran a new one that happened to share an input.

This page is about the difference. A replay runs a recorded run again from captured inputs and fixtures, so that what changes between the original and the replay is only what you chose to change. What it takes, what it proves, and where it stops. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.

What this is not. Replay is not resume. Resuming continues a saved run from its last checkpoint, with real effects, for the same submission; that is saved execution state and belongs to memory.sammuthu.com. A replay re-executes a finished run in a sandbox, writes nothing real, and exists to explain or to test. The example's replay writes to a throwaway store, ledger and log; it never touches the real audit ledger.

1. What a run depended on#

Start with a question: what made run-7f3a turn out the way it did? List everything the agent read from outside its own code.

A snapshot is a record of all of these, taken from the finished run. It is the run's inputs and the outside world's answers, frozen. Combined with the code, a snapshot makes the run a deterministic fixture: a test input whose every outside dependency answers the same way every time.

The example captures a snapshot from the finished UW-006 run in replay.py:

def capture_snapshot(run: dict[str, Any], case: dict[str, Any]) -> dict[str, Any]:
    """Everything a replay needs, taken from one finished run."""
    calls = run["agent"]["tool_calls"]
    return {"run_id": run["run_id"], "trace_id": run["trace_id"], "submission_id": run["submission_id"], "case": run["case"],
            "variant": run["variant"], "input": case["input"],
            "behaviour": {k: case[k] for k in ("agent_behaviour", "caller_scopes") if k in case},
            "tool_responses": [c for c in calls if c["tool"] == "get_roof_assessment"],
            "retrieved_passages": [c for c in calls if c["tool"] in ("search_guidance", "find_similar_decisions")],
            "policy_version": run["policy_version"], "model_outputs": run["agent"]["model_outputs"],
            "clock": {"day": DAY, "start_ms": run["agent"]["start_ms"]}, "seed": run["agent"]["seed"]}

The replay then runs the same agent code against a ReplayWorld: for each captured part it answers from the snapshot; for each missing part it answers as the outside world would answer now, the next morning at 09:00 simulated, when the provider has recovered and the guidance index was rebuilt.

Diagram: A replay: the same code, answered from the snapshot or from today

Read left to right. The snapshot holds what run-7f3a read from outside its code. The replay runs the same agent code. For each captured part the world answers from the snapshot; for each missing part it answers as it would the next morning. The comparison finds the first step where the two runs differ.

  1. 1 · Snapshot of run-7f3a
    1. Tool responses: What get_roof_assessment returned during the run.
    2. Model outputs: What the model wrote at each model step.
    3. Policy, clock, seed: The policy version, the time of the run and the source of randomness.
  2. 2 · Replay
    1. Same agent code: The agent runs again against a replay world and a throwaway store, ledger and log.
  3. 3 · Missing parts
    1. Answered by today: Any part not captured is answered by the outside world at replay time.
  4. 4 · Compare
    1. First divergence: Original and replay compared step by step, field by field.

2. Replay with parts missing#

The interesting experiment is not the full replay. It is the replay with one part left out, because that shows what each part protects against. The runner compares the original run with the replay step by step, field by field (name, attempt, status, output digest, duration, start), and prints the first place they differ.

cd "$TMPDIR/uw-observe-replay" && /opt/homebrew/bin/python3 run_observe.py --show replay
== replay
missing nothing          reproduced True   · recommendation request_info
missing seed             reproduced False first divergence: step 0 plan_steps field duration_ms (812 -> 896) · recommendation request_info
missing tool_responses   reproduced False first divergence: step 1 get_roof_assessment field status (error -> ok) · recommendation request_info
missing model_outputs    reproduced False first divergence: step 6 write_recommendation field output (7341fe053f41 -> 8603dc4d43a3) · recommendation request_info

Four lines, four lessons.

  1. Everything captured: reproduced. Every step, every field, the same. Only now can a difference after a code change be blamed on the code change.
  2. Seed missing. The first difference appears at step 0: the stand-in's simulated model time draws from the seed, so the plan step takes 896 ms instead of 812. The recommendation is unchanged, and the trace is no longer the same trace. Without the seed, timings, retries with jitter and any random choice will differ.
  3. Tool responses missing. At step 1 the original call to get_roof_assessment failed; the replay's call succeeds, because the provider of this morning answers. And look at the last column: the recommendation is still request_info. The roof call succeeded, yet the agent asked for the roof age. That is not what a real agent would do; it happens because the recorded model output was replayed, and a recorded output does not read the new evidence. The replay is internally inconsistent, and nothing in the recommendation warns you.
  4. Model outputs missing. Everything matches until step 6, where the stand-in writes the recommendation with different wording (a different output digest). The recommendation is the same. With a real model, this is the line that changes most often: the same prompt does not guarantee the same text.

The runner prints four combinations. Two more, from the same functions, complete the picture:

Parts capturedFirst divergenceReplay recommendsWhat it means
All sixnonerequest_infoA faithful replay of run-7f3a
All but the seedplan_steps duration, 812 to 896 msrequest_infoSame decision, different trace
All but tool responsesget_roof_assessment status, error to okrequest_infoThe recorded model output hides the new evidence
All but model outputswrite_recommendation output wordingrequest_infoSame decision, different text
Only policy version, clock and seedget_roof_assessment status, error to okapproveA new run of this morning, not a replay
Nothingplan_steps durationapproveThe engineer's mistake from the opening

The last row is the opening story, reproduced. Re-running with nothing captured gave approve, a different recommendation for a different world. In the lab, switch the parts on and off and watch the first divergence move. All of this is from the synthetic example; the stand-in's behavior is scripted.

In our insurance agent. The audit event of run-7f3a records policy_version 2026.2. That one field is what lets any replay use the right version of the policy fixture, months later, after the fixture has moved on. Every record on this site carries it for the same reason.

3. What a replay proves, and what it cannot#

A faithful replay proves one thing well: given exactly these inputs and these answers from the outside world, the code does this. That is enough to test a change to the code around the model: retry logic, rule checks, how the recommendation is stored. It is also how a failed production run becomes a regression test (Incidents does exactly that).

It cannot prove three things, and a careful report says so.

Common confusion. Reproducibility is a property of a run plus its snapshot: can it be executed again to the same result? A trace alone does not give it; a trace is a description. A snapshot alone does not give it either, unless the code version is pinned. Reproducible means snapshot, code version and replay harness together.
Before AI. Teams that recorded HTTP interactions for tests (a recorder that saves real responses and plays them back) or replayed a Kafka topic from an offset into a staging consumer used the same idea. Two things are new: the model is a dependency whose recorded output can contradict new evidence, as line 3 showed, and the snapshot contains prompts and documents that are sensitive.

Diagram: What a faithful replay proves, and what stays outside it

The left lane is what a replay with every part captured can show. The right lane is what no replay can show, because it depends on things the snapshot fixed or did not hold. The boundary decides which questions go to a replay and which to repeated live trials.

  1. Proves
    1. What the code does: Given the same inputs and answers, the code around the model behaves the same way.
    2. A regression test: A failed production run becomes a test that runs on every change.
  2. Cannot prove
    1. What a live model writes: Recorded outputs are played back; a live model may write something else for the same prompt.
    2. How the world behaves: A recorded timeout is fixed; the provider's real behavior is not in the snapshot.
    3. What was not captured: Any dependency missing from the snapshot answers from today.

4. Capturing without leaking#

A snapshot holds everything the run read: the broker's email, the submission, retrieved documents, model outputs. That makes a snapshot more sensitive than a trace with content capture switched off. Three rules follow from Sensitive content:

  1. Capture by reference where you can. Store the id and version of a retrieved passage, not its text, when the document store keeps versions.
  2. Redact before storing, with the same rules as traces, and test that a redacted snapshot still replays (an email address replaced by a fixed token usually does; a redacted number the rules depend on does not).
  3. Retain snapshots like diagnostic records, not like audit events. A snapshot is for explaining and testing. Keep the few that become golden cases under the evaluation set's rules, and let the rest expire.
The same idea for a coding agent. A coding-agent session such as CA-001 replays only if the repository commit, the instruction files, the tool outputs (including the failing test's log) and the model outputs were captured. Re-running the session on today's main branch is the coding agent's version of the opening mistake: a new session that shares a task description. How sessions are run and recorded is covered on harness.sammuthu.com.
Designing for replay from day one

Replay is cheap when the code was written for it and expensive when it was not. Four design choices make the difference.

  • Every outside call goes through one interface (the World in the example). A replay substitutes the interface, not twenty call sites.
  • Time and randomness are injected. The example's SimClock and seeded random.Random are passed in, never read from globals. A code path that calls the system clock directly is a code path that will not replay.
  • Model calls are recorded with their full request, not only the output: model name and version, parameters, the full message list. A replay that plays back an output for a request that has since changed is the inconsistency of line 3, in production.
  • Divergence is reported at the first step, by field. A replay that only compares final outputs would have called lines 2, 3 and 4 "reproduced", because the recommendation matched. The step-by-step comparison is what exposes them.

Partial replays have a use too. Replaying with recorded tool responses but a live model, many times, is a controlled way to measure model variation on a fixed real case. That measurement and its statistics belong to evaluation; the snapshot that makes it possible belongs here.

Checklist#

In one breath#

A trace shows what happened; a replay runs it again from a snapshot of everything the run read from outside its code. With every part captured, run-7f3a reproduces exactly; with the tool responses missing, the provider of the next morning answers and the recorded model output hides the change; with nothing captured, the "replay" is a new run that recommends approve. A faithful replay proves what your code does with fixed inputs, never what a live model will do next time.

Where this leads#

You can now explain one run by re-running it, and say exactly what the replay proves. The next page puts that to work under pressure: an alert at 14:10, a root cause to find, and a failed run that should never fail the same way again. That is Incidents.