The morning after the simulated day, an engineer is asked to explain run-7f3a. That is case UW-006: the roof-data provider timed out twice, and the agent wrote a request_info recommendation for the underwriter. The model recommends; a person decides. The trace is clear about what happened. The engineer wants more than that. She has changed the retry logic overnight and wants to know whether her change would have handled that run differently. The obvious move is to run the agent again on submission SUB-1006 and watch.
She does. The new run recommends approve.
Nothing is wrong with her change. The roof-data provider recovered overnight and answered the second time around. A 12-year-old roof in wind zone B is a clean case, so approve is the right recommendation for the world of this morning. It tells her nothing about the world of 14:07 yesterday. She did not re-run that run. She ran a new one that happened to share an input.
This page is about the difference. A replay runs a recorded run again from captured inputs and fixtures, so that what changes between the original and the replay is only what you chose to change. What it takes, what it proves, and where it stops. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
What this is not. Replay is not resume. Resuming continues a saved run from its last checkpoint, with real effects, for the same submission; that is saved execution state and belongs to memory.sammuthu.com. A replay re-executes a finished run in a sandbox, writes nothing real, and exists to explain or to test. The example's replay writes to a throwaway store, ledger and log; it never touches the real audit ledger.
1. What a run depended on#
Start with a question: what made run-7f3a turn out the way it did? List everything the agent read from outside its own code.
- The input: the submission
SUB-1006, from the broker's email. - Tool responses: the two
get_roof_assessmentcalls, both timeouts. - Retrieved passages: what
search_guidanceandfind_similar_decisionsreturned at 14:07. - The policy version:
2026.2, the version of the policy fixture the run used. - Model outputs: what the model wrote at each model step (the plan, the recommendation).
- The clock: the time of day, which decides, among other things, whether the run fell into the provider outage.
- The seed: the source of every random choice in the run; in the example it drives the simulated time each model step takes.
A snapshot is a record of all of these, taken from the finished run. It is the run's inputs and the outside world's answers, frozen. Combined with the code, a snapshot makes the run a deterministic fixture: a test input whose every outside dependency answers the same way every time.
The example captures a snapshot from the finished UW-006 run in replay.py:
def capture_snapshot(run: dict[str, Any], case: dict[str, Any]) -> dict[str, Any]:
"""Everything a replay needs, taken from one finished run."""
calls = run["agent"]["tool_calls"]
return {"run_id": run["run_id"], "trace_id": run["trace_id"], "submission_id": run["submission_id"], "case": run["case"],
"variant": run["variant"], "input": case["input"],
"behaviour": {k: case[k] for k in ("agent_behaviour", "caller_scopes") if k in case},
"tool_responses": [c for c in calls if c["tool"] == "get_roof_assessment"],
"retrieved_passages": [c for c in calls if c["tool"] in ("search_guidance", "find_similar_decisions")],
"policy_version": run["policy_version"], "model_outputs": run["agent"]["model_outputs"],
"clock": {"day": DAY, "start_ms": run["agent"]["start_ms"]}, "seed": run["agent"]["seed"]}
The replay then runs the same agent code against a ReplayWorld: for each captured part it answers from the snapshot; for each missing part it answers as the outside world would answer now, the next morning at 09:00 simulated, when the provider has recovered and the guidance index was rebuilt.
Diagram: A replay: the same code, answered from the snapshot or from today
Read left to right. The snapshot holds what run-7f3a read from outside its code. The replay runs the same agent code. For each captured part the world answers from the snapshot; for each missing part it answers as it would the next morning. The comparison finds the first step where the two runs differ.
- 1 · Snapshot of run-7f3a
- Tool responses: What get_roof_assessment returned during the run.
- Model outputs: What the model wrote at each model step.
- Policy, clock, seed: The policy version, the time of the run and the source of randomness.
- 2 · Replay
- Same agent code: The agent runs again against a replay world and a throwaway store, ledger and log.
- 3 · Missing parts
- Answered by today: Any part not captured is answered by the outside world at replay time.
- 4 · Compare
- First divergence: Original and replay compared step by step, field by field.
2. Replay with parts missing#
The interesting experiment is not the full replay. It is the replay with one part left out, because that shows what each part protects against. The runner compares the original run with the replay step by step, field by field (name, attempt, status, output digest, duration, start), and prints the first place they differ.
cd "$TMPDIR/uw-observe-replay" && /opt/homebrew/bin/python3 run_observe.py --show replay
== replay
missing nothing reproduced True · recommendation request_info
missing seed reproduced False first divergence: step 0 plan_steps field duration_ms (812 -> 896) · recommendation request_info
missing tool_responses reproduced False first divergence: step 1 get_roof_assessment field status (error -> ok) · recommendation request_info
missing model_outputs reproduced False first divergence: step 6 write_recommendation field output (7341fe053f41 -> 8603dc4d43a3) · recommendation request_info
Four lines, four lessons.
- Everything captured: reproduced. Every step, every field, the same. Only now can a difference after a code change be blamed on the code change.
- Seed missing. The first difference appears at step 0: the stand-in's simulated model time draws from the seed, so the plan step takes 896 ms instead of 812. The recommendation is unchanged, and the trace is no longer the same trace. Without the seed, timings, retries with jitter and any random choice will differ.
- Tool responses missing. At step 1 the original call to
get_roof_assessmentfailed; the replay's call succeeds, because the provider of this morning answers. And look at the last column: the recommendation is stillrequest_info. The roof call succeeded, yet the agent asked for the roof age. That is not what a real agent would do; it happens because the recorded model output was replayed, and a recorded output does not read the new evidence. The replay is internally inconsistent, and nothing in the recommendation warns you. - Model outputs missing. Everything matches until step 6, where the stand-in writes the recommendation with different wording (a different output digest). The recommendation is the same. With a real model, this is the line that changes most often: the same prompt does not guarantee the same text.
The runner prints four combinations. Two more, from the same functions, complete the picture:
| Parts captured | First divergence | Replay recommends | What it means |
|---|---|---|---|
| All six | none | request_info | A faithful replay of run-7f3a |
| All but the seed | plan_steps duration, 812 to 896 ms | request_info | Same decision, different trace |
| All but tool responses | get_roof_assessment status, error to ok | request_info | The recorded model output hides the new evidence |
| All but model outputs | write_recommendation output wording | request_info | Same decision, different text |
| Only policy version, clock and seed | get_roof_assessment status, error to ok | approve | A new run of this morning, not a replay |
| Nothing | plan_steps duration | approve | The engineer's mistake from the opening |
The last row is the opening story, reproduced. Re-running with nothing captured gave approve, a different recommendation for a different world. In the lab, switch the parts on and off and watch the first divergence move. All of this is from the synthetic example; the stand-in's behavior is scripted.
In our insurance agent. The audit event ofrun-7f3arecordspolicy_version2026.2. That one field is what lets any replay use the right version of the policy fixture, months later, after the fixture has moved on. Every record on this site carries it for the same reason.
3. What a replay proves, and what it cannot#
A faithful replay proves one thing well: given exactly these inputs and these answers from the outside world, the code does this. That is enough to test a change to the code around the model: retry logic, rule checks, how the recommendation is stored. It is also how a failed production run becomes a regression test (Incidents does exactly that).
It cannot prove three things, and a careful report says so.
- What the model would write today. With recorded model outputs, the replay never calls the model; it plays back. Without them, the model is called again, and a live model can write something different for the same prompt. Either way, "the replay passed" says nothing about the next live run. Measuring how often a live model gets a case right is a repeated-trials question, owned by eval.sammuthu.com.
- How the outside world behaves. A recorded timeout is a fixed fact. The real provider's behavior under load is not in the snapshot.
- Anything the snapshot did not capture. Feature flags, configuration, the version of a library, a cache. The list in section 1 is the example's list; a production system must write its own and test that it is complete.
Common confusion. Reproducibility is a property of a run plus its snapshot: can it be executed again to the same result? A trace alone does not give it; a trace is a description. A snapshot alone does not give it either, unless the code version is pinned. Reproducible means snapshot, code version and replay harness together.
Before AI. Teams that recorded HTTP interactions for tests (a recorder that saves real responses and plays them back) or replayed a Kafka topic from an offset into a staging consumer used the same idea. Two things are new: the model is a dependency whose recorded output can contradict new evidence, as line 3 showed, and the snapshot contains prompts and documents that are sensitive.
Diagram: What a faithful replay proves, and what stays outside it
The left lane is what a replay with every part captured can show. The right lane is what no replay can show, because it depends on things the snapshot fixed or did not hold. The boundary decides which questions go to a replay and which to repeated live trials.
- Proves
- What the code does: Given the same inputs and answers, the code around the model behaves the same way.
- A regression test: A failed production run becomes a test that runs on every change.
- Cannot prove
- What a live model writes: Recorded outputs are played back; a live model may write something else for the same prompt.
- How the world behaves: A recorded timeout is fixed; the provider's real behavior is not in the snapshot.
- What was not captured: Any dependency missing from the snapshot answers from today.
4. Capturing without leaking#
A snapshot holds everything the run read: the broker's email, the submission, retrieved documents, model outputs. That makes a snapshot more sensitive than a trace with content capture switched off. Three rules follow from Sensitive content:
- Capture by reference where you can. Store the id and version of a retrieved passage, not its text, when the document store keeps versions.
- Redact before storing, with the same rules as traces, and test that a redacted snapshot still replays (an email address replaced by a fixed token usually does; a redacted number the rules depend on does not).
- Retain snapshots like diagnostic records, not like audit events. A snapshot is for explaining and testing. Keep the few that become golden cases under the evaluation set's rules, and let the rest expire.
The same idea for a coding agent. A coding-agent session such as CA-001 replays only if the repository commit, the instruction files, the tool outputs (including the failing test's log) and the model outputs were captured. Re-running the session on today's main branch is the coding agent's version of the opening mistake: a new session that shares a task description. How sessions are run and recorded is covered on harness.sammuthu.com.
Designing for replay from day one
Replay is cheap when the code was written for it and expensive when it was not. Four design choices make the difference.
- Every outside call goes through one interface (the
Worldin the example). A replay substitutes the interface, not twenty call sites. - Time and randomness are injected. The example's
SimClockand seededrandom.Randomare passed in, never read from globals. A code path that calls the system clock directly is a code path that will not replay. - Model calls are recorded with their full request, not only the output: model name and version, parameters, the full message list. A replay that plays back an output for a request that has since changed is the inconsistency of line 3, in production.
- Divergence is reported at the first step, by field. A replay that only compares final outputs would have called lines 2, 3 and 4 "reproduced", because the recommendation matched. The step-by-step comparison is what exposes them.
Partial replays have a use too. Replaying with recorded tool responses but a live model, many times, is a controlled way to measure model variation on a fixed real case. That measurement and its statistics belong to evaluation; the snapshot that makes it possible belongs here.
Checklist#
- Every run records what it read from outside its code: tool responses, retrieved passages, policy version, model outputs, clock and seed.
- Time and randomness are injected, so a replay can control them.
- Outside calls go through one interface that a replay can substitute.
- Replays write to a sandbox, never to the real store or the audit ledger.
- The comparison is step by step and field by field, and reports the first divergence.
- A replay report states which parts were captured and which were live.
- Snapshots are redacted with the trace rules, and expire unless promoted to a golden case.
- Nobody calls a new run on the same input a replay.
In one breath#
A trace shows what happened; a replay runs it again from a snapshot of everything the run read from outside its code. With every part captured, run-7f3a reproduces exactly; with the tool responses missing, the provider of the next morning answers and the recorded model output hides the change; with nothing captured, the "replay" is a new run that recommends approve. A faithful replay proves what your code does with fixed inputs, never what a live model will do next time.
Where this leads#
You can now explain one run by re-running it, and say exactly what the replay proves. The next page puts that to work under pressure: an alert at 14:10, a root cause to find, and a failed run that should never fail the same way again. That is Incidents.