Eight months after the simulated day, an auditor asks the insurer a question about submission SUB-1006. "Who decided this, on what basis, and under which version of your guidelines?" The engineer who gets the question opens the observability backend. The traces of that day expired after 14 days. The diagnostic logs were kept for 30. Even if they had survived, they were sampled, and the line that mattered might be one of the dropped ones. And the agent's own recommendation text says "the underwriter reviews this recommendation and decides", which is a sentence, not a record of a decision.
Every record on the earlier pages was built for engineers who debug. That purpose allows sampling, redaction, short retention and deletion, and it should: those choices keep telemetry affordable and safe. The auditor's question needs a different kind of record with the opposite promises. The model recommends; a person decides, and the business must be able to prove later which person decided what.
This page separates three logs that look alike in a terminal and make three different promises to three different readers. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
1. Three logs, three readers#
All three are lines of JSON with a timestamp. The difference is not the format. It is who reads them and what must remain true about them.
A diagnostic log is a log for engineers who need to understand and fix the system. In the example: "get_roof_assessment failed: timeout after 2000 ms, attempt_number=1". It may be sampled, redacted, aggregated and deleted after a few weeks, and nothing is lost that the business depends on.
An audit event is a record the business must be able to prove later: that something happened, who did it, and on what basis. In the example there are two kinds: recommendation_recorded (the agent stored a recommendation for the underwriter) and human_decision_recorded (the underwriter decided). An audit event must be complete (never sampled), unaltered (never edited), and kept for years. The store that holds audit events in order is the audit ledger.
A security log is a record of access, denials and abuse, read by the security team. In the example: a caller tried to write a recommendation without the permission to do so, and was denied (case UW-503), and a retrieved passage was flagged as untrusted because it contained injected instructions (case UW-005). It must be complete enough to reconstruct an attack and kept long enough to investigate one.
| Diagnostic log | Audit event | Security log | |
|---|---|---|---|
| Reader | Engineers, on call | The business, auditors | The security team |
| Question | What went wrong and why? | Who decided what, on what basis? | Who tried to do what, and was it allowed? |
| May be sampled | Yes | Never | Rarely; never for denials |
| May be redacted | Yes, by rule | It should not need to be: keep personal data out of it | Yes, by rule |
| May be changed | Deleted by retention | Never; only appended to | Never |
| Retention in the example (illustrative) | 30 days (traces 14) | About seven years (2,555 days) | 365 days |
| In the example | Logger.log lines | audit.Ledger.append | audit.SecurityLog.record |
The rule of this page, in one sentence: a diagnostic record may be dropped; an audit event may not.
Diagram: One run, three logs, three readers
Read each lane on its own. The same run of UW-006 writes into three stores with three promises. The diagnostic lane may be sampled, redacted and deleted. The audit lane is complete, append-only and kept for years. The security lane records attempts, including denied ones, for the security team.
- Diagnostic · engineers
- Timeout log line: A structured log for debugging: sampled, redacted and deleted after weeks.
- Kept 30 days: Short retention keeps cost and exposure down; nothing the business depends on is lost.
- Audit · the business
- human_decision_recorded: Who decided what, on what basis, under which policy version; complete and append-only.
- Kept for years: Every event carries the previous hash; a change breaks verification.
- Security · security team
- Denied write: An attempt without the required permission, recorded with the scope it lacked.
- Kept 365 days: Long enough to reconstruct an attack; denials are never dropped.
Before AI. A Java and Spring system had the same split, often without naming it. Application logs went through SLF4J to a log platform. Business records went to an audit table in the relational database, written in the same transaction as the change, with no update or delete grants for the application user. Access events went to a security system. An agent adds one twist: a step that used to be a button press in a web form is now a recommendation written by a model, and the record of that recommendation is evidence too.
2. The decision as an audit event#
Look at the two audit events of run-7f3a (case UW-006). At 14:07:19 the worker appends recommendation_recorded: recommendation request_info, no citations, evidence ids EV-APPL-1006 and EV-GUIDE-1006 (the roof assessment is missing, so it is not listed as evidence), policy_version 2026.2, and an idempotency_key derived from the run. Thirty-eight minutes later the underwriter decides in the cockpit, and the cockpit appends the second event:
{
"seq": 75,
"ts": "2026-09-14T14:45:19.256Z",
"type": "human_decision_recorded",
"actor": "underwriter",
"actor_type": "human",
"run_id": "run-7f3a",
"submission_id": "SUB-1006",
"trace_id": "1fb8aad7c68d8c80d7244e4d8780b70e",
"span_id": "9d96c98c958d5657",
"linked_trace_id": "7540bd55d4e55ccb8f04f3d81880717a",
"policy_version": "2026.2",
"recommendation": "request_info",
"human_decision": {
"decided_by": "underwriter",
"decision": "request_info",
"override": false,
"reason": "Agrees: ask the broker for an inspection report, because the roof-data provider did not answer."
},
"prev_hash": "003ca1054c6465d6009d61247565a12c89eadeaf76db03e213836e4bbda3eb58",
"hash": "4e2e26264384f78eac805c84e84469d07141d19d2a6cb211b542cc89229101b3"
}
This record answers the auditor. Who decided: actor_type human, decided_by the role underwriter (a real system would add an accountable identifier for the person; this site uses roles only). What was decided and whether it followed the agent: decision request_info, override false. On what basis: the recommendation it responded to, the reason, the policy_version. And how to find the engineering evidence if it still exists: trace_id is the cockpit's own trace, linked_trace_id is the run's trace, and run_id joins everything.
The human decision is the decision a person with authority makes on the agent's recommendation, recorded as human_decision: {decided_by, decision, override, reason}. Making it an audit event, rather than a log line or a span attribute, is what gives it the audit promises.
Notice also what the event does not hold: no email address, no building value, no name. It holds identifiers, rule ids, the decision and a role. That keeps the ledger out of the redaction and deletion problems of Sensitive content and Advanced.
In our insurance agent. In caseUW-009the agent recommendsapproveand the underwriter declines. The decision event carriesoverride: trueand the reason. The same record serves two readers: for the auditor it proves who decided; for the quality team it is one point of the override rate on Quality.
3. Tamper-evidence: the hash chain#
"Never edited" is a promise. A database administrator, a bug or an attacker can still change a row. Tamper-evidence means that any change can be detected afterwards, even if it cannot be prevented. The example gets it with a hash chain: each event stores the hash of the event before it, and its own hash covers its content plus that previous hash.
A hash is a short fingerprint of some text: change one character of the text and the fingerprint changes completely. The example uses SHA-256 from Python's standard library:
def event_hash(event: dict[str, Any]) -> str:
"""hash = SHA-256(prev_hash + canonical(event without its own hash))."""
body = {k: v for k, v in event.items() if k != "hash"}
return sha256_hex(event["prev_hash"] + canonical(body))
Appending sets prev_hash to the last event's hash (64 zeros for the first event), then computes the new event's hash. Verification walks the ledger from the start and recomputes every hash:
def verify_chain(events: list[dict[str, Any]]) -> dict[str, Any]:
"""Recompute every hash. Returns {ok, checked, broken_at (seq or None), reason}."""
prev = GENESIS
for i, e in enumerate(events):
if e["prev_hash"] != prev:
return {"ok": False, "checked": i + 1, "broken_at": e["seq"], "reason": "prev_hash does not match the event before"}
if event_hash(e) != e["hash"]:
return {"ok": False, "checked": i + 1, "broken_at": e["seq"], "reason": "content does not match its hash"}
prev = e["hash"]
return {"ok": True, "checked": len(events), "broken_at": None, "reason": "every hash matches"}
The runner verifies the day's ledger, then edits one field of one event (the underwriter's decision on run-7f3a, from request_info to approve) and verifies again:
cd "$TMPDIR/uw-observe-audit" && /opt/homebrew/bin/python3 run_observe.py --show ledger
== ledger
verify: {'ok': True, 'checked': 93, 'broken_at': None, 'reason': 'every hash matches'}
after changing seq 75 human_decision.decision to approve: {'ok': False, 'checked': 75, 'broken_at': 75, 'reason': 'content does not match its hash'}
The edit is caught at exactly the event that was changed. If the attacker also recomputed that event's hash, the next event's prev_hash would no longer match, and verification would stop at seq 76. To hide the change, they would have to rewrite every hash after it, which is why the latest hash is worth publishing somewhere the attacker cannot reach (section 5).
| Control in the lab | Effect on the debugging trace | Effect on the audit ledger |
|---|---|---|
| Apply sampling | The trace of the run may disappear | No effect: audit events are never sampled |
| Retention after N days | The trace is gone after 14 days, logs after 30 | Kept for the illustrative 2,555 days |
| Apply redaction | Email addresses and amounts masked in span content | No effect: the events hold no personal data |
| Tamper with one entry | Not applicable | Verification breaks at the edited event |
Production pitfall. Writing audit events through the same pipeline as diagnostic logs is the most common way to break the audit promise without noticing. The pipeline samples, drops under load, truncates long fields and expires records. Audit events need their own path: written synchronously with the action they record (in the same transaction where possible), acknowledged, and stored apart.
4. The security log: denials leave no audit trail#
Case UW-503 is a worker running with a credential that holds submissions:read but not recommendations:write. It tries to store a recommendation and is denied. The security log records it:
{
"ts": "2026-09-14T13:05:33.108Z",
"severity": "WARN",
"category": "security",
"service.name": "agent-worker",
"action": "record_recommendation",
"outcome": "denied",
"actor": "underwriting-agent",
"run_id": "run-cff5",
"request_id": "req-be3b09e5",
"trace_id": "feb20d47a884f29762c8de661042d953",
"resource": "SUB-1503",
"required_scope": "recommendations:write",
"held_scopes": [
"submissions:read"
]
}
And the audit ledger holds nothing for that run. That is correct: no recommendation was stored, so there is nothing for the business to prove. The difference between the two logs is the lesson. The audit ledger records what the business did. The security log records what someone attempted, including what they were not allowed to do. Ask the audit ledger "who tried to write without permission?" and it has no answer; ask the security log "who decided SUB-1006?" and it has none either.
On the simulated day the security log holds six entries: one denial (UW-503) and five flags of untrusted content in a retrieved passage, the injected text of case UW-005 showing up in runs that retrieved the same note. How to stop an injection is not this site's subject; see guardrails.sammuthu.com. Recording that it was seen is.
What this is not. Authorization, the check that denied UW-503, belongs to guardrails and tools engineering (guardrails.sammuthu.com, tools.sammuthu.com). Grading whether the underwriter's decision was right belongs to eval.sammuthu.com. This page is about what is recorded and what each record promises.
Diagram: The hash chain: change one event and verification stops there
Read left to right. Each event's hash covers its content and the hash of the event before. Verification recomputes every hash from the start. Editing the decision of seq 75 makes its content disagree with its hash; recomputing that hash makes seq 76 disagree instead. To hide an edit, every later hash must be rewritten.
- seq 74
- Event 74: Any earlier event; its hash is the next event's prev_hash.
- seq 75
- The decision: The underwriter's decision on run-7f3a, hashed together with the previous hash.
- Edited
- decision set to approve: The content no longer matches its stored hash; verification stops here.
- seq 76
- Event 76: If the attacker recomputes h75, this event's prev_hash no longer matches.
The same idea for a coding agent. A coding agent has the same three logs. Diagnostic: the tool calls and test output of session CA-001. Audit: that a pull request was opened by the agent and approved and merged by a named reviewer role, kept with the repository's history. Security: a tool call that the harness's permission rules denied. How a harness decides and logs permissions is covered on harness.sammuthu.com.
From a hash chain to a ledger you can defend
A hash chain in one database detects edits by anyone who does not also rewrite the rest of the chain. A determined insider with write access can rewrite it. Four additions close most of the gap.
- Anchor the head. Periodically publish the latest hash somewhere the ledger's operators cannot change: a separate account, a write-once store, a signed message to a second party. A rewritten chain then disagrees with the published anchor.
- Sign events. The service that appends an event signs it with a key the database administrators do not hold. A forged event fails the signature check even if its hash is consistent.
- Write once. Storage with object lock or write-once retention prevents changes during the retention period, instead of only detecting them.
- Verify on a schedule. A chain that is never verified is a chain whose break nobody notices. Verify daily and alert on failure.
Ordering is a design choice too. One chain per ledger gives a total order and a single writer bottleneck. One chain per submission or per tenant scales and still detects edits inside each chain, at the price of no order across chains.
Canonical serialization matters more than it looks: canonical sorts keys and fixes separators so that Python and the lab's JavaScript compute the same text and the same hash. A chain verified by a second implementation is a chain whose format is really specified.
Exactly one audit event per decision, even with retries
Case UW-504 delivers the same queue message twice. Two consumers run, with one run_id. If each appended recommendation_recorded, the ledger would claim two recommendations. The example prevents it with the idempotency_key (rec-run-7f3a style, derived from run_id): the second record_recommendation finds the key, stores nothing, and appends nothing. The trace shows two consumer spans; the ledger shows one event. That asymmetry is correct: the trace records attempts, the ledger records facts.
Checklist#
- Every record type is named as diagnostic, audit or security, with its reader and its promises written down.
- Audit events are written on their own path, never through the sampled diagnostic pipeline.
- The human decision is an audit event with
decided_by,decision,override,reason,policy_versionand the ids that join it to the run. - Audit events hold identifiers, rule ids and decisions, not personal data.
- The ledger is append-only and hash-chained, verified on a schedule, and its head is anchored outside the ledger.
- Denials and abuse go to the security log, which the security team reads, and are never sampled.
- Retention is set per record type: short for diagnostics, long for audit and security.
- An idempotency key makes a retried write produce one audit event.
In one breath#
Three logs look alike and promise different things: diagnostic records help engineers and may be sampled, redacted and deleted; audit events prove to the business who decided what and may never be dropped or changed; security records show who tried what and must last long enough to investigate. The underwriter's decision on run-7f3a is an audit event in a hash-chained ledger, so changing one field breaks verification at that event, and the denied write of UW-503 appears only in the security log, because nothing happened that the business must prove.
Where this leads#
You can now decide, for every record, which promise it makes and which store keeps that promise. The diagnostic records are still full of things nobody should read casually: the broker's email, the building value, the text of prompts and documents. What to capture, what to mask, who may read it, how long it lives and how much of it to keep is the subject of Sensitive content.