The platform engineer turns on content capture for model spans during the first week of production. It is the fastest way to debug: when a recommendation looks odd, the span shows exactly what the model read and what it wrote. On the simulated day, the plan_steps span of run-7f3a holds the broker's email as it arrived: the sender's address, the insurer's intake address, the building, the roof age, the wind zone. The submission inside it carries the building value. The model recommends; a person decides, and to explain its recommendation the trace keeps a copy of everything it saw.
Three weeks later the copy is everywhere. It is in the collector's buffer, in the trace backend, in a vendor's hosted service, in an export someone made for an analysis, and in a screenshot pasted into an incident channel. The underwriting team never sees most of these places. The broker never agreed to any of them. Nothing leaked in the sense of an attack. The telemetry system did what it was built to do: it copied records to every place that might need them.
This page is about the decisions that keep that copy small and safe: what to capture at all, what to mask before it leaves the process, who may read what remains, how long each kind of record lives, and how much of it to keep. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
1. What makes trace content sensitive#
Personal data is information about an identifiable person: a name, an email address, a phone number, and often less obvious things such as a building address that identifies an owner. In the example, the broker's email address is personal data, and the building value is confidential business data. Both arrive in the email that starts the run.
A conventional trace rarely holds such content. An HTTP span records a route, a status and a duration, not the request body. A model span is different, because the interesting part of a model call is its content: the prompt, the retrieved documents, the output. GenAI telemetry calls recording it content capture, and treats it as a decision, not a default. The OpenTelemetry GenAI conventions agree: instrumentations "SHOULD NOT capture them by default", and the content switch OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT defaults to NO_CONTENT. The same conventions describe a third option for production: store content externally and record a reference on the span.
Plain English. A trace is a copy machine. Whatever you put on a span is copied to every store, screen and export downstream. Decide what goes on the span as if it will be read by everyone who can read any trace, because eventually it will be.
2. Redaction: mask at the source#
Redaction means removing or masking sensitive content before it is stored. The example's redaction.py has four rules, each a sentence:
| Rule | What it does |
|---|---|
drop_content | Do not store prompt and response text at all; keep its size |
email | Replace every email address with [EMAIL] |
email_keep_domain | Replace the part of an email before @ with [USER]; keep the domain |
money | Replace every field that ends in _usd with [AMOUNT] |
redact_span applies a set of rules to one span and returns the span as it would be stored, without changing the original. Then a second function asks what an engineer can still answer from the stored version. That second step is the point: every rule removes something an attacker could read, and something a debugger could use. The runner shows four settings on the plan_steps span of run-7f3a:
cd "$TMPDIR/uw-observe-sensitive" && /opt/homebrew/bin/python3 run_observe.py --show redact
== redact
rules none: masked 0 · answerable 6 of 6 · first line: {"input": {"source_email": "From: broker@brokerage.example\nTo: submissions@insurer.exampl
rules ['email_keep_domain']: masked 2 · answerable 5 of 6 · first line: {"input": {"source_email": "From: [USER]@brokerage.example\nTo: [USER]@insurer.example\nSu
rules ['email', 'money']: masked 3 · answerable 3 of 6 · first line: {"input": {"source_email": "From: [EMAIL]\nTo: [EMAIL]\nSubject: Quote request SUB-1006\n\
rules ['drop_content']: masked 0 · answerable 2 of 6 · first line: {"captured": false, "chars": 325}
The six questions are the ones an engineer asks during debugging: which step ran and how long it took; which run and submission it was; which brokerage sent the request; which person at the brokerage; what the model read; what building value the model saw.
| Rules on | Values masked | Questions still answerable (of 6) | What is lost |
|---|---|---|---|
| none | 0 | 6 | Nothing; everything is exposed |
email_keep_domain | 2 | 5 | Which person sent it; the brokerage is still visible |
email, money | 3 | 3 | The brokerage, the person and the building value |
drop_content | 0 (content not stored) | 2 | Everything about content; only step, timing and ids remain |
Read the table as a set of trade-offs, not a ranking. email_keep_domain is a good default for this agent: the brokerage matters for debugging (one brokerage's emails might be formatted differently), the person does not. drop_content is right for production spans that nobody reads, with a separate, access-controlled path for the few runs that need their content (for example, flagged runs in the review queue of Quality). Every number here is from the synthetic example.
Where redaction runs matters as much as what it masks.
- At the source (in the process, before export): the content never leaves. A bug in a downstream system cannot expose what it never received.
- In the pipeline (in a collector processor): one place to change rules for every service. The content still crossed the network to the collector, and anything that taps the process's export sees it.
The safest design does both: coarse rules at the source (no content, or no personal data) and finer rules in the pipeline. Redaction after storage, in the backend's display, protects the screen and nothing else.
Common confusion. Redaction is not encryption. An encrypted span can be decrypted by anyone with the key, and the key is usually held by the backend. A redacted span no longer holds the value at all.
Production pitfall. Pattern rules miss what they do not match. An email address written as "broker at brokerage dot example" passes themoneyrule, which only reads fields ending in_usd. Test redaction against real-looking content, and prefer not capturing a field over masking it.
3. Access control: who may read what remains#
Access control decides which people and systems may read which records. After redaction, some content remains, and not every reader needs it.
| Reader | Needs | Should see |
|---|---|---|
| On-call engineer | Is the system healthy? Which step failed? | Metrics, diagnostic logs, spans without content |
| Platform engineer debugging one run | What did the model read? | Content of that run, through an audited access path |
| Underwriting team lead | Why did the agent recommend this? | The run in the underwriter cockpit, with the recommendation and its evidence |
| Security team | Who tried what? | The security log |
| Auditor | Who decided what? | The audit ledger |
Two design rules follow. Split stores by sensitivity, so that access follows the store (a content store separate from the span store, for example). And log access to content: reading the content of a run is itself a security-log event.
What this is not. Deciding whether the agent may read the broker's email at all, and checking permissions on its tools, belongs to guardrails.sammuthu.com. This page is about who may read the records of what the agent did.
4. Retention: each record lives as long as its purpose#
Retention is how long a record is kept before it is deleted. One retention period for "all logs" is either too long for diagnostics or too short for audit. The example sets it per record type (audit.RETENTION_DAYS; illustrative numbers, not any regulator's rule):
| Record type | Kept (days) | Why that long |
|---|---|---|
| Trace | 14 | Debugging recent runs; the most content, the shortest life |
| Diagnostic log | 30 | Incidents and trends of the last month |
| Security log | 365 | Long enough to investigate an attack found late |
| Audit event | 2,555 (about seven years) | The business must be able to prove decisions for years |
The ledger lab shows what retention does to each record of run-7f3a: after 30 days the trace and logs are gone and the audit events remain, which is why the audit events must not depend on them.
5. Sampling: keep less, expose less#
Sampling means keeping only a share of the traces. It is usually introduced as a cost control, and it is also an exposure control: a trace that was not kept cannot leak. Head sampling decides at the start of a trace, from the trace id alone, whether to keep it. The example keeps a trace when the first eight hex characters of its trace id, read as a number between 0 and 1, fall below the rate. Because the decision depends only on the trace id, every service that sees the same id makes the same decision, and the W3C traceparent header carries a sampled flag so that downstream services can follow the caller's decision.
Head sampling has one weakness for an agent: at the start of a run nothing says it will fail. On the simulated day, head sampling at 25 percent keeps 10 of 48 traces and drops the trace of run-7f3a. Adding tail rules (keep every trace with an error, a slow run or a quality flag) keeps 18, including run-7f3a.
| Setting | Traces kept of 48 | Stored | run-7f3a kept |
|---|---|---|---|
| Head 25 percent and tail rules | 18 | 193 KB of 467 KB | yes, by the error rule |
| Head 25 percent only | 10 | 109 KB of 467 KB | no |
How tail sampling works at scale, and what it costs, is on Advanced. The rule that matters here: sampling applies to diagnostic traces only. A diagnostic record may be dropped; an audit event may not.
Diagram: Five controls on one model span, in the order they act
Read left to right. Each control removes exposure at a different point. Capture decides whether content is recorded at all. Redaction masks what is recorded before it leaves the process. Sampling decides whether the trace is kept. Access control decides who can read it. Retention decides how long it exists.
- 1 · Capture
- Record content?: Decide per span kind and environment whether prompts, documents and outputs go on the span.
- 2 · Redact
- Mask at the source: Rules mask emails, amounts or drop the text before the span leaves the process.
- 3 · Sample
- Keep this trace?: Keep a share by trace id, plus every error, slow or flagged trace.
- 4 · Access
- Who may read it?: Stores split by sensitivity; reading content is itself logged.
- 5 · Retain
- How long?: Each record type has its own retention period.
Before AI. A Spring service that logged full request bodies at DEBUG level in production had the same problem on a smaller scale, and the usual fix was the same: a logging filter that masked fields by name before the appender wrote them, plus a retention policy on the log index. The agent raises the stakes, because the body is now the most useful part of the record and the prompt mixes the user's data with retrieved documents.
Diagram: Where redaction runs decides what it protects
Each lane is one place redaction can run. At the source, content never leaves the process. In the pipeline, rules live in one place but the content crossed the network first. In the backend's display, only the screen is protected; the stored copy is complete.
- In the process
- At the source: Rules run before export; nothing downstream ever receives the value.
- In the collector
- In the pipeline: One configuration for every service; the content has already crossed the network.
- In the backend
- On the screen: Masking in a display hides the value from viewers; exports and queries still return it.
Store content by reference, not on the span
The GenAI conventions describe storing content externally and recording a reference as the pattern recommended for production. The design has three parts.
- The span records a reference (a content id and a hash), the size, and the redaction rules applied. It stays small and safe to export anywhere.
- The content goes to a separate store with its own access control, its own retention (often shorter than the span's), and access logging.
- Readers resolve the reference only when they need the content and are allowed to see it.
The benefits are concrete: spans can go to a hosted backend without the content; deleting a person's data means deleting from one store; the content store can expire after a week while the span lives for two. The cost is a second store and a resolver in the debugging tools.
The same idea keeps snapshots for Replay manageable: a retrieved passage is recorded by document id and version, not by text, when the document store keeps versions.
Testing that redaction works
Redaction fails silently: a missed field is stored like any other. Three tests catch most misses.
- A canary value. Put a unique, fake address (for example
canary-7f3a@brokerage.example) into a test submission, run it through the full pipeline, and search every store for it. Any hit is a leak. - A schema of allowed attributes. Exported spans may carry only attributes from an allow list. A new attribute fails the test until someone decides how to treat it.
- Debuggability check. The other direction: after redaction, the questions the on-call engineer must answer (step, timing, ids, error) are still answerable. The example's
debugger_questionsis a small version of this.
The same idea for a coding agent. A coding-agent session such as CA-001 reads repository files that may hold secrets (configuration, keys committed by mistake) and writes tool outputs that echo them. The same controls apply: do not capture file content on spans by default, mask secret patterns at the source, and keep session transcripts in an access-controlled store with short retention. How a harness limits what the agent may read is covered on harness.sammuthu.com.
Checklist#
- Content capture on model, retrieval and tool spans is off by default and switched on deliberately, per environment.
- Redaction rules run in the process before export, and again in the pipeline.
- Each rule is tested with a canary value searched for in every store.
- After redaction, the on-call engineer can still answer which step failed, when, and for which run.
- Content, when kept, lives in a separate store with its own access control, retention and access log.
- Retention is set per record type, and audit events never depend on diagnostic retention.
- Head sampling is consistent by trace id, and tail rules keep every error trace.
In one breath#
A trace of an agent is a copy of what the agent saw, and telemetry copies it everywhere. Decide what to capture, mask what you keep before it leaves the process, and know which questions each rule takes away: on run-7f3a, masking emails and amounts leaves three of six debugging questions answerable, dropping content leaves two. Give each reader only the records they need, keep each record type only as long as its purpose, and sample diagnostic traces, never audit events.
Where this leads#
You now know what to keep and how to keep it safe. The records still have to go somewhere: through a collector and a pipeline into a store that can be searched. The path a record takes after it leaves the process, and what five widely used backends do at each step, is the subject of Backends.