Everything on the earlier pages worked on the simulated day. The worker wrote structured logs; the trace of run-7f3a showed both timeouts of the roof-data provider; the SLO alert fired during the outage; the audit ledger held the underwriter's decision. The model recommends; a person decides, and the records prove who did what.
Now multiply. The insurer adds brokers, then regions. A platform engineer, wanting a per-run latency chart, adds run_id as a label on the span-duration metric. A week later the metrics store is slow and its bill has tripled. To save space, someone lowers the trace sampling rate to a quarter, and the next incident's one failing run is not among the kept traces. A broker asks to see its own submissions' runs in a dashboard, and the dashboard can see every broker's. And a person asks the insurer to delete their personal data, which sits in diagnostic logs, in traces and in an audit ledger that is built never to change.
None of these is a bug in the code of the earlier pages. Each is what happens when observability itself meets scale, people and law. This page takes them one at a time: cardinality, sampling at scale, exemplars, multi-tenancy, compliance, and watching the pipeline that watches everything else. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
1. Cardinality: when one label multiplies the bill#
Start with the plain idea. A metric is a number computed over many measurements, and a label is a name-value pair that splits it: route="/submissions" and route="/decisions" turn one duration metric into two lines on a chart. Each distinct combination of label values is its own time series, stored and indexed separately. Prometheus's naming guide says it in one sentence: "every unique combination of key-value label pairs represents a new time series".
Cardinality is the number of distinct values a label takes, and, by extension, the number of series a metric ends up with. Two labels with three values each allow up to nine series. Add a third label with fifty values and the ceiling is 450. Add a label whose value is new for every run, and the ceiling grows with traffic, forever.
The example measures this on the simulated day, over the span-duration histogram of all 48 runs (745 measurements). The command adds one candidate label at a time and counts the series, first as the theoretical ceiling (the product of distinct values), then as the day really produced them, then multiplied by the eight buckets a histogram stores per series.
cd "$TMPDIR/uw-observe-advanced" && /opt/homebrew/bin/python3 run_observe.py --show cardinality
== cardinality
metric span_duration_ms · 745 measurements · distinct values {'route': 2, 'status': 2, 'model': 3, 'tenant': 6, 'run_id': 48}
labels (none): at most 1 series, observed 1, x 8 buckets = 8
labels ['route']: at most 2 series, observed 2, x 8 buckets = 16
labels ['route', 'status']: at most 4 series, observed 3, x 8 buckets = 24
labels ['route', 'status', 'model']: at most 12 series, observed 5, x 8 buckets = 40
labels ['route', 'status', 'model', 'tenant']: at most 72 series, observed 28, x 8 buckets = 224
labels ['route', 'status', 'model', 'tenant', 'run_id']: at most 3456 series, observed 196, x 8 buckets = 1568
Read the last two lines together. Up to tenant, the labels have bounded values: two routes, two statuses, three model values, six brokers. The count grows, then stops. run_id has 48 values on a quiet simulated day and a new one for every run tomorrow. On this day it multiplies the stored counters by seven; on a production month it multiplies them by the number of runs. Nothing about the label is wrong in a log record or a span. On a metric it is the most expensive mistake on this page. These numbers are synthetic and illustrative.
| Labels on the duration metric | Ceiling (product of distinct values) | Series the day produced | Counters stored (x 8 buckets) |
|---|---|---|---|
| none | 1 | 1 | 8 |
| route | 2 | 2 | 16 |
| route, status | 4 | 3 | 24 |
| route, status, model | 12 | 5 | 40 |
| route, status, model, tenant | 72 | 28 | 224 |
| route, status, model, tenant, run_id | 3456 | 196 | 1568 |
The rule that follows is short. Identifiers go on logs and spans; bounded categories go on metrics. Prometheus's guide says not to use labels for "user IDs, email addresses, or other unbounded sets of values". Grafana's Loki documentation says the same for log labels and points high-cardinality data to structured metadata. When someone needs "the latency of this one run", the answer is the trace, not a metric label.
Before AI. Every team that ran a JVM service with Micrometer has met this, usually through auserIdor a URL with an id in its path used as a tag. The agent adds new temptations:modelis fine (a handful of values),prompt_versionis usually fine,run_id,submission_id,session_idand the tool's arguments are not.
Common confusion. High cardinality is not a problem of the trace store or the log store in the same way. A span carries run_id as an attribute and the store indexes it per span; that is what spans are for. The cost explodes when the identifier becomes part of the identity of a stored series.
Ceiling, observed and the histogram multiplier
For labels with distinct value counts d1 … dk, the ceiling is the product d1 × … × dk. The observed number is the count of distinct tuples among the measurements, always at most the ceiling and usually far below it, because labels are correlated: a model value appears only on model spans, an error status on few routes. The example computes both (cardinality.series_upper_bound, cardinality.observed_series).
A histogram stores one counter per bucket boundary plus an overflow counter for each series (cardinality.with_buckets), so a histogram's storage is the series count times the bucket count plus one. Adding buckets for resolution is therefore also a cardinality decision.
The OpenTelemetry metrics data model names a second cost. With cumulative temporality the sender must remember every series it has ever reported, "an 'up-front' memory cost proportional to cardinality"; delta temporality shifts that cost out of the process. An unbounded label therefore hurts twice: in the store, and in the memory of every process that emits the metric.
Budgets help. Set a ceiling on series per metric per service, alert when a deployment raises it, and review new labels the way you review a schema change.
2. Sampling at scale: head, tail and consistent#
Sampling means keeping only a share of the traces. Sensitive content introduced head sampling: the decision is made at the start of a trace, from the trace id alone. It is cheap, it needs no memory, and every service that sees the same trace id makes the same decision. OpenTelemetry's sampling page names the cost: with head sampling "it is not possible to make a sampling decision based on data in the entire trace". At the start of run-7f3a, nothing says the roof-data provider is about to time out.
Tail sampling decides after the trace is complete, so it can keep exactly what matters: traces with an error, slow traces, traces a quality rule flagged. The example combines the two, as OpenTelemetry's page describes: a head rate keeps a random share, and tail rules rescue the important traces that the head decision dropped.
cd "$TMPDIR/uw-observe-advanced" && /opt/homebrew/bin/python3 run_observe.py --show sampling
== sampling
head 25% + tail rules: kept 18 of 48 traces {'head': 10, 'tail:flagged': 1, 'tail:error': 6, 'tail:slow': 1} · stored 193 KB of 467 KB · run-7f3a kept: True (tail:error)
head 25% only: kept 10 of 48 traces {'head': 10} · stored 109 KB of 467 KB · run-7f3a kept: False (None)
The second line is the incident of the opening. With head sampling alone at 25 percent, the trace of run-7f3a was not kept: its trace id fell outside the kept quarter. With the tail rule "keep errors", it is kept, and so are the five other error traces of the day. The stored volume rises from 109 KB to 193 KB of 467 KB. That difference is the price of never losing an error trace. All sizes are from the synthetic day.
| Setting | Traces kept of 48 | Kept by | Stored | run-7f3a kept |
|---|---|---|---|---|
| Head 25 percent and tail rules (errors, slow, flagged) | 18 | head 10, error 6, slow 1, flagged 1 | 193 KB of 467 KB | yes, by the error rule |
| Head 25 percent only | 10 | head 10 | 109 KB of 467 KB | no |
The decision function is short. The head decision reads the first eight hex characters of the trace id as a number, so any service holding the same trace id decides the same way:
def head_keep(trace_id: str, rate: float) -> bool:
"""Deterministic head decision: the first 8 hex characters of the trace id as a number in [0, 1)."""
return int(trace_id[:8], 16) / 0x100000000 < rate
That property is called consistent sampling, and OpenTelemetry describes the same idea as consistent probability sampling: the decision is "made based on the trace ID and the desired percentage of traces to sample". Without it, the gateway keeps a trace that the worker drops, and the stored trace has a hole in the middle.
Tail sampling has an operational price, and the OpenTelemetry documentation is direct about it. Tail samplers "must be stateful systems that can accept and store a large amount of data", because every span of a trace must wait until the trace is complete. The Collector's tail sampling processor (stability "beta" for traces on the day of reading) waits 30 seconds by default (decision_wait) and requires that "all spans for a given trace MUST be received by the same collector instance". In practice that means two tiers: a first tier that routes spans by trace id with a load-balancing exporter, and a second tier that holds them and decides.
Production pitfall. A run of the underwriting agent is not over when the worker finishes. The underwriter's decision arrives hours later as a second trace, joined by a span link and run_id. No tail sampler waits hours. Decide the automated trace on its own, and make the rule for the decision trace separately: on this site, every decision trace is kept, because it is short and it is the evidence of who decided.
What this is not. Sampling decides which diagnostic traces are stored. It never applies to the audit ledger. A diagnostic record may be dropped; an audit event may not. If a sampler can drop it, it is not an audit event.
What sampling does to your metrics, and how to keep them honest
If dashboards are computed from stored traces, tail sampling biases them: the kept set over-represents errors and slow runs by design. Three practices keep numbers honest.
- Compute rates from metrics, not from sampled traces. Metrics are counted before sampling, so an error rate from a counter is exact while the trace store holds a skewed subset.
- Record the sampling weight. If a kept trace stands for four unsampled ones (head rate 25 percent), store that weight with it, so that counts computed from traces can be re-weighted. Tail-rescued traces carry weight one: they were kept because of what they are.
- Keep the head decision deterministic across services, so that partial traces never enter the store and distort span counts.
A last case for agents: a run that is slow because it loops (case UW-502, nine calls to search_guidance) is caught by a slow rule only if its duration crosses the threshold. A tail rule on span count, or on a cost attribute, catches it earlier. Cost uses that signal.
3. Exemplars: from a spike on a chart to one trace#
A latency chart over the outage shows the 95th percentile climbing above seven seconds. The on-call engineer wants one of those slow runs, now. The metric cannot tell which runs those were, and it must not carry run_id as a label (section 1). An exemplar bridges the two. The OpenTelemetry metrics data model defines it: "An exemplar is a recorded value that associates OpenTelemetry context to a metric event within a Metric". In practice it is one sample measurement stored next to a histogram bucket, with the trace_id and span_id of the run that produced it, the value and the time.
So the histogram stays low-cardinality, and a few points on the chart become links: click the dot in the seven-second bucket at 14:07 and land in a trace like the one of run-7f3a. Exemplars are a sample, not an index: they answer "show me one" and never "show me all".
Diagram: An exemplar: from a dot on a latency chart to one trace
Read left to right. The histogram keeps counts per bucket and stays low-cardinality. Next to a bucket it keeps a few sample measurements, each with the trace id of the run that produced it. Clicking that sample opens the trace. No run_id label was needed on the metric.
- 1 · The metric
- Duration histogram: Counts of run durations per bucket, split only by bounded labels.
- 2 · The exemplar
- One sample measurement: A recorded value stored next to the bucket with the trace and span ids of the run that produced it.
- 3 · The trace
- The slow run: The full span tree of the run, where the time went and why.
4. Multi-tenancy: brokers who must not see each other#
On the simulated day every run belongs to one of six brokers, broker-01 to broker-06. In this example a broker is the tenant: the customer whose records must be kept apart from other customers' records. Multi-tenancy means one telemetry system serves several tenants, and each can see only its own.
There are three places to draw the line, from weakest to strongest:
| Where the line is | How | What can go wrong |
|---|---|---|
| In the query | Every dashboard filters on a tenant attribute | One missing filter shows everyone's runs |
| In the store | The store keeps tenants apart and every query names one; Grafana Mimir describes itself as multi-tenant storage | A shared admin role still sees all |
| In separate systems | One store per tenant or per sensitivity class | Cost and operations multiply |
The tenant label is bounded (six values, and a contract per new one), so it may sit on metrics; section 1's table shows the series it adds. The tenant must also travel with the trace context so that every service tags its records, which is the same propagation problem as trace_id on Tracing.
The insurer has two kinds of reader inside, too. The platform team needs latency and errors for everyone; the underwriting team lead needs the content of runs for one line of business. Access by role is on Sensitive content; here the point is that the same split must hold in every backend that holds a copy.
5. Compliance: deletion, retention and residency#
Three demands arrive from outside engineering. This section describes the engineering conflict each creates; it does not state what any law requires. Ask counsel for that.
A deletion request. A person whose data appears in a submission asks for it to be deleted. In diagnostic logs and traces that is hard but possible when content was redacted at the source and identifiers are pseudonymous: delete or expire the few records that still hold it. In the audit ledger it collides with the design. The ledger is append-only and hash-chained; changing one entry breaks every hash after it, which is the point of the chain (Audit logs). The common design is to keep personal data out of the ledger in the first place: the audit event holds ids, rule ids, the decision and the role, and the personal data lives in a separate store that can delete it while the ledger keeps a reference that no longer resolves.
Retention. Each record type lives as long as its purpose. The example's illustrative policy (audit.RETENTION_DAYS) keeps traces 14 days, diagnostic logs 30 days, security logs 365 days and audit events about seven years. These numbers are invented for the example, not any regulator's rule. Written down per record type, they turn "how long do we keep logs?" from a debate into a configuration.
Residency. A rule may require that records about one region's customers stay in that region. Telemetry pipelines copy data by design: to a collector, to a backend, to a vendor, into a support ticket. Residency therefore has to be a routing rule in the pipeline (records tagged with a region go only to stores in that region), not a promise about one database.
In our insurance agent. The audit event forrun-7f3acarriessubmission_idSUB-1006, the rule ids,policy_version2026.2, the recommendation, andhuman_decisionwithdecided_by: "underwriter". It carries no email address, no building address and no name. That is a design decision made on the audit page so that this page's deletion request has an answer.
6. Observing the observers#
Every pipeline on this site can fail silently. A collector under load drops spans. A sampler is misconfigured to zero. An exporter's credentials expire at night. Each failure looks the same from a dashboard: a quiet chart. Quiet is the most dangerous state of an observability system, because it reads as health.
Four checks cover most of it:
- Count in and count out. Every collector reports records received, dropped and exported. Alert on drops, not on volume.
- A heartbeat run. A synthetic run every few minutes (in this example it could be the clean case
UW-002) must appear end to end: gateway log, trace, metric point. Alert when it does not. - Watch the telemetry of the telemetry system from somewhere else. If the metrics store is down, its own alert cannot fire.
- Test the expensive rules. A deployment that raises series counts or turns off redaction is caught in CI, not by the bill or by an auditor.
The same idea for a coding agent. A coding-agent session such asCA-001adds its own cardinality traps: a label per repository, per branch or per pull request on a metric multiplies series the same wayrun_iddid. Keep those as span attributes, and count sessions by bounded categories (outcome, test result).
Diagram: Four ways observability fails at scale, and the guard for each
Each lane is one failure that appears only when the system grows, with the guard that prevents it. None of them is a bug in the code of earlier pages; all of them are decisions about the observability system itself.
- Cardinality
- run_id on a metric: An identifier used as a metric label creates a new series for every run.
- Ids on spans only: Identifiers stay on logs and spans; metrics carry bounded labels and a budget.
- Sampling
- The failing run dropped: A decision at the start cannot know that the run will fail.
- Tail rule keeps errors: Keep every error, slow or flagged trace after completion; head decisions agree across services.
- Tenants
- A broker sees all brokers: Separation only in dashboard filters fails with one missing filter.
- Separate in the store: The store keeps tenants apart and every query names one.
- Compliance
- Personal data in the ledger: An append-only, hash-chained ledger cannot delete one field without breaking every later hash.
- Ids and decisions only: Audit events hold identifiers, rule ids and decisions; personal data lives in a store that can delete it.
Checklist#
- No metric carries
run_id,trace_id,submission_id, a session id or a tool argument as a label. - Each metric has a series budget, and a deployment that raises it is caught before release.
- Head sampling is deterministic from the trace id, so that every service keeps or drops the same traces.
- A tail rule keeps every error trace, and its collectors receive all spans of a trace on one instance.
- The decision trace and the audit events are outside every sampler.
- Exemplars link latency histograms to traces, so a spike leads to a run without a high-cardinality label.
- Tenants are separated in the store, not only in dashboard filters.
- Audit events hold identifiers and decisions, not personal data, so a deletion request can be met without breaking the chain.
- Collectors report drops, and a heartbeat run proves the pipeline end to end.
In one breath#
At scale, observability fails in its own ways: an unbounded label multiplies series until the store slows, a head sampler drops the one failing run while a tail sampler needs memory and routing to keep it, and tenants and the law decide who may see what and for how long. Keep identifiers on spans and logs, bounded categories on metrics, errors in every sample, personal data out of the audit ledger, and a heartbeat through the whole pipeline.
Where this leads#
You can now reason about the costs and failure modes that appear only when the system grows, and design against them. Every idea on this site has been shown one piece at a time. The last numbered page puts them back together on one run: case UW-006, from the gateway's first log line to the underwriter's decision, the audit event, the replay and the promotion to a golden case. That is the insurance lab.