All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Reference · 16

Advanced: cardinality, tail sampling, tenants and compliance

Where observability itself fails at scale: a label that creates a million series, a sampler that drops the one trace you need, tenants who must not see each other, and records the law makes you keep.

Too many series
One label with an unbounded value turns one metric into hundreds of series, then millions.
The trace you dropped
A sampler that decides at the start cannot know which run will fail. One that decides at the end must hold every span.
Who may see what
Tenants, deletion requests and residency rules turn telemetry into regulated data.

Everything on the earlier pages worked on the simulated day. The worker wrote structured logs; the trace of run-7f3a showed both timeouts of the roof-data provider; the SLO alert fired during the outage; the audit ledger held the underwriter's decision. The model recommends; a person decides, and the records prove who did what.

Now multiply. The insurer adds brokers, then regions. A platform engineer, wanting a per-run latency chart, adds run_id as a label on the span-duration metric. A week later the metrics store is slow and its bill has tripled. To save space, someone lowers the trace sampling rate to a quarter, and the next incident's one failing run is not among the kept traces. A broker asks to see its own submissions' runs in a dashboard, and the dashboard can see every broker's. And a person asks the insurer to delete their personal data, which sits in diagnostic logs, in traces and in an audit ledger that is built never to change.

None of these is a bug in the code of the earlier pages. Each is what happens when observability itself meets scale, people and law. This page takes them one at a time: cardinality, sampling at scale, exemplars, multi-tenancy, compliance, and watching the pipeline that watches everything else. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.

1. Cardinality: when one label multiplies the bill#

Start with the plain idea. A metric is a number computed over many measurements, and a label is a name-value pair that splits it: route="/submissions" and route="/decisions" turn one duration metric into two lines on a chart. Each distinct combination of label values is its own time series, stored and indexed separately. Prometheus's naming guide says it in one sentence: "every unique combination of key-value label pairs represents a new time series".

Cardinality is the number of distinct values a label takes, and, by extension, the number of series a metric ends up with. Two labels with three values each allow up to nine series. Add a third label with fifty values and the ceiling is 450. Add a label whose value is new for every run, and the ceiling grows with traffic, forever.

The example measures this on the simulated day, over the span-duration histogram of all 48 runs (745 measurements). The command adds one candidate label at a time and counts the series, first as the theoretical ceiling (the product of distinct values), then as the day really produced them, then multiplied by the eight buckets a histogram stores per series.

cd "$TMPDIR/uw-observe-advanced" && /opt/homebrew/bin/python3 run_observe.py --show cardinality
== cardinality
metric span_duration_ms · 745 measurements · distinct values {'route': 2, 'status': 2, 'model': 3, 'tenant': 6, 'run_id': 48}
  labels (none): at most 1 series, observed 1, x 8 buckets = 8
  labels ['route']: at most 2 series, observed 2, x 8 buckets = 16
  labels ['route', 'status']: at most 4 series, observed 3, x 8 buckets = 24
  labels ['route', 'status', 'model']: at most 12 series, observed 5, x 8 buckets = 40
  labels ['route', 'status', 'model', 'tenant']: at most 72 series, observed 28, x 8 buckets = 224
  labels ['route', 'status', 'model', 'tenant', 'run_id']: at most 3456 series, observed 196, x 8 buckets = 1568

Read the last two lines together. Up to tenant, the labels have bounded values: two routes, two statuses, three model values, six brokers. The count grows, then stops. run_id has 48 values on a quiet simulated day and a new one for every run tomorrow. On this day it multiplies the stored counters by seven; on a production month it multiplies them by the number of runs. Nothing about the label is wrong in a log record or a span. On a metric it is the most expensive mistake on this page. These numbers are synthetic and illustrative.

Labels on the duration metricCeiling (product of distinct values)Series the day producedCounters stored (x 8 buckets)
none118
route2216
route, status4324
route, status, model12540
route, status, model, tenant7228224
route, status, model, tenant, run_id34561961568

The rule that follows is short. Identifiers go on logs and spans; bounded categories go on metrics. Prometheus's guide says not to use labels for "user IDs, email addresses, or other unbounded sets of values". Grafana's Loki documentation says the same for log labels and points high-cardinality data to structured metadata. When someone needs "the latency of this one run", the answer is the trace, not a metric label.

Before AI. Every team that ran a JVM service with Micrometer has met this, usually through a userId or a URL with an id in its path used as a tag. The agent adds new temptations: model is fine (a handful of values), prompt_version is usually fine, run_id, submission_id, session_id and the tool's arguments are not.
Common confusion. High cardinality is not a problem of the trace store or the log store in the same way. A span carries run_id as an attribute and the store indexes it per span; that is what spans are for. The cost explodes when the identifier becomes part of the identity of a stored series.
Ceiling, observed and the histogram multiplier

For labels with distinct value counts d1 … dk, the ceiling is the product d1 × … × dk. The observed number is the count of distinct tuples among the measurements, always at most the ceiling and usually far below it, because labels are correlated: a model value appears only on model spans, an error status on few routes. The example computes both (cardinality.series_upper_bound, cardinality.observed_series).

A histogram stores one counter per bucket boundary plus an overflow counter for each series (cardinality.with_buckets), so a histogram's storage is the series count times the bucket count plus one. Adding buckets for resolution is therefore also a cardinality decision.

The OpenTelemetry metrics data model names a second cost. With cumulative temporality the sender must remember every series it has ever reported, "an 'up-front' memory cost proportional to cardinality"; delta temporality shifts that cost out of the process. An unbounded label therefore hurts twice: in the store, and in the memory of every process that emits the metric.

Budgets help. Set a ceiling on series per metric per service, alert when a deployment raises it, and review new labels the way you review a schema change.

2. Sampling at scale: head, tail and consistent#

Sampling means keeping only a share of the traces. Sensitive content introduced head sampling: the decision is made at the start of a trace, from the trace id alone. It is cheap, it needs no memory, and every service that sees the same trace id makes the same decision. OpenTelemetry's sampling page names the cost: with head sampling "it is not possible to make a sampling decision based on data in the entire trace". At the start of run-7f3a, nothing says the roof-data provider is about to time out.

Tail sampling decides after the trace is complete, so it can keep exactly what matters: traces with an error, slow traces, traces a quality rule flagged. The example combines the two, as OpenTelemetry's page describes: a head rate keeps a random share, and tail rules rescue the important traces that the head decision dropped.

cd "$TMPDIR/uw-observe-advanced" && /opt/homebrew/bin/python3 run_observe.py --show sampling
== sampling
head 25% + tail rules: kept 18 of 48 traces {'head': 10, 'tail:flagged': 1, 'tail:error': 6, 'tail:slow': 1} · stored 193 KB of 467 KB · run-7f3a kept: True (tail:error)
head 25% only: kept 10 of 48 traces {'head': 10} · stored 109 KB of 467 KB · run-7f3a kept: False (None)

The second line is the incident of the opening. With head sampling alone at 25 percent, the trace of run-7f3a was not kept: its trace id fell outside the kept quarter. With the tail rule "keep errors", it is kept, and so are the five other error traces of the day. The stored volume rises from 109 KB to 193 KB of 467 KB. That difference is the price of never losing an error trace. All sizes are from the synthetic day.

SettingTraces kept of 48Kept byStoredrun-7f3a kept
Head 25 percent and tail rules (errors, slow, flagged)18head 10, error 6, slow 1, flagged 1193 KB of 467 KByes, by the error rule
Head 25 percent only10head 10109 KB of 467 KBno

The decision function is short. The head decision reads the first eight hex characters of the trace id as a number, so any service holding the same trace id decides the same way:

def head_keep(trace_id: str, rate: float) -> bool:
    """Deterministic head decision: the first 8 hex characters of the trace id as a number in [0, 1)."""
    return int(trace_id[:8], 16) / 0x100000000 < rate

That property is called consistent sampling, and OpenTelemetry describes the same idea as consistent probability sampling: the decision is "made based on the trace ID and the desired percentage of traces to sample". Without it, the gateway keeps a trace that the worker drops, and the stored trace has a hole in the middle.

Tail sampling has an operational price, and the OpenTelemetry documentation is direct about it. Tail samplers "must be stateful systems that can accept and store a large amount of data", because every span of a trace must wait until the trace is complete. The Collector's tail sampling processor (stability "beta" for traces on the day of reading) waits 30 seconds by default (decision_wait) and requires that "all spans for a given trace MUST be received by the same collector instance". In practice that means two tiers: a first tier that routes spans by trace id with a load-balancing exporter, and a second tier that holds them and decides.

Production pitfall. A run of the underwriting agent is not over when the worker finishes. The underwriter's decision arrives hours later as a second trace, joined by a span link and run_id. No tail sampler waits hours. Decide the automated trace on its own, and make the rule for the decision trace separately: on this site, every decision trace is kept, because it is short and it is the evidence of who decided.
What this is not. Sampling decides which diagnostic traces are stored. It never applies to the audit ledger. A diagnostic record may be dropped; an audit event may not. If a sampler can drop it, it is not an audit event.
What sampling does to your metrics, and how to keep them honest

If dashboards are computed from stored traces, tail sampling biases them: the kept set over-represents errors and slow runs by design. Three practices keep numbers honest.

  • Compute rates from metrics, not from sampled traces. Metrics are counted before sampling, so an error rate from a counter is exact while the trace store holds a skewed subset.
  • Record the sampling weight. If a kept trace stands for four unsampled ones (head rate 25 percent), store that weight with it, so that counts computed from traces can be re-weighted. Tail-rescued traces carry weight one: they were kept because of what they are.
  • Keep the head decision deterministic across services, so that partial traces never enter the store and distort span counts.

A last case for agents: a run that is slow because it loops (case UW-502, nine calls to search_guidance) is caught by a slow rule only if its duration crosses the threshold. A tail rule on span count, or on a cost attribute, catches it earlier. Cost uses that signal.

3. Exemplars: from a spike on a chart to one trace#

A latency chart over the outage shows the 95th percentile climbing above seven seconds. The on-call engineer wants one of those slow runs, now. The metric cannot tell which runs those were, and it must not carry run_id as a label (section 1). An exemplar bridges the two. The OpenTelemetry metrics data model defines it: "An exemplar is a recorded value that associates OpenTelemetry context to a metric event within a Metric". In practice it is one sample measurement stored next to a histogram bucket, with the trace_id and span_id of the run that produced it, the value and the time.

So the histogram stays low-cardinality, and a few points on the chart become links: click the dot in the seven-second bucket at 14:07 and land in a trace like the one of run-7f3a. Exemplars are a sample, not an index: they answer "show me one" and never "show me all".

Diagram: An exemplar: from a dot on a latency chart to one trace

Read left to right. The histogram keeps counts per bucket and stays low-cardinality. Next to a bucket it keeps a few sample measurements, each with the trace id of the run that produced it. Clicking that sample opens the trace. No run_id label was needed on the metric.

  1. 1 · The metric
    1. Duration histogram: Counts of run durations per bucket, split only by bounded labels.
  2. 2 · The exemplar
    1. One sample measurement: A recorded value stored next to the bucket with the trace and span ids of the run that produced it.
  3. 3 · The trace
    1. The slow run: The full span tree of the run, where the time went and why.

4. Multi-tenancy: brokers who must not see each other#

On the simulated day every run belongs to one of six brokers, broker-01 to broker-06. In this example a broker is the tenant: the customer whose records must be kept apart from other customers' records. Multi-tenancy means one telemetry system serves several tenants, and each can see only its own.

There are three places to draw the line, from weakest to strongest:

Where the line isHowWhat can go wrong
In the queryEvery dashboard filters on a tenant attributeOne missing filter shows everyone's runs
In the storeThe store keeps tenants apart and every query names one; Grafana Mimir describes itself as multi-tenant storageA shared admin role still sees all
In separate systemsOne store per tenant or per sensitivity classCost and operations multiply

The tenant label is bounded (six values, and a contract per new one), so it may sit on metrics; section 1's table shows the series it adds. The tenant must also travel with the trace context so that every service tags its records, which is the same propagation problem as trace_id on Tracing.

The insurer has two kinds of reader inside, too. The platform team needs latency and errors for everyone; the underwriting team lead needs the content of runs for one line of business. Access by role is on Sensitive content; here the point is that the same split must hold in every backend that holds a copy.

5. Compliance: deletion, retention and residency#

Three demands arrive from outside engineering. This section describes the engineering conflict each creates; it does not state what any law requires. Ask counsel for that.

A deletion request. A person whose data appears in a submission asks for it to be deleted. In diagnostic logs and traces that is hard but possible when content was redacted at the source and identifiers are pseudonymous: delete or expire the few records that still hold it. In the audit ledger it collides with the design. The ledger is append-only and hash-chained; changing one entry breaks every hash after it, which is the point of the chain (Audit logs). The common design is to keep personal data out of the ledger in the first place: the audit event holds ids, rule ids, the decision and the role, and the personal data lives in a separate store that can delete it while the ledger keeps a reference that no longer resolves.

Retention. Each record type lives as long as its purpose. The example's illustrative policy (audit.RETENTION_DAYS) keeps traces 14 days, diagnostic logs 30 days, security logs 365 days and audit events about seven years. These numbers are invented for the example, not any regulator's rule. Written down per record type, they turn "how long do we keep logs?" from a debate into a configuration.

Residency. A rule may require that records about one region's customers stay in that region. Telemetry pipelines copy data by design: to a collector, to a backend, to a vendor, into a support ticket. Residency therefore has to be a routing rule in the pipeline (records tagged with a region go only to stores in that region), not a promise about one database.

In our insurance agent. The audit event for run-7f3a carries submission_id SUB-1006, the rule ids, policy_version 2026.2, the recommendation, and human_decision with decided_by: "underwriter". It carries no email address, no building address and no name. That is a design decision made on the audit page so that this page's deletion request has an answer.

6. Observing the observers#

Every pipeline on this site can fail silently. A collector under load drops spans. A sampler is misconfigured to zero. An exporter's credentials expire at night. Each failure looks the same from a dashboard: a quiet chart. Quiet is the most dangerous state of an observability system, because it reads as health.

Four checks cover most of it:

  1. Count in and count out. Every collector reports records received, dropped and exported. Alert on drops, not on volume.
  2. A heartbeat run. A synthetic run every few minutes (in this example it could be the clean case UW-002) must appear end to end: gateway log, trace, metric point. Alert when it does not.
  3. Watch the telemetry of the telemetry system from somewhere else. If the metrics store is down, its own alert cannot fire.
  4. Test the expensive rules. A deployment that raises series counts or turns off redaction is caught in CI, not by the bill or by an auditor.
The same idea for a coding agent. A coding-agent session such as CA-001 adds its own cardinality traps: a label per repository, per branch or per pull request on a metric multiplies series the same way run_id did. Keep those as span attributes, and count sessions by bounded categories (outcome, test result).

Diagram: Four ways observability fails at scale, and the guard for each

Each lane is one failure that appears only when the system grows, with the guard that prevents it. None of them is a bug in the code of earlier pages; all of them are decisions about the observability system itself.

  1. Cardinality
    1. run_id on a metric: An identifier used as a metric label creates a new series for every run.
    2. Ids on spans only: Identifiers stay on logs and spans; metrics carry bounded labels and a budget.
  2. Sampling
    1. The failing run dropped: A decision at the start cannot know that the run will fail.
    2. Tail rule keeps errors: Keep every error, slow or flagged trace after completion; head decisions agree across services.
  3. Tenants
    1. A broker sees all brokers: Separation only in dashboard filters fails with one missing filter.
    2. Separate in the store: The store keeps tenants apart and every query names one.
  4. Compliance
    1. Personal data in the ledger: An append-only, hash-chained ledger cannot delete one field without breaking every later hash.
    2. Ids and decisions only: Audit events hold identifiers, rule ids and decisions; personal data lives in a store that can delete it.

Checklist#

In one breath#

At scale, observability fails in its own ways: an unbounded label multiplies series until the store slows, a head sampler drops the one failing run while a tail sampler needs memory and routing to keep it, and tenants and the law decide who may see what and for how long. Keep identifiers on spans and logs, bounded categories on metrics, errors in every sample, personal data out of the audit ledger, and a heartbeat through the whole pipeline.

Where this leads#

You can now reason about the costs and failure modes that appear only when the system grows, and design against them. Every idea on this site has been shown one piece at a time. The last numbered page puts them back together on one run: case UW-006, from the gateway's first log line to the underwriter's decision, the audit event, the replay and the promotion to a golden case. That is the insurance lab.