All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Reference

Glossary

Every term this site uses, each with what it is, what it is not, and how it shows up in the insurance example.

Every term this site uses, in one place. Each card opens the same drill-down as the underlined phrases in the text: one line, why it matters here, what it is, what it is not, and how it shows up in the insurance example.

Three sentences hold the words apart. A log records an event. A metric summarizes measurements. A trace connects the steps of one run.

Access controlThe rules that decide which people and systems may read which telemetry records.Agent runThe span that covers one execution of the agent for one submission, holding every step it took and the run's identity and outcome.AlertA rule over telemetry that notifies a person when something needs action, with enough context to start.Arize PhoenixA program you run yourself that receives traces over OTLP, shows each run step by step, and keeps evaluation scores, datasets and experiments next to them.attempt_numberThe count of tries of the same step inside one run: 1 for the first call, 2 for the retry.Audit eventA record of something the business did, who did it and on what basis, kept complete, unaltered and for years.Audit ledgerThe store that holds audit events in order, with no update and no delete, each event chained to the one before by a hash.BudgetAn amount of cost units allowed for one run, one tenant or one period, with an alert when it is used up or burning too fast.Burn rateThe error rate in a time window divided by the error rate the objective allows; 1 spends exactly the budget over the period.CardinalityThe number of distinct values of a label; on a metric, every distinct combination of label values becomes its own stored time series.CheckpointA saved copy of a run's state at a step, from which the run could resume after a failure.ComplianceThe obligations that law, contracts and auditors place on records: what may be kept, for how long, where, and what must be deleted on request.Content captureThe decision to record the text of model inputs and outputs, retrieved documents and tool arguments and results on or next to a span.Context propagationSending the trace id and the current span id with every call or message, so that the receiver continues the same trace.Correlation idAn identifier copied onto every record that belongs together, so that records written by different services at different times can be joined by an exact match.Cost anomalyA run whose cost is far above the usual cost of runs of its kind, flagged by a rule on the cost records.Cost attributionSplitting a cost total by something a person can act on: run, step, tenant or feature.CounterA metric that only increases: each time something happens, it adds one (or an amount, such as tokens).DashboardA set of charts on one screen, built to answer a stated question at a glance.Data residencyA requirement that records about certain customers are stored and processed only in a named region.DatadogA vendor service that collects logs, metrics and traces through its agent, integrations and OpenTelemetry, and shows them in one place.Deterministic fixtureA test input together with fixed answers for every outside dependency, so that running it twice gives the same result.Diagnostic logLog records written to help engineers understand and fix the system; they may be sampled, redacted, aggregated and deleted.DriftA sustained change in a quality signal compared with its baseline, often after a change nobody connected to it.Elastic StackA set of open source tools in which Elasticsearch stores records as documents with mapped fields and searches them by field and by text.Error budgetThe number of bad events the objective allows over its period: one minus the objective, times the expected events.EventA named fact that happened at a moment. This site always says which kind: a span event (inside one span), a business event (a fact of the business process), or an audit event (a business event kept as proof).ExemplarA recorded measurement kept next to a metric, with the trace id and span id of the request that produced it, so that a point on a chart leads to one trace.GatewayThe HTTP service that receives a submission, assigns request_id and run_id, starts the trace and puts the run on the queue.GaugeA metric that goes up and down and reports a level at the moment it is read: queue depth, memory in use, runs in flight.GenAI semantic conventions · names for model, tool and agent spansThe OpenTelemetry names for generative AI telemetry: inference, tool, retrieval and agent spans, their metrics and events.Golden caseAn input with an agreed expected result and a record of who agreed, kept in a versioned dataset that changes are checked against.GrafanaOpen source software that queries, shows and alerts on metrics, logs and traces wherever they are stored, with separate projects for each store: Loki for logs, Tempo for traces, Mimir for metrics.HandoffA step in which one agent passes a task and its context to another agent, and responsibility moves with it.Hash chainA sequence of records in which each record's hash covers its own content and the previous record's hash, so a change anywhere breaks every later link.Head samplingA sampling decision made when a trace starts, from the trace id alone, so every service makes the same decision.HistogramA metric that records many measurements of one thing, usually a duration, as a count, a sum and a count per bucket.Human decisionThe decision a person with authority makes on the agent's recommendation, recorded as {decided_by, decision, override, reason}.Human feedbackAny judgement a person makes about the agent's output in the course of their work, recorded so it can be counted.Idempotency keyA key sent with a write so that a second write with the same key stores nothing new.Incident timelineThe ordered list of events and actions during an incident, each with a time and the record it comes from.IncidentAn unplanned event that degrades a service enough that people must respond now; it has a start, a response, an end and a written record.Instrumentation · the code that creates telemetryThe code that records spans, metrics and logs: written by you (code-based) or added around code you did not change (zero-code).JaegerA program that receives traces, stores them in a chosen storage backend, and shows them as waterfalls and service dependencies.LangSmithA vendor platform that records each traced step of an LLM application as a run, keeps the trees of runs as traces, and builds datasets and experiments from them.Log aggregationCollecting the log lines of every process on every machine into one store where they are parsed, indexed and searched together.Log categoryA field from a fixed list that says what kind of step wrote a log line: http, queue, agent, model, tool, retrieval, policy, checkpoint, audit or security in the example.Log indexA lookup structure a log store builds from chosen fields, so that a search on those fields does not read every record.LogOne record of one event at one moment: something happened here, now.MetricA number computed over many measurements, such as a count, a sum or a distribution, with a name and a few labels.Model routingSending each step of a run to a model tier chosen by a rule, instead of one model for everything.Model spanThe span of one model call, recording which model, tokens in and out, why the model stopped and how long the first token took.Multi-tenancyServing several customers (tenants) from one observability system while each can see only its own records.ObservabilityThe ability to answer questions about what a running system did from the records it left, without changing the system to ask.Online eval labelA named label that a deterministic rule attaches to a production run after it finishes, recording which rule produced it.OpenInferenceA published specification that says what to call the steps and fields of an AI application's trace, built on OpenTelemetry spans.OpenTelemetry Collector · receive, process, exportA separate program that receives telemetry, works on it, and exports it to one or more backends.OpenTelemetry · the vendor-neutral standard for telemetryAn open standard and toolkit for making, naming and moving telemetry: an API, SDKs, a Collector, the OTLP protocol and semantic conventions.OTLP · the OpenTelemetry ProtocolThe format and protocol telemetry uses to travel from a process to a Collector or a backend.Override rateOverrides divided by decisions over a window: how often the underwriter decided differently from what the agent recommended.Parent spanThe span whose step called a given step, named by the child's parent_span_id; a span with no parent is the root of its trace.Per-run costThe sum of the costs of all model steps of one run, each computed from its token counts and a rate.PercentileThe value at or below which a given share of the measurements falls: p50 is the median, p95 the value 95 percent stay within.Personal dataInformation that identifies a person or can be tied to one, such as a name, an email address or a phone number.Policy fixtureThe synthetic rule book UW-GUIDE that the agent's deterministic checks apply, in versions 2025.4 and 2026.2.PostmortemA document written after an incident with the timeline, impact, causes, what went well, what went badly, and actions with owners.PromotionThe steps that turn a failed or doubtful production run into a golden case: take the snapshot, redact, agree the expected result, record the role and policy version, bump the dataset version.Quality signalA measurement attached to a run that says something about the quality of its output, not about how fast or cheaply it ran.RecommendationThe agent's written output: approve, decline, request_info or escalate, with reasons, citations, missing items, stated confidence and the policy version.Recorded responseA response from a tool or a model, saved during the original run, that a replay returns instead of calling the dependency again.RedactionReplacing or removing sensitive values in a record before it is stored, so that the stored copy no longer holds them.ReplayExecuting a finished run again, in a sandbox, with its inputs and the outside world's answers taken from a snapshot, and comparing the result step by step.ReproducibilityThe property of a run, its snapshot, its code version and a replay harness together: running it again gives the same steps and the same result.request_idThe identifier of one HTTP request at the gateway or at the underwriter cockpit; it lives as long as that request.Resource · who produced the telemetryAttributes describing the process that emits telemetry, attached once to everything it sends.RetentionThe period after which a record is deleted, set per record type according to its purpose.Retrieval spanThe span of one retrieval step, recording the query, the documents that came back, their scores and, here, a trust mark.Review queueThe list of production runs a rule flagged for human review, each with a link to its trace.Roof-data providerThe synthetic outside provider that returns a roof assessment for a submission, called through the tool get_roof_assessment.Root causeThe condition that, if it had not been present, would have prevented the failure; often there is an outside trigger and an inside contributor.run_idThe identifier of one execution of the underwriting workflow for a submission, from the gateway to the underwriter's decision.RunbookA short page linked from an alert that says what it means, the first checks to make, and who to call.SamplingDeciding, for each trace, whether to keep it, so that only a share is stored.Security logRecords of who tried to do what and whether it was allowed, plus signs of abuse, read by the security team.Semantic conventions · the shared names for telemetryAgreed names (and sometimes values) for common facts in telemetry, such as the HTTP method or the number of input tokens.session_idThe identifier of a person's signed-in session in the underwriter cockpit; records name the role, never the person.SeverityThe ordered level of a log line: DEBUG, INFO, WARN, ERROR in the example.SignalOne kind of telemetry record: logs, metrics and traces, and events with a time.SLIA service level indicator: a defined measure of the service, here the share of runs judged good by one written rule.SLOA service level objective: the target value for an SLI over a stated period.SnapshotA record, taken from a finished run, of its input and of every answer the outside world gave it: tool responses, retrieved passages, policy version, model outputs, clock and seed.Span eventA named, timestamped annotation inside a span, for a moment that matters during the step.Span kindOne of SERVER, CLIENT, PRODUCER, CONSUMER or INTERNAL: the role a span plays in a call between processes.Span linkA pointer from a span to another span, usually in another trace, meaning this was caused by that.SpanThe record of one step of a run: its name, when it started, how long it took, its status, its parent and a few named facts.span_idThe identifier of one span, one step of a trace: 16 lowercase hex characters (an 8-byte value in W3C Trace Context; all zeros is invalid).SplunkSoftware that takes in data from many sources, stores it as events in indexes, and searches it with its Search Processing Language.stdoutThe output stream every process has by default; a terminal shows it, and a container platform can collect it.Structured loggingWriting each log line as named fields (typically one JSON object per line) instead of a free-text sentence.SubmissionThe record an intake step makes from a broker's quote request email: one JSON object with the building's facts.submission_idThe identifier of the business object, one broker's quote request, which lives for years and can have several runs.Tail samplingA sampling decision made after all or most spans of a trace have arrived, so it can keep traces because of what they contain: an error, a long duration, a flag.Tamper-evidenceThe property that a change to a stored record can be detected later, even if it could not be prevented.Telemetry pipelineThe chain of programs that moves records from the services that emit them to a store where people can ask questions: collect, process, store and index, query.TelemetryThe records a running system writes about what it is doing: logs, metrics, traces and events.TimestampThe moment a record says its event happened, written in a fixed format.Token usageThe number of tokens a model call read (the input) and wrote (the output), recorded on the model span.Tool spanThe span of one tool call by the agent, recording the tool's name, the outcome and the attempt number.TraceThe record of one run across services: a tree of spans that share one trace id, each with a start, a length, a status and a parent.trace_idThe identifier shared by every span of one trace: 32 lowercase hex characters (a 16-byte value in W3C Trace Context; all zeros is invalid).traceparentA header defined by W3C Trace Context with four dash-separated fields: version, trace id (32 hex), the caller's span id (16 hex), and flags.Tracing SDKThe code library linked into a service that creates spans, attaches attributes, and hands them to an exporter.Underwriter cockpitThe web application in which underwriters open the agent's recommendations and record their decisions.UnderwriterThe human at the insurer who reads the agent's recommendation in the underwriter cockpit and decides.Underwriting agentThe synthetic agent that reads a submission, gathers evidence with tools, applies the policy fixture's rules and writes a recommendation for a human underwriter.WaterfallA picture of one trace: each span is a horizontal bar from its start to its end, indented under its parent.

Each term is explained in full on one page and used everywhere else.