On the simulated production day the roof-data provider stops answering for part of the afternoon. Case UW-006 is one of the runs caught in it: the worker calls get_roof_assessment, waits, times out, waits again, times out a second time, and ends the run in a controlled state with a request_info recommendation for the underwriter. The model recommends; a person decides.
Every service did its job and wrote its records. The gateway wrote an access log line. The worker wrote structured logs and spans with the run_id run-7f3a. The cockpit wrote its own trace hours later, when the underwriter decided. Now the on-call engineer wants one thing: every record of run-7f3a, in time order, on one screen.
On a laptop that is one grep over one file. In production the gateway runs on several machines, the worker on a dozen, and each machine rotates its own log files. The records exist. They are scattered across disks that nobody can search together, and some of those machines were replaced since the morning. A record you cannot find in time is, for the incident, a record you do not have.
This page follows a record after it leaves the program: how it is collected, moved, stored, indexed and queried, and what five widely used backends do at each step. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
What this is not. This page does not say what to record or what to call the fields; that is Log lines, Four signals and OpenTelemetry. It also does not compare AI-specific trace viewers; those are on Tracing tools.
1. The path a record takes#
Start with the obvious design. Every process writes JSON lines to a local file, and a nightly job copies the files to a shared disk. It works for a week. Then three things happen. The incident is at 14:10, and the copy runs at 02:00. A machine that was scaled down at noon took its unsent file with it. And grep over a month of files takes longer than the incident.
Each of those failures names one stage of a telemetry pipeline: the path that moves records from the processes that emit them to a place where people can ask questions. A telemetry pipeline is like the mail system for records: something collects them at the door, something sorts them, something files them, and something finds them again.
Diagram: The path of one record, from the worker to a query
Read left to right. The worker writes a record about run-7f3a. A collector near the process picks it up so a lost machine loses little. One processing stage parses, redacts, samples and routes. The store keeps it and builds an index over some fields. A query finds it again. Every backend on this page fills these five stages in its own way.
- 1 · Emit
- The worker writes: The process writes a structured log line or hands a span to its tracing SDK.
- 2 · Collect
- Collector near the process: A small program on the host or beside the service that picks records up and sends them on promptly.
- 3 · Process
- Parse, redact, sample, route: Records are parsed into fields, sensitive content is masked, a share is kept, and each goes to its store.
- 4 · Store and index
- Store with an index: Records land in a store; an index over chosen fields makes some questions fast.
- 5 · Query
- A person or an alert asks: A query in the backend's language returns records, counts or a waterfall.
Five stages, each with one job:
- Emit. The process writes the record: a line to stdout or a file, or a span handed to a tracing SDK. In the example,
telemetry.py(Logger.log,Tracer.span) does this. - Collect. A small program near the process picks the records up and sends them on, so that a crash or a scale-down loses as little as possible. Splunk calls it a forwarder; Elastic ships Beats and Elastic Agent; Datadog ships the Datadog Agent; OpenTelemetry has the Collector.
- Process. Records are parsed, enriched (which machine, which version), filtered, redacted, sampled and routed. This can happen in the collector or in a separate stage.
- Store and index. Records land in a store. An index is a lookup structure built from some of the fields, so that a query for
run_id = run-7f3adoes not read every record of the day. What gets indexed is the most consequential choice on this page. - Query. A person or an alert asks a question in the backend's query language and gets records, counts or a waterfall back.
Before AI. A Spring service with Logback writing JSON to stdout, a log shipper on each host, a central log store, and a search screen: that is the same five stages, and most backend engineers have used them. An agent changes what is in the records (model calls, tokens, prompts) and how many of them a run produces. It does not change the path.
2. The explorer, before the products#
Try the question of the opening against the simulated day. The explorer below works like a log search screen: a filter by field, a free-text search, and the matching records in time order.
| Filter | What it answers | Try |
|---|---|---|
run_id | Every record of one run, from every service | run-7f3a |
| severity | Only the records at this level or above | ERROR |
| category | Only records of one kind of step | tool |
| service | Only the records of one service | agent-worker |
| free text | Records whose message contains the words | timeout |
Two searches show the point of the page. A free-text search for "timeout" finds every timeout of the afternoon, across many runs, mixed together. A field filter run_id = run-7f3a finds every record of one run, including the ones that do not contain the word, such as the gateway's access line and the checkpoint. The first search needs the message text to be indexed. The second needs run_id to be a field that the backend knows about. Which of those your backend builds when the record arrives decides which question is fast.
3. Three trade-offs that decide most of the cost#
Each backend below makes the same three choices differently. Read them first; the product sections are easier once you know what to look for.
Index-heavy or label-only. An index-heavy store builds a lookup over many fields, often over the words of the message too. Search is fast for almost any question, and the index costs storage and work on every record that arrives. A label-only store indexes a few labels per stream of records (for example service and severity), keeps the records themselves compressed in chunks, and scans those chunks at query time. Writing is cheap; a question about a field that is not a label reads more data. Loki, in its own words, "does not index the contents of the logs, but only indexes metadata about your logs as a set of labels".
Schema on write or schema on read. Schema on write means the fields and their types are decided when a record is stored: a query is fast and the type is guaranteed, and a new field may need a change before it can be searched well. Schema on read means the record is stored as it comes, and fields are extracted when a query runs: flexible, slower per query. Elasticsearch shows both on one page of its documentation: mapped fields decided at indexing, and runtime fields that "change the schema without reindexing", with a smaller index and slower search.
Self-hosted or hosted. Self-hosted means you run the collectors, the store and the screens: the data stays inside your network, and capacity, upgrades, retention and access control are your work. Hosted means a vendor runs the store: less operational work, and the records leave your network, so masking before export becomes a requirement rather than a preference.
| Choice | Fast and easy | What you pay | Where it bites in UW-006 |
|---|---|---|---|
| Index-heavy | Any field, any word, at query time | Storage and work on every record written | A busy afternoon writes many records; all of them are indexed |
| Label-only | Writing; questions by label | Scans when the question is not a label | run_id must not be a label (too many values); finding one run needs a filter over chunks |
| Schema on write | Typed, fast queries | Changes when a new field appears | attempt_number added later needs a mapping before it filters well |
| Schema on read | Any new field, at once | Slower queries | Fine for an incident, costly for a dashboard that runs every minute |
| Self-hosted | Data stays inside | Operations work | The broker's email never leaves the building |
| Hosted | Little operations work | Records leave the building | Masking before export is mandatory |
Production pitfall.run_idis the field the on-call engineer filters by most, and it is also the field with the most distinct values. In a label-only store it must not become a label; Grafana's Loki documentation advises against labels for unbounded values such as user or customer ids and points high-cardinality data to structured metadata instead. In an index-heavy store the same field is cheap to filter. The same record, two right answers. Advanced shows why with thecardinalitylab.
4. Standards apart from products#
One standard runs through all five products: OpenTelemetry. It defines the data model, the names and the wire protocol, OTLP, and it is not a backend. Every product below accepts OpenTelemetry data in some way, which is what makes the collect stage portable. How the standard works is on OpenTelemetry; this page states only which route each product documents.
The five products are grouped by what they are. Splunk, Elastic and Datadog are platforms that cover logs, metrics and traces. Grafana is a query and visualization layer with separate open source stores (Loki, Tempo, Mimir). Jaeger is a tracing backend only.
Splunk#
Definition. Splunk Enterprise is software that lets you, in its documentation's words, "search, analyze, and visualize the data gathered from the components of your IT infrastructure or business". It indexes a data stream and parses it into individual events.
The real problem it solves. Searching machine data from many sources in one place, at volume, with one query language.
Protocol or data model. Data is stored as events in an index, which holds the raw compressed data plus index files that point to it. A deployment has three tiers: forwarders take in data and pass it on, usually to an indexer, and "reside lightly" on the machines that generate the data; indexers store events and run searches; search heads send searches to the indexers and combine the results. Searches are written in SPL, the Search Processing Language.
Minimum example. Documented shape; not run here. A search for one run in a JSON-structured index looks like index=uw run_id="run-7f3a" | sort _time. The pages read here do not show this exact search; treat it as a shape to check against the SPL reference.
Strengths. One search language over many sources; indexer clusters that replicate data, "a group of indexers configured to replicate each others' data".
Limitations. Whether fields are extracted when a record is written or when it is searched: the documentation read here does not say (the field extraction pages did not load on the day of reading). Capacity planning for indexers is your work when self-hosted.
Integration. Forwarders on each host. The Splunk Distribution of the OpenTelemetry Collector exports to Splunk Observability Cloud; on the day of reading, its page said forwarding to the Splunk platform through the Collector is "not supported at the moment" and recommended the Universal Forwarder instead. Read that page before you design a route.
Security considerations. Whatever reaches an index can be searched by anyone with access to that index. Redact before the forwarder sends, and split indexes by sensitivity so that access can follow them.
When plain code is better. One service, one team, a few days of records: a JSON file and grep answer the opening question without a platform.
Version and date verified. Splunk Enterprise documentation version 10.6, read on 2026-11-17.
Primary sources. Splunk Enterprise overview · Deployments · Indexes and indexers.
Elastic Stack (ELK)#
Definition. The Elastic Stack is, in its documentation's words, "our suite of open-source tools, including Elasticsearch, Kibana, Beats, and Logstash". ELK is the older name, from Elasticsearch, Logstash and Kibana.
The real problem it solves. Storing documents (a log record is one) with a mapping of their fields, and searching them fast by field and by text.
Protocol or data model. Each record is a document in an index. A mapping says what type each field has. Dynamic mapping "automatically detects the data types of fields" as documents arrive; explicit mapping, which the documentation calls "recommended for production use cases", declares them. Runtime fields change the schema without reindexing, with a smaller index and slower search.
Minimum example. Documented shape; not run here. An explicit mapping for the identifiers of this site would declare run_id, trace_id and submission_id as exact-match keyword fields, attempt_number as an integer, and message as text for word search.
Strengths. Mapped fields give fast, typed filters; text fields give word search; both in one store.
Limitations. Dynamic mapping can add a field for every new key. The documentation warns that too many fields can cause "out of memory errors" and offers mapping limit settings. Agent spans with free-form tool arguments are exactly the records that create new keys. Map them deliberately, or store arguments as one text field.
Integration. Beats and Logstash in the classic stack. Elastic also publishes EDOT, its distributions of OpenTelemetry; the documentation says the OpenTelemetry Collector capability is built into Elastic Agent from version 9.5, and lists generally available SDKs for .NET, Java, Node.js, PHP, Python, Android and iOS.
Security considerations. Field-level decisions happen at mapping time: decide which fields are searchable at all, and keep prompt text out of indexed fields unless someone needs to search it.
When plain code is better. When the only question is "every record of one run", a store with a single exact-match field answers it; a full search engine pays when the questions are many and not known in advance.
Version and date verified. Documentation read on 2026-11-17; the getting-started page states no version; EDOT refers to Elastic Agent 9.5.
Primary sources. Get started · Mapping · Elastic OpenTelemetry.
Datadog#
Definition. Datadog is, in its documentation's words, "an observability platform that supports every phase of software development on any stack". It is used as a service: you select "the appropriate Datadog site for your region". The pages read here do not describe running Datadog on your own infrastructure.
The real problem it solves. Logs, metrics, traces and many other products in one hosted place, with an agent and integrations that collect from common technologies.
Protocol or data model. The Datadog Agent sends metrics and events from your hosts; integrations collect from other technologies. For AI applications, the product formerly named LLM Observability is now called Agent Observability: each request is a trace, and its spans represent each step of a workflow or each choice an agent made.
Production pitfall. The rename is recent. On the day of reading, the Datadog page said LLM Observability is "now called Agent Observability", and the instrumentation page carried the new name in its title, while the address still contains llm_observability. Search for both names when you read about it.
Minimum example. Documented shape; not run here. The documented OpenTelemetry routes are: an OpenTelemetry Collector deployed with the Datadog Agent; an OpenTelemetry Collector (upstream, or Datadog's own DDOT build) that exports over OTLP; and direct OTLP ingestion for serverless or constrained environments.
Strengths. One hosted place for many signals; several documented OpenTelemetry routes, so instrumentation can stay vendor-neutral.
Limitations. Records leave your network. The pages read here describe region selection, not self-hosting.
Integration. Datadog Agent, integrations, OpenTelemetry as above; for agent traces, a Python SDK that instruments common model and agent libraries.
Security considerations. The Agent Observability page says the product can "automatically scan and redact any sensitive data". Treat that as a second line of defence. Redaction before export is the first, because a record that never left cannot leak from the vendor's side.
When plain code is better. When the records must not leave the building, a hosted platform is not an option, however capable.
Version and date verified. Read on 2026-11-17. Hosted service; no version number on the pages read.
Primary sources. Getting started · OpenTelemetry · Agent Observability.
Grafana (with Loki, Tempo and Mimir)#
Definition. Grafana lets you, in its documentation's words, "query, visualize, alert on, and explore your metrics, logs, and traces wherever they are stored". It is the screen and the alerting layer. The stores are separate open source projects: Loki for logs, Tempo for traces, Mimir for metrics. Grafana Cloud is a managed service from the same company.
The real problem it solves. One place to look at signals that live in different stores, and stores designed to keep large volumes cheaply.
Protocol or data model. Loki indexes only labels and stores log lines compressed in chunks in an object store; its query language is LogQL. Tempo is "a high-scale distributed tracing backend" that "only requires an object storage to operate"; its query language is TraceQL; it works with Jaeger, Zipkin and OpenTelemetry protocols. Mimir is "horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus and OpenTelemetry metrics".
Minimum example. Documented shape; not run here. In Loki, the labels select the stream and a filter reads the lines: a selector on service followed by a JSON parse and a filter on run_id. The run_id stays inside the line, not in a label.
Strengths. Label-only logs and object-storage traces keep write cost low; one screen across stores; Mimir is multi-tenant by design.
Limitations. A question about a field that is not a label reads more chunks. Loki's documentation asks for "10 - 15 labels at a maximum", bounded values, and structured metadata for high-cardinality data.
Integration. OpenTelemetry Collector into each store; Grafana queries them all, and other data sources too.
Security considerations. Multi-tenancy in Mimir separates tenants at the store; give the cockpit team and the platform team different tenants or different data sources if they must not see each other's records.
When plain code is better. For one service and a single question, the same as for the others: a file and a filter.
Version and date verified. Read on 2026-11-17; pages under latest state no version. The Loki overview notes that one deployment mode will be removed in Loki 4.0, which places the documented release before 4.0.
Primary sources. Grafana introduction · Loki overview · Loki labels · Tempo · Mimir.
Jaeger#
Definition. Jaeger is "a distributed tracing platform", open source since 2016 and a graduated project of the Cloud Native Computing Foundation. It stores and shows traces only, not logs or metrics.
The real problem it solves. Seeing where the time went in one request across services, and how services depend on each other.
Protocol or data model. Jaeger "receives trace data in the standard OpenTelemetry Protocol (OTLP)" and also accepts Zipkin formats over HTTP. Version 2, released in November 2024, is "built on the OpenTelemetry Collector framework". Its documentation names several storage backends, among them Cassandra, ClickHouse, Elasticsearch, OpenSearch, Badger and an in-memory store.
Minimum example. Documented shape; not run here. Point an OTLP exporter at Jaeger and search by service agent-worker and the tag run_id=run-7f3a; the waterfall shows the two failed get_roof_assessment spans with their attempt_number.
Strengths. One job, done with the standard protocol; sampling documented as head-based with central remote configuration (static or adaptive) and tail-based.
Limitations. Traces only. The UW-006 access log line and the audit event live elsewhere; you join them by trace_id and run_id.
Integration. Any OpenTelemetry SDK or Collector that exports OTLP.
Security considerations. Span attributes and events are stored as sent. Content in a model span is in Jaeger's store unless it was removed before export.
When plain code is better. A trace you read once in a test is clearer as a printed waterfall; the example prints one from telemetry.waterfall.
Version and date verified. Documentation version 2.23, read on 2026-11-17.
Primary sources. Jaeger documentation · Features.
5. Side by side#
The filter above the table narrows its rows; without JavaScript the full table is the content.
| Backend | Kind | Signals it stores | Indexing, per the docs read here | Hosting | OpenTelemetry route documented |
|---|---|---|---|---|---|
| Splunk | Platform | Events from any source | Index of events plus index files | Self-hosted Enterprise; a separate cloud observability product | Distribution of the Collector to its observability cloud; Universal Forwarder for the platform |
| Elastic Stack | Platform | Documents (logs, metrics, traces) | Mapped fields, dynamic or explicit; runtime fields at query time | The documentation read here does not settle it on the pages listed | EDOT; Collector built into Elastic Agent from 9.5 |
| Datadog | Hosted platform | Logs, metrics, traces, agent traces | The documentation read here does not say | Hosted, per region | Collector with the Agent, Collector over OTLP, direct OTLP |
| Grafana with Loki, Tempo, Mimir | Query layer plus stores | Logs (Loki), traces (Tempo), metrics (Mimir) | Loki indexes labels only; chunks in object storage | Self-hosted open source, or Grafana Cloud | Collector into each store |
| Jaeger | Tracing backend | Traces | Depends on the storage backend chosen | Self-hosted | OTLP natively; built on the Collector |
6. Choosing for the underwriting agent#
The table does not choose for you, and this page will not rank. It can turn the opening into questions a team can answer for itself.
- Where may the broker's email go? If nowhere outside the network, the hosted options need masking before export, or drop out.
- Which question must be fast during an incident? For "every record of
run-7f3a",run_idmust be a cheap filter: a mapped exact-match field, or structured metadata, never a label. - Which question must be cheap every minute? Dashboards and alerts run constantly; they belong on metrics, not on log searches.
- Who else must not see what? Tenants, indexes or data sources, decided before the first record arrives.
- Where do audit events live? In none of these as the only copy. A diagnostic record may be dropped; an audit event may not. Audit logs keeps them in their own append-only ledger.
Diagram: Three choices every backend makes, and the question each one makes cheap
Each lane is one choice with two sides. The left box of a lane makes one kind of question cheap; the right box makes another cheap. No backend is on the cheap side of every lane. Pick by the questions you must answer fast and by where your records may go.
- Indexing
- Index-heavy: A lookup over many fields makes almost any search fast and costs work on every record written.
- Label-only: Only a few labels are indexed; the lines are compressed in chunks and scanned when asked.
- Schema
- On write: Fields and types are fixed at storage time: fast typed filters, changes for new fields.
- On read: Records are kept as they came and fields are pulled out at query time: flexible and slower.
- Hosting
- Self-hosted: The data stays inside your network; capacity, upgrades and access are your work.
- Hosted: Less operations work; records leave your network, so masking before export is required.
The same idea for a coding agent. The session CA-001 (read the repository instructions, edit, run tests, fail, edit, pass, open a pull request, CI) produces the same kinds of records, and the same path applies: the CI system's logs and the agent's spans meet in whichever backend the team already queries. How the pipeline itself is built is on diff-to-deploy.sammuthu.com.
Why a collector tier sits between the process and the store
Exporting straight from each process to the backend works and is the shortest path. A separate collector tier earns its place for four reasons.
- Buffering. A store that is slow or down for a minute should not slow the agent. The collector queues records and retries; the process hands off and moves on.
- One place for policy. Redaction, sampling, enrichment and routing in one configuration instead of in every service's code. A new rule is one deployment, not twelve.
- Fan-out. The same spans can go to the general tracing backend and to an AI trace viewer at once, without the services knowing.
- Backend changes. Switching stores changes the collector's exporters, not the services.
The cost is one more component to run, size and monitor. A collector that drops records under load drops them silently unless its own metrics are watched: observe the observers (Advanced).
Where the volume comes from in an agent workload
A conventional HTTP request produces one access line and a handful of spans. One run of the underwriting agent produces a gateway span, queue spans, a worker span, model spans, tool spans with retries, a retrieval span, a rule-check span, a checkpoint span, several diagnostic log lines and audit events. A model span may carry the full prompt and output as content.
Three levers move most of the volume, in this order: whether content is captured at all (GenAI telemetry), what share of traces is kept (Sensitive content and Advanced), and which fields are indexed. The third is the one this page owns, and it is the one teams set last, after the bill arrives. Set it first.
Checklist#
- You can name the five stages for your own system, and which component does each.
- A record that is written survives the loss of the machine that wrote it (a collector or shipper sends it promptly).
run_id,trace_idandsubmission_idare cheap exact-match filters in your store, and none is a label in a label-only store.- Redaction happens before the record leaves the process or the collector, not only in the backend.
- Dashboards and alerts read metrics, not repeated log searches.
- Each product's OpenTelemetry route was read in its current documentation, not assumed.
- Audit events have their own store; the backend holds diagnostic copies at most.
- Retention is set per record type and written down.
In one breath#
A record is only useful if it can be found in time, so it travels a path: emitted, collected near the process, processed once, stored with an index, and queried. The five backends differ mainly in what they index when a record is written, whether they run on your machines or a vendor's, and how they take in OpenTelemetry data. Choose by the questions you must answer fast and by where your data may go, and never let a diagnostic store be the only copy of an audit event.
Where this leads#
You can now follow a record from the process to a query and reason about what makes a question fast or slow. A screen full of searchable records does not tell anyone when to act. The next page turns records into a measured promise: an indicator, an objective, an error budget and an alert a person can act on. That is SLOs and alerts.