All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Reference · 15

The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith

Two standards and two products, each read in its own documentation and described by what it defines, what it stores and what stays your job.

Two layers
A standard names the fields of a record. A product stores and shows the records. They are different kinds of thing.
Four names, one template
OpenTelemetry, OpenInference, Phoenix and LangSmith, each read in its own documentation on 2026-11-17.
What stays yours
Which steps you record, what you leave out, and where the audit events live. No tool decides that for you.

The underwriting agent has run in production for two months. Every run writes its own records: structured logs, one trace per run, and audit events in a hash-chained ledger. The model recommends; a person decides. The code that writes all of this lives in one folder of standard-library Python, examples/uw-observe/, and its trace format is its own: a list of spans with fields such as kind, span_kind, start_ms and attributes.

Then the on-call engineer opens the trace of case UW-006, the run in which the roof-data provider timed out twice. The trace is correct. It is also a JSON file on a disk, and the only way to read it is a terminal. The platform engineer asks for three things in one week: a screen on which the underwriting team lead can click through a run, a way to send the same spans to the company's existing tracing backend, and a promise that the broker's email inside a model span never leaves the building.

Someone writes in the team channel: "We should adopt a tracing framework." Within an hour four names are on the table: OpenTelemetry, OpenInference, Phoenix and LangSmith. They look like four answers to one question. They are not. Two of them are documents that say what a record is called. Two of them are programs that store records and show them. The question that sorts them is this: does this name tell me how to write a span, or where the span goes and who can read it?

A standard decides the names. A product decides where the records live. Your own code still decides what is worth recording and what must never be recorded.

This is a reference page. It is the only page of this site that makes statements about Phoenix, LangSmith and OpenInference. OpenTelemetry has its own page, OpenTelemetry; here it gets one section in the same template as the others, so that you can read across. Where the records go after they leave the process (Splunk, Elastic, Datadog, Grafana, Jaeger) is the subject of Backends. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.

1. Two layers, and why the difference matters#

Start with one span from the example. A span is the record of one step: when it started, how long it took, whether it failed, and a few named facts about it. In UW-006 the second call to the roof-data provider is one span. In the example's own format it carries the trace id, its own span id, its parent, the step kind tool, the name get_roof_assessment, a status of error, and attributes such as attempt_number: 2.

Now imagine you want any tool other than your own code to read that span. Two separate agreements are needed.

The first agreement is about names. Your code calls the step kind tool. Another program has to know that tool means "a function the agent called", and that the attribute holding the tool's name is the one to show in a list. If every team picks its own names, every viewer needs one adapter per team. A naming agreement is called a semantic convention: a published list of attribute names and their meaning. OpenTelemetry publishes semantic conventions; OpenInference publishes a second set aimed at AI applications.

The second agreement is about where the span goes. Something has to receive spans over the network, store them, index them and draw a waterfall. That is a program you run or a service you sign up for. Phoenix and LangSmith are programs of that kind for AI applications. So are the general backends of the Backends page.

Diagram: Two layers: the names on a span, and the place it goes

Read left to right. Your code makes the span. A convention decides what its fields are called. The wire format carries it. A product stores it and shows it. A standard sits in the middle two lanes; a product sits in the last one. You can swap the product without touching the services, as long as the names and the wire stay the same.

  1. 1 · Your code
    1. A span of UW-006: The record of one step, written by the worker while the run happens.
  2. 2 · Names (a standard)
    1. OpenTelemetry conventions: A published list of attribute names and meanings, maintained by the OpenTelemetry project.
    2. OpenInference: A second published list for AI steps, defined on top of OpenTelemetry spans.
  3. 3 · Wire
    1. OTLP export: The protocol that carries spans from the process to whatever receives them.
  4. 4 · Place (a product)
    1. Phoenix: A program you run that stores AI traces and shows them step by step.
    2. LangSmith: A vendor service that stores traces as trees of runs and builds datasets from them.
    3. A general backend: Any tracing backend that accepts OTLP: the one the on-call engineer already uses.

The diagram shows why the two layers are kept apart on this page. You can change the product and keep the names. You can also keep the product and change the names, if the product reads both conventions. Most of the questions in the team channel become clear once each name is put in its layer.

Plain English. A standard is a dictionary: it says what a word means. A product is a library building: it keeps the books and lets people read them. You need both, and buying a building does not write the dictionary.
Before AI. The same split exists in a Java service. SLF4J is an interface for logging calls; Logback or Log4j writes the lines; a log platform stores them. The JDBC interface is not a database. Nobody confuses the two there, because the names are old. In AI tracing the names are new, and several of them cover both layers at once in their marketing.
What this is not. Grading whether a run was good is not part of tracing. A tool on this page may store a score next to a trace, but the score, the grader and the release rule belong to evaluation: see the eval harness on eval.sammuthu.com.

2. How the four were read#

Each name below was read on its own documentation on 2026-11-17, and every claim is logged with its address in docs/facts/frameworks.md in the repository. Three rules hold for the whole page.

Every section follows the same order, so that a row can be compared with the same row of another section: definition, the real problem it solves, protocol or data model, a minimum example, strengths, limitations, integration, security considerations, when plain code is better, version and date verified, primary sources.

The four are grouped by kind. Standards and specifications come first: OpenTelemetry and OpenInference. They define names and formats and run nothing on their own. Products come second: Phoenix and LangSmith. They are programs that receive, store and show traces.

3. Standards and specifications#

OpenTelemetry (standard)#

Definition. OpenTelemetry describes itself as an "observability framework and toolkit", open source and vendor- and tool-agnostic. It is a Cloud Native Computing Foundation project, formed from the merger of two earlier projects, OpenTracing and OpenCensus. It has three parts that matter here: a data model for logs, metrics and traces; a set of semantic conventions (the names); and the code that produces and moves telemetry (API, SDK, Collector, and the wire protocol OTLP).

The real problem it solves. Without a shared model, every backend ships its own client library, and moving from one backend to another means rewriting the instrumentation in every service. With it, the services emit one format and the destination becomes a configuration choice.

Protocol or data model. Spans with a trace id, a span id, a parent, a kind, a status, attributes and events, sent over OTLP. The full treatment, including the GenAI conventions, is on OpenTelemetry.

Minimum example. See OpenTelemetry for the SDK and Collector shapes. This page does not repeat them.

Strengths. A neutral format that the other three names on this page build on or accept. OpenInference is defined on top of it; Phoenix and LangSmith both accept OTLP.

Limitations. OpenTelemetry itself says it "is not an observability backend": storage, search and screens are left to other tools. Its GenAI conventions are newer than its HTTP and database conventions.

Production pitfall. On the day this page was read, the GenAI conventions page on opentelemetry.io said: "GenAI semantic conventions have moved to the OpenTelemetry GenAI semantic conventions repository", and was no longer maintained there. The new home is the GitHub repository open-telemetry/semantic-conventions-genai. A bookmark to the old page will show you stale names. Read the conventions where they live now; OpenTelemetry says which names this site uses.

Integration. Everything else on this page sits on it. OpenInference adds attributes to OpenTelemetry spans; Phoenix and LangSmith receive OpenTelemetry spans over OTLP.

Security considerations. The standard defines where content may go (attributes, events); what you put there is your decision. Sensitive content covers redaction before export.

When plain code is better. A single process that writes a few hundred runs a day to one file, read by one team, does not need a collector. The example on this site writes its own format on purpose, so that every field can be read in one screen.

Version and date verified. Read on 2026-11-17. The GenAI conventions repository was created on 2026-05-05; its README section "Schema URL" read "TODO" that day.

Primary sources. What is OpenTelemetry · GenAI semantic conventions repository.

OpenInference (specification)#

Definition. OpenInference is, in its own words, "a semantic convention specification for AI application observability, built on OpenTelemetry". It is a document plus a set of instrumentation packages. It "adds no new signal or wire format": every OpenInference trace "is a valid OTLP trace".

The real problem it solves. Its specification lists the gaps it fills in the general conventions: structured inputs and outputs (message arrays, not one string), token counts as first-class numbers, agentic control flow (agents, tools, retrieval, sub-agents), privacy (prompts must be maskable before export), nondeterminism (traces must carry enough context to explain a run), and quality feedback (scores and labels that stay attached to the step they judge).

Protocol or data model. The central attribute is openinference.span.kind. The specification makes it required on every OpenInference span and says it is distinct from OpenTelemetry's own span kind (server, client, internal and the others). Its values are LLM, EMBEDDING, CHAIN, RETRIEVER, RERANKER, TOOL, AGENT, GUARDRAIL, EVALUATOR, PROMPT and DECISION. Other attributes use dotted names, and lists are flattened with zero-based indices, such as llm.input_messages.0.message.role.

Put the UW-006 spans next to these names and most of them line up. The table maps the example's step kinds to OpenInference kinds and names. The mapping is a design proposed here, not something the specification says about this example.

Step in UW-006Example kindOpenInference openinference.span.kindAttributes that carry its facts
The whole run in the workeragentAGENTsession.id could carry the run's grouping key
plan_steps, write_recommendationmodelLLMllm.model_name, llm.token_count.prompt, llm.token_count.completion, llm.token_count.total
get_roof_assessment attempts 1 and 2toolTOOLtool.name, tool.parameters; attempt_number stays a custom attribute
search_guidanceretrievalRETRIEVERretrieval.documents
check_eligibilitypolicy_checkCHAIN or TOOL (a choice)input.value, output.value
checkpointcheckpointnone fits exactly; CHAINcustom attributes

Two rows have no clean match, and that is useful information. A deterministic rule check and a saved checkpoint are not AI-specific steps, so an AI-specific convention does not name them. Choose a kind, write the choice down, and keep the example's own attribute (attempt_number, policy_version) next to it.

Minimum example. Documented shape; not run here. The specification shows a root span in a logical JSON form; for our tool span it would read:

{
  "name": "get_roof_assessment",
  "span_kind": "SPAN_KIND_CLIENT",
  "status_code": "ERROR",
  "attributes": {
    "openinference.span.kind": "TOOL",
    "tool.name": "get_roof_assessment",
    "tool.parameters": "{\"submission_id\": \"SUB-1006\"}",
    "attempt_number": 2,
    "run_id": "run-7f3a"
  }
}

Strengths. One attribute that tells any reader what role a span plays; names for token counts and messages; masking switches defined in the specification itself (below). Packages exist for Python, JavaScript, Java and Go, and its repository says traces can go to "any backend that accepts OTLP".

Limitations. It is one convention among two for AI spans; OpenTelemetry's GenAI conventions cover overlapping ground with different names. A backend shows OpenInference kinds well only if it reads them. Steps that are not AI-specific (rule checks, checkpoints, queue hops) fall back to general kinds.

Integration. Instrumentation packages wrap common model clients and agent libraries and emit OpenInference spans. Phoenix reads them natively. LangSmith's documentation lists a mapping for input.value, output.value, openinference.span.kind and llm.model_name.

Security considerations. The specification defines environment variables that hide content before export, all off by default: OPENINFERENCE_HIDE_INPUTS, OPENINFERENCE_HIDE_OUTPUTS, OPENINFERENCE_HIDE_INPUT_MESSAGES, OPENINFERENCE_HIDE_OUTPUT_MESSAGES, OPENINFERENCE_HIDE_RETRIEVAL_DOCUMENTS and several more. Off by default means a fresh setup exports the broker's email inside a model span. Turn on what you need before the first export, not after.

When plain code is better. If your spans never leave your own code and your own viewer, a naming convention buys little. The moment a second program reads them, it pays.

Version and date verified. Specification files on the main branch, read through the GitHub API on 2026-11-17. Repository license: Apache-2.0.

Primary sources. Specification README · Span kinds · Trace configuration.

4. Products#

Arize Phoenix (product)#

Definition. Phoenix is a program for understanding and improving AI applications. Its documentation lists tracing, evaluation, datasets, experiments and prompt management. It is "built on OpenTelemetry and powered by OpenInference instrumentation". Its README says "Phoenix is open-source and self-hosted"; its LICENSE file is the Elastic License 2.0. The vendor's managed platform is a separate product.

The real problem it solves. The screen the team lead asked for. A trace stored as JSON is correct and unreadable; Phoenix receives the spans and shows a run step by step, with AI-aware views of model calls, tools and retrieval.

Protocol or data model. It accepts traces over OTLP and reads the OpenInference attributes above. Evaluation results are scores attached to traces and spans, produced by model-based evaluators, code checks or human labels.

Minimum example. Documented shape; not run here. The tracing quickstart registers an OpenTelemetry tracer provider that sends to Phoenix:

from phoenix.otel import register

register(
    project_name="tracing-quickstart",
    protocol="http/protobuf",
    auto_instrument=True,
)

The README gives pip install arize-phoenix for a local install, container images on Docker Hub and a Helm chart. The quickstart page read here does not give the default port or the start command; it links to a separate start page.

Strengths. Runs on your own machine or cluster, so the traces stay where you put them. Speaks the two standards above. Keeps evaluation scores next to the trace they belong to.

Limitations. It is a trace store and workbench, not an audit ledger: the documentation read here does not describe append-only or tamper-evident storage, so human_decision_recorded events do not belong only there. Retention and scaling are yours when you host it.

Integration. Any OTLP exporter can send to it. The README lists instrumentation for common model providers and agent libraries through OpenInference packages.

Security considerations. Self-hosting keeps the data inside your network, which also makes access control, encryption at rest and deletion your job. Masking happens before export (the OpenInference variables), not in the viewer.

When plain code is better. One engineer reading a few traces a week can read JSON. A screen pays when people who do not read JSON need to look at runs.

Version and date verified. Latest release tag on 2026-11-17: arize-phoenix-v21.2.0.

Primary sources. Phoenix documentation · Tracing quickstart · Repository.

LangSmith (product)#

Definition. LangSmith is a platform whose observability part gives, in its documentation's words, visibility "from individual traces to production-wide performance metrics". In its vocabulary a run is one traced step and a trace is the tree of runs. Note the word clash: on this site a run is one execution of the whole workflow (run_id); a LangSmith run is closer to what this site calls a span.

The real problem it solves. Instrumenting an application with little code, storing the traces centrally, and using them to debug failures, monitor quality and build evaluation datasets.

Protocol or data model. Its own SDK, and also OTLP: its documentation gives an OTLP base URL ending in /otel, with the exporter appending /v1/traces. It maps incoming gen_ai.* attributes and OpenInference attributes onto its runs, and has its own run types: llm, chain, tool, retriever, embedding, prompt, parser.

Minimum example. Documented shape; not run here. With the SDK, each decorated function becomes a run, and nested calls become child runs:

from langsmith import traceable

@traceable(run_type="tool")
def get_roof_assessment(submission_id):
    ...  # call the roof-data provider

@traceable
def underwrite(submission_id):
    return get_roof_assessment(submission_id)

Tracing is switched on with the environment variable LANGSMITH_TRACING=true and authenticated with LANGSMITH_API_KEY.

Common confusion. The OpenTelemetry route needs headers that depend on the kind of workspace. The documentation read here describes agent-based workspaces, which need Langsmith-Agent-Id and Langsmith-Agent-Environment together, and project-based workspaces, which use Langsmith-Project. Sending the wrong combination returns HTTP 400. Read the current page before wiring a collector to it.

Strengths. Low effort to start from Python or JavaScript; accepts both AI conventions over OTLP; traces feed datasets and experiments in the same product.

Limitations. The hosted service holds your traces unless you run the self-hosted edition, which its documentation calls "an add-on to the Enterprise plan" running in your own Kubernetes cluster. The observability page read here does not detail retention; it points to the usage and billing pages.

Integration. The SDK, framework integrations, or any OTLP exporter or collector with the right headers.

Security considerations. Content leaves your network on the hosted route. Decide what is masked before export, and check the vendor's retention terms against your own retention rules for prompts that hold personal data.

When plain code is better. When the traces must not leave the building and self-hosting is out of reach, or when your spans already go to a general backend that the on-call engineer uses for everything else.

Version and date verified. Pages read on 2026-11-17; the OpenTelemetry page refers to Python SDK langsmith>=0.14.9 for one header behavior.

Primary sources. Observability · Annotate code · Trace with OpenTelemetry · Self-hosted.

5. Side by side#

The table puts the four on the same rows. The filter above it narrows the rows by kind or by question; without JavaScript the full table is the content.

NameKindWhat it defines or runsWire formatWhere the data livesContent masking, per the docs read hereLicense or hosting, per the docs read here
OpenTelemetryStandardData model, semantic conventions, API, SDK, CollectorOTLPWherever you export it; it is not a backendYour code and the Collector decideOpen source, CNCF project
OpenInferenceSpecificationAI span kinds and attribute names on top of OpenTelemetryOTLP (no new format)Wherever you export itOPENINFERENCE_HIDE_* variables, off by defaultApache-2.0
Arize PhoenixProductTrace store and viewer, evaluation, datasets, experiments, promptsOTLPYour machine or clusterBefore export, through the instrumentationSelf-hosted; Elastic License 2.0
LangSmithProductTrace store and viewer, datasets, experimentsIts SDK, or OTLP at /otelThe vendor's service, or self-hosted as an Enterprise add-onThe documentation read here does not say on the pages listedHosted; self-hosted add-on

Read the table by column, not by row. The "Kind" column says which questions a name can answer at all. The "Where the data lives" column is the one your security team will ask about first.

6. When plain code is better#

The example on this site uses none of the four, on purpose. Its tracer is about sixty lines in telemetry.py (class Tracer), and every field it writes can be seen on one screen. That is the right choice while the goal is to understand what a span is. It stops being the right choice at three points.

  1. A second reader appears. The moment another program has to read your spans, use a published convention so that you do not maintain the translation.
  2. People who do not read JSON need the runs. Then a viewer pays for itself, whichever one you choose.
  3. Volume and retention matter. A file per run on one disk does not survive a month of production; a store with retention rules does.

What never moves into a tool is the audit ledger. A trace store may sample, truncate or expire spans, and that is acceptable for diagnostic records. The record that the underwriter declined a recommendation must stay complete and unaltered. A diagnostic record may be dropped; an audit event may not. Keep the audit events where Audit logs puts them, and let the trace link to them by run_id.

Diagram: When a tool starts to pay, and what never moves into it

Read left to right. Plain code is enough while one team reads its own spans. Each new need on the middle lane is a reason to adopt a convention or a product. The right lane stays outside every trace tool: the audit ledger, which must stay complete.

  1. 1 · Plain code is enough
    1. One reader: One team writes the spans and reads them in its own format.
  2. 2 · A tool pays when
    1. A second program reads: Another tool must understand the spans, so a published naming list saves a translation.
    2. People need a screen: Readers who do not read JSON need to click through runs.
    3. Volume and retention: A file per run on one disk does not survive a month of production.
  3. 3 · Never moves into a trace tool
    1. The audit ledger: Records the business must prove later stay in their own store; traces link to them by run_id.
The same idea for a coding agent. Session CA-001 (read the repository instructions, edit, run the tests, fail, edit, pass, open a pull request) is a trace too, and the same split holds: a convention names its tool and model steps, a product shows them. How the coding agent's harness emits that telemetry is covered on harness.sammuthu.com.
Two AI conventions on one span: what to do when they overlap

OpenTelemetry's GenAI conventions and OpenInference both name model calls, token counts and tools, with different attribute names. Three designs are in use, each with a cost.

Emit one, translate at the edge. The service emits one convention; a Collector processor or the receiving product maps it to the other. One source of truth in the code; the mapping lives in configuration and must be tested when either convention changes. LangSmith's documentation lists such a mapping on its intake side.

Emit both. Every span carries both sets of names. Every reader is happy; span size grows, and two names for one fact can drift apart when one is updated and the other is not.

Emit your own, map on export. Keep a small internal vocabulary (as the example does with kind) and map it to a convention in the exporter. You control the names; you also own the mapping forever.

Whatever you choose, pin it: record which convention and which version a service emits, as a resource attribute or in the service's documentation. The GenAI conventions moved repository in 2026; a pinned version is what lets you tell later which names a stored span used.

Where lock-in hides when the wire format is open

An open wire format removes lock-in at the export step. It does not remove it everywhere.

  • Instrumentation helpers. A vendor decorator (@traceable, register(...)) is code in your services. Swapping vendors means editing it. Wrapping it in one internal module keeps the edit in one place.
  • Derived data. Scores, annotations, datasets and saved queries created inside a product are often not part of the OTLP export. Ask how they leave before they become the record of a quarter's quality reviews.
  • Query languages and dashboards. Each backend has its own. Dashboards are rewritten on a move, not migrated.
  • Retention defaults. A product's retention decides what you can investigate next month. That is a policy decision; make it explicitly, and keep audit events outside the product.

Checklist#

In one breath#

Four names that look alike sort into two layers: OpenTelemetry and OpenInference say what a record is called, Phoenix and LangSmith store the records and show them. Because all four speak OTLP, you can change the product without rewriting the services, provided you picked a convention, pinned its version and masked the content before export. What no tool decides for you is what to record, what to leave out, and where the audit events live.

Where this leads#

You can now place any tracing name in its layer and ask the questions that matter: which convention, where the data lives, what is masked, what leaves with you. What this page did not cover is what goes wrong when the volume grows: a label that multiplies time series, a sampler that drops the one trace you need, and tenants who must not see each other's runs. That is the subject of Advanced.