It is 14:10 on the simulated day. The on-call engineer has the trace of case UW-006 open: run run-7f3a, submission SUB-1006, the run in which the roof-data provider never answered. The waterfall is clear. Two get_roof_assessment spans end in error after 2000 ms each, then the agent writes request_info. (The model recommends; a person decides. request_info is a recommendation to the underwriter.)
Now the engineer opens the team's shared dashboard for model usage, the one another team built. It says the agent-worker service made no model calls today. That is false: the trace on the other screen holds two model spans with token counts. The dashboard is not broken. It asks for a field named gen_ai.usage.input_tokens. Our worker writes tokens_in. Both mean the same thing, and no tool can know that.
This scene is illustrative, but the problem is common. Every team that writes its own tracer also invents its own names: tokens_in, input_tokens, prompt_tokens. Every backend, dashboard and alert then needs a translation for every team. And when the company changes backend, every exporter is rewritten. The records are good evidence, but only the team that wrote them can read them.
Evidence that only its author can read is half the work. A shared standard for the shape and the names of telemetry is the other half.
The agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
1. One standard, several parts#
Plain English. OpenTelemetry is an open project that defines how a program records telemetry (the records a running system emits about itself), what the records are called, and how they travel to wherever they are stored. It is not the place where they are stored.
A tiny example. Our worker records one span for the first roof lookup. With OpenTelemetry, the same span gets a standard name (execute_tool get_roof_assessment), a standard attribute for the tool name (gen_ai.tool.name), and a standard way to leave the process (a protocol called OTLP). Any backend that speaks OTLP and knows the names can show it.
The project's own documentation calls it "an observability framework and toolkit" that is "vendor- and tool-agnostic", and says plainly that "OpenTelemetry is not an observability backend itself". It is a Cloud Native Computing Foundation project. Its parts, as the documentation lists them:
| Part | What it gives you | In our example's terms |
|---|---|---|
| Specification | The rules every implementation follows | What a span, a trace id and a status are |
| API | The calls your code makes: start a span, set an attribute | What Tracer.span in telemetry.py does for us |
| SDK | The implementation behind the API: sampling, batching, export | What our example does not have: it keeps spans in a list |
| Instrumentation libraries | Spans for common libraries without writing them yourself | Spans for an HTTP client or a web framework |
| Collector | A separate program that receives, processes and exports telemetry | Between the worker and the backends |
| OTLP | The wire protocol from process to Collector to backend | How the span leaves the worker |
| Semantic conventions | The dictionary of names | gen_ai.usage.input_tokens instead of tokens_in |
The signals it covers today are traces, metrics, logs and baggage (context passed between signals). Profiles are listed as "under development or at the proposal stage". The words are the ones this site already uses: a log records an event, a metric summarizes measurements, a trace connects the steps of one run. What a span, a parent and a traceparent header are is explained on Tracing; what a model span or a tool span should record is on GenAI telemetry. This page is about the standard that names them.
What this is not. OpenTelemetry records what happened; it does not judge whether the run was acceptable. Grading a run from its trace belongs to Eval Engineering, for example its page on traces. And the runtime that wraps the model call, where these spans are opened, is the subject of Harness Engineering.
Diagram: From one span to any backend · the parts of OpenTelemetry
Read left to right. The worker's code calls the API; the SDK decides what to keep and sends it over OTLP; the Collector receives, processes and exports; backends store and query. The names in the middle come from the semantic conventions.
- 1 · In the worker process
- API: Start a span, set an attribute, mark an error. Libraries depend on this alone; with no SDK it records nothing.
- SDK: The implementation behind the API, configured once at startup by the application owner.
- Resource: Attributes describing the process, attached to everything it emits.
- 2 · Names and wire
- Semantic conventions: Agreed attribute names and values, so every tool reads the same field the same way.
- OTLP: The protocol spans, metrics and logs use to leave the process. Stable for traces, metrics and logs.
- 3 · Collector
- Receivers: Collect telemetry from one or more sources. Active only when a pipeline names them.
- Processors: Modify or transform telemetry before export: batch, filter, drop attributes.
- Exporters: Send telemetry to one or more backends or destinations.
- 4 · Backends
- Store and query: Where traces are indexed and searched. OpenTelemetry is not a backend itself.
2. API and SDK: two halves on purpose#
Plain English. The API is the set of calls your code makes ("start a span called X"). The SDK is the machinery that decides what happens to those calls ("keep one trace in ten, batch them, send them to this address").
Tiny example: the code around the roof lookup says "start a span get_roof_assessment, set attempt_number to 1, mark it as an error". That is API. Whether the span is kept, how many spans are batched together, and where they are sent: that is SDK, configured once at startup.
The specification makes the split strict. Instrumentation authors "MUST NOT directly reference any SDK package of any kind, only the API." And without an SDK installed, the trace API is a "no-op": calls have no effect, with one exception. The API still returns a non-recording span that carries the parent's context, so the trace id keeps flowing to the next service even when this process records nothing.
Why build it that way? A library author, say of an HTTP client, can add spans to the library without forcing every user to ship telemetry. The application owner, who knows where the data should go, installs the SDK. The documentation puts it this way: a library that depends on the API "will have no impact on applications using that library, unless the OpenTelemetry SDK is imported."
Before AI. A Java team has seen this split already. SLF4J is "a simple facade or abstraction for various logging frameworks": libraries log through the SLF4J API, the application picks Logback or another provider, and with no provider SLF4J falls back to a no-operation logger. Spring Boot does the same for tracing: Actuator auto-configures Micrometer Tracing, "a facade for popular tracer libraries", which can bridge to OpenTelemetry and export with OTLP, and it putstraceIdandspanIdinto the logging MDC so every log line carries the trace it belongs to. The OpenTelemetry API is the facade; the SDK is the provider.
3. Resource: who is speaking#
Plain English. A resource is the description of the thing that produced the telemetry: which service, which version, which machine. It is attached once, to everything a process emits, rather than to each span.
Tiny example: every span the worker emits carries service.name = agent-worker. The gateway's spans carry service.name = gateway. In our example this is the service field on each span. In OpenTelemetry it moves out of the span and into the resource.
The documentation defines a resource as "the entity producing telemetry as resource attributes". It warns that SDKs set service.name to unknown_service by default, "so it is recommended to set it explicitly", either in code or with the OTEL_SERVICE_NAME environment variable. Other resource attributes can come from OTEL_RESOURCE_ATTRIBUTES or from resource detectors that most SDKs provide (for example the pod and namespace on Kubernetes).
Production pitfall. A fleet of workers that forgotservice.nameshows up in the backend as one service calledunknown_service. The spans are all there; nobody can find them. Set it on day one, and make it the same string as theservicefield in your logs.
4. Instrumentation: written by you, or added for you#
Plain English. Instrumentation is the code that creates telemetry. Either you write it (code-based), or an agent or library adds it around code you did not change (zero-code).
Our example is all code-based: agent_sim.py opens every span itself with tracer.span(...). A zero-code setup would add, for instance, a server span for each HTTP request the gateway receives, without a line in the gateway's code. The documentation says zero-code solutions "are great for getting started" and capture activity "at the edges" of an application, while code-based ones give "deeper insight", and that "you can use both solutions simultaneously."
For an agent, the split matters. Zero-code instrumentation sees the HTTP call to the model provider and the HTTP call to the roof-data provider. It cannot know that the second call was the agent's retry of a tool, that the run belongs to SUB-1006, or that the policy version was 2026.2. Those facts exist only in your code, so the spans that carry them must be code-based.
5. Semantic conventions: one dictionary of names#
Plain English. Semantic conventions are the agreed names (and sometimes the agreed values) for common facts: the HTTP method, the database system, the model, the number of input tokens.
Tiny example: our worker writes tokens_in: 723. The convention says the attribute is gen_ai.usage.input_tokens. Same number, a name every compliant tool can read. The specification describes them as definitions of "the keys and values which describe commonly observed concepts, protocols, and operations." The general conventions are at version 1.45.0 on the day this page was written, and cover areas from HTTP and messaging to databases and CI/CD.
Each convention also has a stability level. This matters more than it sounds. For a stable convention, the project allows only changes that can be described in a schema file, so a dashboard built on it keeps working. For anything in development, "breaking changes and performance issues MAY occur". A name in development can be renamed next quarter.
6. The GenAI conventions, as they read today#
This is where the ground moves. Three findings from the documentation read on 2026-11-17, each surprising enough that the fact log marks it for a second read before publishing:
| Finding | What the documentation shows | What it means for you |
|---|---|---|
| They moved | The pages under opentelemetry.io/docs/specs/semconv/gen-ai/ now say the GenAI conventions "have moved" to a separate repository, open-telemetry/semantic-conventions-genai | Bookmarks and links in older articles point to a notice page |
| Still Development | The new repository's GenAI overview reads "Status: Development"; it has no published release yet | Expect renames; pin the version your instrumentation follows |
| Names were renamed | gen_ai.system is replaced by gen_ai.provider.name; gen_ai.usage.prompt_tokens by gen_ai.usage.input_tokens; gen_ai.usage.completion_tokens by gen_ai.usage.output_tokens; gen_ai.prompt and gen_ai.completion are removed with no replacement | Data from older instrumentation and newer instrumentation may sit side by side under different names |
With that warning, here is what the conventions define for the kinds of step an agent takes. Every name below is copied from the documents in the new repository.
| Step | gen_ai.operation.name | Span name | Span kind | Required attributes |
|---|---|---|---|---|
| A model call (inference) | chat (or generate_content, text_completion) | {gen_ai.operation.name} {gen_ai.request.model} | CLIENT (INTERNAL for a model in the same process) | gen_ai.operation.name, gen_ai.provider.name |
| A tool call | execute_tool | execute_tool {gen_ai.tool.name} | INTERNAL | gen_ai.operation.name, gen_ai.tool.name |
| A retrieval | retrieval | {gen_ai.operation.name} | INTERNAL | gen_ai.operation.name |
| An agent the model steers | invoke_agent | {gen_ai.operation.name} {gen_ai.agent.name} | INTERNAL | gen_ai.operation.name |
| A workflow the application steers | invoke_workflow | same pattern | INTERNAL | gen_ai.operation.name |
| A planning phase | plan | plan {gen_ai.agent.name} | INTERNAL | gen_ai.operation.name |
The choice between invoke_agent and invoke_workflow is worth a sentence. The documentation says invoke_agent applies "when an LLM drives the control flow through tool calls", and invoke_workflow "when the application defines the control flow". Our underwriting agent runs a fixed script of steps, so its root span is a workflow. A coding agent that lets the model choose the next tool, like session CA-001 (read the repository instructions, edit, run tests, open a pull request), would be an agent.
On a model span, the recommended attributes include gen_ai.request.model, gen_ai.response.model, gen_ai.response.finish_reasons (an array of strings), gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, the cache and reasoning token counts, and gen_ai.response.time_to_first_chunk for streaming responses (in seconds). An error sets the stable attribute error.type. On a tool span, gen_ai.tool.call.id is set only "if the model or provider supplied an identifier", and gen_ai.conversation.id comes with a sharp instruction: "Don't invent a fallback such as a UUID or trace ID."
Common confusion.gen_ai.operation.nametakes well-known values when one applies; the registry says the matching value "MUST be used; otherwise, a custom value MAY be used." The same rule holds forgen_ai.provider.name. Our stand-in model is not a listed provider, so it gets a custom value.
7. The UW-006 tool span, renamed#
Now the mapping. On the left, the fields our example's Tracer writes for the first get_roof_assessment call of UW-006 and for the write_recommendation model call. On the right, where each one goes under OpenTelemetry. The prose first, then the table: names that the conventions define move to those names; facts that only our business knows keep their meaning under our own prefix, uw., so nobody mistakes them for standard names.
| Our field (example) | OpenTelemetry | Why |
|---|---|---|
service = agent-worker | resource service.name | Describes the process, not the step |
kind = tool, name = get_roof_assessment | span name execute_tool get_roof_assessment, kind INTERNAL, gen_ai.operation.name = execute_tool, gen_ai.tool.name | The execute-tool convention |
status = error, attribute error = timeout | status ERROR, error.type = timeout | error.type wants a low-cardinality identifier; timeout is one |
status = ok | status UNSET | Ok is for an operation "validated by an Application developer or Operator"; the default is Unset |
attempt_number = 1 | uw.attempt_number | The GenAI documents read here define no attribute for it |
peer_service, timeout_ms | uw.peer_service, uw.timeout_ms | Ours. The network call to the provider would be its own client span under this one |
event retry_scheduled | span event retry_scheduled | Span events exist in OpenTelemetry under the same idea |
model = large-model | gen_ai.request.model | Inference convention |
tokens_in, tokens_out | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens | Inference convention (the old prompt_tokens names are deprecated) |
finish_reason = stop | gen_ai.response.finish_reasons = ["stop"] | An array: one reason per generation |
time_to_first_token_ms | uw.time_to_first_token_ms | The standard gen_ai.response.time_to_first_chunk is for streaming responses, in seconds; the stand-in does not stream |
run_id, submission_id, policy_version | uw.run_id, uw.submission_id, uw.policy_version | Business ids. Not gen_ai.conversation.id: a run is not a conversation, and the convention says not to invent one |
content (input and output) | not exported by default; opt-in gen_ai.tool.call.arguments and gen_ai.tool.call.result | Section 8 |
The snippet below takes two spans of run-7f3a from the example's simulated day (the first get_roof_assessment attempt and write_recommendation), then converts them with a small function, to_otel. The function is a design for this page, not part of any library. It prints a readable view of the span; the SDK and OTLP encode the same content in their own format. A successful run proves that every field of our span has a home: a standard name, our own prefixed name, or a deliberate choice to leave it out.
import json
from day import run_by_case, simulate_day
from telemetry import inject
# 1. Two spans of run-7f3a (case UW-006), taken from the simulated day.
run = run_by_case(simulate_day(), "UW-006")
spans = run["trace"]["spans"]
tool = next(s for s in spans if s["name"] == "get_roof_assessment") # attempt 1
model = next(s for s in spans if s["name"] == "write_recommendation")
# 2. The same spans with OpenTelemetry names (a design for this site, not a library).
def to_otel(span):
a, attrs = span["attributes"], {}
if span["kind"] == "tool":
name, kind = f"execute_tool {span['name']}", "INTERNAL"
attrs = {"gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": span["name"]}
elif span["kind"] == "model":
name, kind = f"chat {a['model']}", "CLIENT"
attrs = {"gen_ai.operation.name": "chat", "gen_ai.provider.name": "stand-in",
"gen_ai.request.model": a["model"], "gen_ai.usage.input_tokens": a["tokens_in"],
"gen_ai.usage.output_tokens": a["tokens_out"],
"gen_ai.response.finish_reasons": [a["finish_reason"]]}
else: # the root: the application runs the steps
name, kind = "invoke_workflow underwriting-agent", "INTERNAL"
attrs = {"gen_ai.operation.name": "invoke_workflow", "gen_ai.agent.name": "underwriting-agent"}
if span["status"] == "error":
attrs["error.type"] = a["error"]
used = {"model", "tokens_in", "tokens_out", "finish_reason", "error", "stand_in"}
attrs |= {f"uw.{k}": v for k, v in a.items() if k not in used} # our own facts, our own prefix
return {"resource": {"service.name": span["service"]}, "name": name, "kind": kind,
"trace_id": span["trace_id"], "span_id": span["span_id"],
"parent_span_id": span["parent_span_id"],
"status": "ERROR" if span["status"] == "error" else "UNSET",
"attributes": attrs, "events": [{"name": e["name"], **e["attributes"]} for e in span["events"]]}
print(json.dumps(to_otel(tool), indent=2))
print(json.dumps(to_otel(model)["attributes"], indent=2))
print(inject(tool))
Run in a scratch copy of examples/uw-observe/ (saved as otel_view.py, run with /opt/homebrew/bin/python3 otel_view.py). The output, pasted:
{
"resource": {
"service.name": "agent-worker"
},
"name": "execute_tool get_roof_assessment",
"kind": "INTERNAL",
"trace_id": "7540bd55d4e55ccb8f04f3d81880717a",
"span_id": "a557613ffd8a0c4f",
"parent_span_id": "0feeb0f297585ee0",
"status": "ERROR",
"attributes": {
"gen_ai.operation.name": "execute_tool",
"gen_ai.tool.name": "get_roof_assessment",
"error.type": "timeout",
"uw.attempt_number": 1,
"uw.peer_service": "roof-provider",
"uw.timeout_ms": 2000
},
"events": [
{
"name": "retry_scheduled",
"backoff_ms": 500,
"next_attempt_number": 2
}
]
}
{
"gen_ai.operation.name": "chat",
"gen_ai.provider.name": "stand-in",
"gen_ai.request.model": "large-model",
"gen_ai.usage.input_tokens": 723,
"gen_ai.usage.output_tokens": 111,
"gen_ai.response.finish_reasons": [
"stop"
],
"uw.tier": "large",
"uw.time_to_first_token_ms": 344
}
00-7540bd55d4e55ccb8f04f3d81880717a-a557613ffd8a0c4f-01
Three things to read in it. The ids did not change: trace_id is 32 lowercase hex characters and span_id 16, exactly what OpenTelemetry uses, so a renamed span still joins the same trace. The last line is the traceparent value that telemetry.inject writes; OpenTelemetry's default propagator uses the same W3C Trace Context header, so the example's context already travels in the standard format. And the dashboard from the opening scene now finds gen_ai.usage.input_tokens on the model span.
In our insurance agent. The token counts above (723 in, 111 out) come from the scripted stand-in and are illustrative. They describe the mechanism, not any model.
Here is the same trace in the waterfall lab. Open any span and switch to the OpenTelemetry view to see its standard name, kind and attributes next to the example's own fields.
The spans of UW-006, for a reader without the lab. The worker rows are the ones the example's agent_sim.py records; the gateway, queue and cockpit rows follow the series plan for the example.
| Span | Example kind | OpenTelemetry name | OpenTelemetry span kind | Status (example → OTel) |
|---|---|---|---|---|
The gateway's request span (POST /submissions) | http | per the HTTP conventions | SERVER | ok → Unset |
| The queue's producer span | queue | per the messaging conventions | PRODUCER | ok → Unset |
| The queue's consumer span | queue | per the messaging conventions | CONSUMER | ok → Unset |
underwrite_submission | agent | invoke_workflow underwriting-agent | INTERNAL | ok → Unset |
plan_steps | model | chat small-model | CLIENT | ok → Unset |
gather_evidence | agent | gather_evidence (no GenAI convention; our own name) | INTERNAL | ok → Unset |
get_roof_assessment, attempt 1 | tool | execute_tool get_roof_assessment | INTERNAL | error → Error, error.type = timeout |
get_roof_assessment, attempt 2 | tool | execute_tool get_roof_assessment | INTERNAL | error → Error, error.type = timeout |
search_guidance | retrieval | retrieval | INTERNAL | ok → Unset |
find_similar_decisions | retrieval | retrieval | INTERNAL | ok → Unset |
check_eligibility | policy_check | check_eligibility (our own name) | INTERNAL | ok → Unset |
write_recommendation | model | chat large-model | CLIENT | ok → Unset |
save_checkpoint | checkpoint | save_checkpoint (our own name) | INTERNAL | ok → Unset |
record_recommendation | tool | execute_tool record_recommendation | INTERNAL | ok → Unset |
| The underwriter's decision (second trace, span link to this one) | human | the cockpit's request span | SERVER | ok → Unset |
Diagram: The worker spans of UW-006 · and the GenAI operation each one maps to
Left: the spans the example's worker records for run run-7f3a, indented under their parent. Right: the OpenTelemetry GenAI operation names, as documented today (status Development), wired to the spans they name. Spans without a GenAI convention keep our own names. Times come from a simulated clock; the agent is a scripted stand-in.
- underwrite_submission: agent · ok
- plan_steps: model · small-model · ok
- gather_evidence: agent · grouping span · ok
- get_roof_assessment (attempt 1): tool · 2000 ms · error
- get_roof_assessment (attempt 2): tool · 2000 ms · error
- search_guidance: retrieval · ok
- find_similar_decisions: retrieval · ok
- check_eligibility: policy_check · ok
- write_recommendation: model · large-model · ok
- save_checkpoint: checkpoint · ok
- record_recommendation: tool · write · ok
Notice what the table loses. Two retrieval spans are now both called retrieval, because the convention names the span after the operation. The difference between guidance and past decisions has to live in an attribute (a data source id, or our own uw. attribute). A standard name buys interoperability; it can cost you a distinction you relied on. Decide where each distinction lives before you rename.
8. Prompts and tool results: attributes, events, or nothing#
The example keeps every span's content: the broker's email in the model input, the submission in the tool input. Under OpenTelemetry, that content is a separate decision, and the default is "no".
Plain English. Content capture means recording what went into and came out of a model or tool: messages, instructions, arguments, results. The GenAI conventions treat it as opt-in, because it is large and often sensitive.
The documentation is direct: model instructions, user messages and model outputs "are considered sensitive and are often large in size", and instrumentations "SHOULD NOT capture them by default". It names three patterns:
| Pattern | Where content goes | When the documentation suggests it |
|---|---|---|
| Do not record (default) | Nowhere | The default |
| Record on span attributes | gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions on the span | Volume is manageable and privacy rules do not apply or the storage complies, "for example, in pre-production environments" |
| Store externally, record references | A separate store; the span carries a reference | "Recommended in production environments", because external storage "enables separate access controls" |
One switch governs this. If it is set by environment variable, the documentation says the variable "MUST be named OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT", with the values NO_CONTENT (the default), SPAN_ONLY, EVENT_ONLY and SPAN_AND_EVENT, and an invalid value should fall back to NO_CONTENT. It governs the message attributes, the tool's gen_ai.tool.call.arguments and gen_ai.tool.call.result, the retrieval's gen_ai.retrieval.query.text, prompt variables and memory records. For an inference call, the content can also travel on an event, gen_ai.client.inference.operation.details, at DEBUG severity.
Production pitfall. The variable your library reads may not be the one in the specification. The Python OpenAI instrumentation's README, read the same day, documentsOTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT(with valuesspan_only,event_only,span_and_event, and a legacytrue), and it follows the newer conventions only whenOTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimentalis set. Read the README of the exact package you install, and test with a fake secret that the content is absent.
For UW-006, the safe reading is: the tool span keeps its name, status, error.type and our uw.attempt_number, which is everything the on-call engineer needed at 14:10. The submission and the email stay out of the trace unless someone decides otherwise. Who may decide, how redaction works and how long content may be kept is the subject of Sensitive data.
9. Out of the process: OTLP and the Collector#
Plain English. OTLP is the format and protocol spans use to leave a process. The Collector is a separate program that receives them, does work on them (batching, filtering, removing fields) and sends them on to one or more backends.
Tiny example: the worker sends its spans over OTLP to a Collector on the same host. The Collector drops the uw.submission_id attribute for the copy that goes to a vendor's trace store, keeps it for the in-house store, and retries when either one is slow. The worker does not know either backend exists.
What the documentation says about each:
| Topic | Documented today |
|---|---|
| OTLP transports | gRPC and HTTP |
| Default ports | 4317 for OTLP/gRPC, 4318 for OTLP/HTTP |
| HTTP paths | /v1/traces, /v1/metrics, /v1/logs |
| Encodings over HTTP | Protobuf, binary or JSON |
| Stability | "Stable for the trace, metric and log signals"; Development for profiles |
| Collector components | Receivers "collect telemetry"; processors "modify or transform it"; exporters "send data to one or more backends"; connectors "join two pipelines" |
| Why a Collector | Recommended "alongside your service" in production, so the service offloads data quickly and the Collector handles retry, batching, encryption and sensitive-data filtering |
Below are two configurations in their documented shape. The first is the SDK setup in Python, from the OpenTelemetry Python pages. Run, it would make every span of the process carry service.name and leave over OTLP/HTTP. Documented shape; not run here.
# Documented shape; not run here (opentelemetry.io/docs/languages/python/, read 2026-11-17)
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace import Status, StatusCode
resource = Resource.create(attributes={SERVICE_NAME: "agent-worker"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("uw.agent_worker")
with tracer.start_as_current_span("execute_tool get_roof_assessment") as span:
span.set_attribute("gen_ai.operation.name", "execute_tool")
span.set_attribute("gen_ai.tool.name", "get_roof_assessment")
span.set_attribute("uw.attempt_number", 1)
span.add_event("retry_scheduled", {"backoff_ms": 500})
span.set_attribute("error.type", "timeout")
span.set_status(Status(StatusCode.ERROR))
The second is a Collector pipeline: receive OTLP on both default ports, batch, export over OTLP/gRPC. Note the last block. The documentation says "Configuring a receiver does not enable it": a component is active only when a pipeline in service names it. Documented shape; not run here.
# Documented shape; not run here (opentelemetry.io/docs/collector/configuration/, read 2026-11-17)
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
exporters:
otlp_grpc:
endpoint: otelcol:4317
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp_grpc]
The exporter key otlp_grpc is the one the configuration page uses today. Older material often shows otlp. Use the name your Collector release documents. Which backend sits behind the exporter, and what each one is good at, is on Backends.
10. Where it goes wrong#
| Failure | What you see | The fix |
|---|---|---|
No service.name | Every worker appears as unknown_service | Set OTEL_SERVICE_NAME or the resource in code, the same string as in your logs |
| Old and new names mixed | Half the model spans have gen_ai.usage.prompt_tokens, half gen_ai.usage.input_tokens; token totals are half of reality | Pin the convention version per instrumentation; translate in the Collector or the query until all services move |
| Content assumed off, actually on | The broker's email appears in the trace store | Set the capture variable explicitly to NO_CONTENT; check with a planted fake value |
| Business ids squeezed into standard names | run_id written into gen_ai.conversation.id | Keep business ids under your own prefix; the convention says not to invent a conversation id |
| A configured but unused component | The Collector starts, but a processor never runs | Name every component in a service.pipelines entry |
| Distinctions lost in renaming | Two different retrievals both read retrieval | Carry the distinction in an attribute before you rename |
| A Development name treated as stable | A dashboard goes blank after an upgrade | Treat every GenAI name as changeable; alert on "zero model spans" as well as on high values |
Living with a convention in DevelopmentWhat "Development" allows, what is changing now, and how to pin
The versioning document says that while signals are in development "breaking changes and performance issues MAY occur", and that a Development signal may even be removed. For stable semantic conventions, only changes that a schema file can describe are allowed, and the same document records "a moratorium on relying on schema transformations for telemetry stability". In practice: a Stable name will not move under you; a Development name may, and no automatic translation is promised.
The GenAI repository's changelog.d/ folder, read on 2026-11-17, already holds unreleased breaking changes: gen_ai.request.top_k becomes an integer for decoding only, with retrieval moving to gen_ai.retrieval.top_k; the provider value mistral_ai becomes mistral; JSON-schema attributes on spans must be structured values, with the JSON-string fallback removed; and invoke-agent client calls move from gen_ai.client.operation.duration to gen_ai.client.invoke_agent.duration. The token metrics are already named gen_ai.client.inference.usage.input_tokens and gen_ai.client.inference.usage.output_tokens.
A workable policy for a team:
- Record, per service, which convention version each instrumentation follows (the Python OpenAI instrumentation, for instance, aligns with v1.30.0 unless the opt-in variable is set).
- Upgrade instrumentation in one change across services, not one service at a time, or translate old names to new in the Collector during the window.
- Build dashboards on a small set of names, and keep a test that sends one known span and checks the dashboard query finds it.
- Re-read the GenAI documents on each upgrade. The fact log of this page lists the URLs.
One span per attempt, or one span for all attempts?The convention's retry rule, and why UW-006 still records two tool spans
The GenAI spans document says: "If a transient issue happened and the request was retried automatically, the corresponding span SHOULD cover the duration of the logical operation with all retries." Read alone, that suggests one execute_tool span for both roof attempts.
The rule describes an automatic retry inside a client, invisible to the caller. UW-006 is different. The retry is the agent's own decision: it waits 500 ms, records a retry_scheduled event, and calls the tool a second time. Each attempt has its own outcome, and the reliability question ("did it retry exactly once, with a backoff?") needs both. So the example records two tool spans, each with uw.attempt_number.
If an HTTP client library under the tool span also retried at the network level, those retries belong inside one client span, as the convention says. Two levels, two answers: one span per attempt the agent chose, one span per logical network operation below it. Write down which level each span is at, so the next engineer does not count network retries as agent attempts. How retries and timeouts should be designed is the subject of Agent Reliability Engineering.
The Collector in production: what it is for, and what it is notOffload, filter, fan out; and why redaction there is a second line, not the first
The Collector page's case for running one "alongside your service" is operational: the service hands data off quickly, and the Collector does retry, batching, encryption and sensitive-data filtering. Connectors, which "join two pipelines, acting as both exporter and receiver", let one pipeline feed another, for example traces into a metrics pipeline.
Three trade-offs to decide explicitly:
| Decision | Option A | Option B |
|---|---|---|
| Where content is removed | In the process, before export (nothing sensitive crosses the network) | In the Collector (one place to change rules, but data has already left the process) |
| How many Collectors | One per host or pod, close to the service | A shared tier that all services send to |
| Who owns the configuration | The platform team, centrally | Each service team, per service |
For agent telemetry the first row matters most. A processor in the Collector can drop an attribute, but the content was already serialized and sent over the network to reach it. If the capture switch is NO_CONTENT in the process, the Collector never sees the email at all. Treat Collector filtering as the second line, after the capture decision.
The same idea for a coding agent. SessionCA-001maps the same way: the session is aninvoke_agentspan (the model chooses the next tool), each test run or file edit is anexecute_toolspan, and the test output stays out of the trace unless content capture is turned on. How a coding agent's change then travels through review and CI is on Diff to Deploy.
Checklist#
- Every service sets
service.name, and it matches theservicefield in its logs. - Libraries depend on the OpenTelemetry API only; the application installs and configures the SDK once.
- Model spans use
gen_ai.operation.name,gen_ai.provider.nameand thegen_ai.usage.*token names, notprompt_tokens,completion_tokensorgen_ai.system. - Tool spans are named
execute_tool {tool name}with kindINTERNAL, and a failure sets status Error anderror.type. - Business ids (
run_id,submission_id,policy_version,attempt_number) live under your own prefix, never in a standard name with another meaning. - The content capture setting is explicit in every environment, and a planted fake secret proves it is absent where it should be.
- Each instrumentation's convention version is written down, and upgrades move all services together.
- Every Collector component you configured appears in a
service.pipelinesentry. - A test sends one known span through the pipeline and checks that the dashboard query finds it.
In one breath#
OpenTelemetry is the vendor-neutral standard for how telemetry is made, named and moved: an API your code calls, an SDK that decides what happens next, a Collector in the middle, OTLP on the wire, and semantic conventions as the shared dictionary. Our UW-006 tool span becomes execute_tool get_roof_assessment with status Error and error.type timeout, while our business ids keep their own prefix. The GenAI conventions that name model, tool and agent spans have moved to their own repository and are still in Development, with renamed attributes and content capture off by default. Use them, pin them, and expect them to change.
The spans now have names any backend can read, which makes them easy to share, sample and delete. That is right for diagnostics and wrong for one kind of record: the proof that a recommendation was stored and that a named role decided. Audit logs is about the records that may not be dropped.