All chapters
  1. Start hereA log records. A metric summarizes. A trace connects
  2. Foundations · 1Before AI: stdout, structured logs, severity and aggregation
  3. Foundations · 2Logs, metrics, traces and events: what each one answers
  4. Foundations · 3Correlation identifiers: the ids that join the records
  5. Foundations · 4Distributed tracing: one span tree across HTTP, queues, tools and models
  6. Agent telemetry · 5GenAI telemetry: model calls, tokens, retrieval, tool steps and handoffs
  7. Agent telemetry · 6OpenTelemetry: semantic conventions and instrumentation
  8. Agent telemetry · 7Audit, diagnostic and security logs: three promises, three readers
  9. Agent telemetry · 8Sensitive trace content: redaction, access, retention and sampling
  10. Operations · 9From the process to Splunk, ELK, Datadog, Grafana and Jaeger
  11. Operations · 10Dashboards, SLIs, SLOs, error budgets and alerts a person can act on
  12. Operations · 11Online quality: eval labels, drift and human feedback
  13. Operations · 12Reproducibility and replay: fixtures, snapshots and their limits
  14. Operations · 13Incident response: from an alert to a root cause to a golden case
  15. Operations · 14Cost engineering: per-run cost, budget anomalies and model routing
  16. Reference · 15The tracing ecosystem: OpenTelemetry, OpenInference, Phoenix and LangSmith
  17. Reference · 16Advanced: cardinality, tail sampling, tenants and compliance
  18. Reference · 17The insurance lab: one timeout, followed through every record
  19. ReferenceGlossary
  20. AboutAbout this site and its sources
Operations · 11

Online quality: eval labels, drift and human feedback

A run can be fast, cheap and wrong. How grader labels, underwriter overrides and drift over time become telemetry next to latency and errors.

Fast, cheap, wrong
Latency and errors say a run worked. They cannot say the recommendation was right.
Labels on traces
Cheap rules mark finished runs: missing evidence, hedged reasons, a skipped step. Some go to a person for review.
The underwriter as a signal
Each override is a judgement on the agent. The override rate drifted after a policy change while every other chart stayed flat.

On 1 July the insurer switched the policy fixture from version 2025.4 to 2026.2. The new version lowers the roof-age limit from 30 years to 25 in wind zone C. The underwriting agent picked up the new version the same morning. Its runs stayed as fast as before. Error rates did not move. The SLO dashboard of the SLOs and alerts page was green for a week. The model recommends; a person decides.

And the underwriters started to disagree with it. Not always; one decision in five instead of one in twelve. Each of them did the right thing for the submission in front of them: they read the recommendation, saw it was off, and decided otherwise. Nobody saw the pattern, because no chart counted it. A week later a senior underwriter mentions in a meeting that "the agent seems worse since the policy update". She is right, and the telemetry could have said so on 2 July.

This page adds the third kind of signal. Availability and latency say whether the system worked. Cost says what it spent. Quality signals say whether the output was any good, and they come from three sources: cheap rules applied to every finished run, the people who act on the output, and changes in those numbers over time. One note before any number appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.

What this is not. Deciding whether a behavior is acceptable (graders, rubrics, judges, golden datasets, release gates) is evaluation, and it lives on eval.sammuthu.com. This page is about the telemetry side: attaching quality signals to production traces, watching them as time series, and routing the doubtful runs to a person.

1. Why the usual signals are blind here#

Take case UW-001: a roof of 31 years in wind zone C. Under the policy fixture, rule R-ROOF-01 says not eligible, and the agent recommends decline. Now imagine a version of the agent that recommends approve instead. What does the trace show?

The same spans. The same durations, give or take a few milliseconds. The same token counts. Status ok on every span. The recommendation is a string inside one span's content, and no chart reads it. Every signal on the earlier pages describes how the run happened. None describes whether the result was right.

That is the gap a quality signal fills: a measurement attached to a run that says something about the quality of its output, not about its mechanics. It can be cheap and certain (the recommendation lists missing evidence), cheap and suggestive (the reasons contain three hedge words), or expensive and authoritative (an underwriter overrode it).

Diagram: Three sources of quality signals, from seconds to days after a run

Read left to right in time. Rules label every finished run within seconds. Flagged runs wait in a review queue for a person. The underwriter's decision, hours later, records whether they overrode the agent. All three are joined to the run by run_id and counted over time, where drift becomes visible.

  1. Seconds · rules
    1. Online eval labels: Cheap deterministic rules read the output and the trace and attach named labels.
  2. Hours · review
    1. Review queue: Runs with low_confidence, hedged_language or step_limit wait for a person with authority.
  3. Hours to days · decision
    1. Underwriter's override: The decision records override true or false and a reason; counted, it becomes the override rate.
  4. Over days · series
    1. Drift detector: A rolling rate compared with its usual level, split by policy version.

2. Online eval labels: cheap rules on every finished run#

An online eval label is a label that a rule attaches to a production run after it finishes, while it is still fresh. "Online" means in production, on real runs, as opposed to an offline test suite. The rules are small and deterministic, and each label records the rule that produced it, so that anyone reading the label can see why it is there.

The example's labeller reads the finished recommendation and the trace:

def label_run(output: dict[str, Any], trace: dict[str, Any]) -> list[dict[str, Any]]:
    """Online labels for one finished run. Each label names the rule that produced it."""
    names = [s["name"] for s in trace["spans"]]
    root = next(s for s in trace["spans"] if s["name"] == "underwrite_submission")
    labels = []
    if output["stated_confidence"] < 0.5 and output["recommendation"] in ("approve", "decline"):
        labels.append({"name": "low_confidence", "value": output["stated_confidence"], "rule": "stated_confidence < 0.5 on approve/decline"})
    hedges = sorted({m.group(0).lower() for r in output["reasons"] for m in HEDGES.finditer(r)})
    if len(hedges) >= 2:
        labels.append({"name": "hedged_language", "value": hedges, "rule": "two or more hedge words in the reasons"})
    if root["attributes"].get("run_status") == "step_limit":
        labels.append({"name": "step_limit", "value": True, "rule": "run stopped at its step limit"})
    if "search_guidance" not in names:
        labels.append({"name": "skipped_step", "value": "search_guidance", "rule": "search_guidance not in the trace"})
    if output["missing"]:
        labels.append({"name": "missing_evidence", "value": output["missing"], "rule": "recommendation lists missing evidence"})
    if any(s["attributes"].get("untrusted") for s in trace["spans"]):
        labels.append({"name": "untrusted_content", "value": True, "rule": "a retrieved passage was marked untrusted"})
    return labels

Notice what the rules read. Two read the output (stated confidence, hedge words in the reasons). Two read the trace (a skipped step, a step limit). Two read both. The trace is what makes several of them possible: skipped_step is the label for case UW-008, where the agent reached the right answer without calling search_guidance. The output of UW-008 looks identical to a careful run. Only the trace shows the missing step.

On the simulated day the labeller marks six runs:

cd "$TMPDIR/uw-observe-quality" && /opt/homebrew/bin/python3 run_observe.py --show labels
== labels
UW-003  run-2f2f missing_evidence
UW-005  run-6cf9 untrusted_content
UW-502  run-48f5 step_limit, missing_evidence · flagged
UW-501  run-eb9a low_confidence, hedged_language · flagged
UW-008  run-df27 skipped_step
UW-006  run-7f3a missing_evidence

Each line is a fact about one run, stored with the trace by run_id. Two of them carry the word flagged. All labels are synthetic, from the scripted stand-in.

In our insurance agent. UW-006 gets the label missing_evidence, because the roof-data provider timed out twice and the recommendation lists roof_assessment under missing. That label is not a failure: the agent did what it should. It is still worth counting, because a rising share of runs with missing evidence is a dependency problem that the controlled SLI hides by design.

3. The review queue: where a person looks#

Some labels mean "a person should look at this". The example flags a run for human review when it carries low_confidence, hedged_language or step_limit. A review queue is the list of flagged runs, each with a link to its trace, waiting for someone with the authority to judge them.

FLAG_FOR_REVIEW = ("low_confidence", "hedged_language", "step_limit")

def flagged(labels: list[dict[str, Any]]) -> bool:
    return any(lb["name"] in FLAG_FOR_REVIEW for lb in labels)

UW-501 is this site's own case of an ambiguous recommendation: its reasons contain two or more hedge words and its stated confidence is low. It is flagged. UW-502, the run that looped on search_guidance until its step limit, is flagged too. The Cost page found the same run from another side: 21 times the median cost.

The queue has two jobs. In the short term, a reviewer catches a doubtful recommendation before the underwriter relies on it, or confirms it. In the long term, each reviewed run is a labelled example: the reviewer's verdict is exactly the kind of judgement an evaluation set is built from. Incidents shows how one such run becomes a versioned golden case.

Production pitfall. A queue that grows faster than people can review it is noise. Sample from it (the sampling idea applies) and measure the queue's age, not only its length. A flag that nobody reads within a day is a flag that does not exist.

4. Human feedback: the override as telemetry#

The underwriter's decision is recorded as an audit event, human_decision_recorded, with human_decision: {decided_by, decision, override, reason}. Audit logs treats it as evidence of who decided. Here the same record is a quality signal. Human feedback is any judgement a person makes about the agent's output in the course of their work; the override is the strongest one the underwriter cockpit records.

In case UW-009 the agent recommends approve and the underwriter declines. The audit event carries override: true and the reason. One override is one person's judgement on one submission. The override rate (overrides divided by decisions over a window) is a time series, and that series is what the opening story was missing.

On the simulated day the cockpit recorded 46 decisions and 5 overrides: an override rate of 0.109.

Source of the signalArrivesCoversStrength
Online eval label (a rule)Seconds after the runEvery runCheap and narrow: says what the rule checks, nothing else
Review of a flagged runHoursA sample of runsExpensive and careful
Underwriter's overrideHours to daysEvery decided runAuthoritative for that submission; noisy one by one
Common confusion. An override is not proof that the agent was wrong. The underwriter may know something the agent could not (a phone call with the broker). The override rate is a signal to investigate, not a score. Scores belong to the evals.

5. Drift: when the numbers move together#

Drift means a quality signal moves away from its usual level over time, while nothing in the code changed, or after a change nobody connected to it. A drift detector watches a rate over a rolling window and compares it with a baseline.

The example builds a 14-day series of daily aggregates around the policy change of 1 July (seven days on 2025.4, seven on 2026.2) and runs the simplest detector that works: the rate over the last three days, compared with the rate of the first seven days; alert when it is more than 0.08 above.

cd "$TMPDIR/uw-observe-quality" && /opt/homebrew/bin/python3 run_observe.py --show drift
== drift
override rate on the simulated day: 0.109 (5 of 46 decisions)
14-day series around the policy change on 2026-07-01: baseline 0.084 · window 3 days · threshold +0.08 · first alert 2026-07-02
  2026-06-24 2025.4 override 0.041 p95 3670 ms errors 0.000
  2026-06-25 2025.4 override 0.151 p95 3479 ms errors 0.054
  2026-06-26 2025.4 override 0.020 p95 3635 ms errors 0.019
  2026-06-27 2025.4 override 0.091 p95 3521 ms errors 0.065
  2026-06-28 2025.4 override 0.137 p95 3560 ms errors 0.000
  2026-06-29 2025.4 override 0.077 p95 3383 ms errors 0.038
  2026-06-30 2025.4 override 0.067 p95 3617 ms errors 0.044
  2026-07-01 2026.2 override 0.209 p95 3644 ms errors 0.023  window 0.114
  2026-07-02 2026.2 override 0.260 p95 3580 ms errors 0.039  window 0.181 ALERT
  2026-07-03 2026.2 override 0.184 p95 3477 ms errors 0.000  window 0.218 ALERT
  2026-07-04 2026.2 override 0.164 p95 3543 ms errors 0.018  window 0.201 ALERT
  2026-07-05 2026.2 override 0.192 p95 3627 ms errors 0.021  window 0.179 ALERT
  2026-07-06 2026.2 override 0.250 p95 3424 ms errors 0.041  window 0.200 ALERT
  2026-07-07 2026.2 override 0.209 p95 3556 ms errors 0.023  window 0.217 ALERT
same detector on the error rate: first alert None

Read three columns side by side. The p95 latency stays between 3.4 and 3.7 seconds on every day. The error rate wanders between 0 and 6.5 percent, before and after the change alike. The override rate sits around 0.08 for a week, then jumps above 0.2. The detector fires on 2 July, the second day on the new policy version. The last line is the opening story in one sentence: the same detector on the error rate never fires. All numbers are synthetic and illustrative.

Two details make this work. First, every record carries policy_version, so the series can be split by version and the jump lines up with the change. Second, the detector does not know about the change; it sees only numbers. The connection between "alert on 2 July" and "policy version 2026.2 since 1 July" is made by a person reading a chart where both are visible.

Control in the labWhat it changesWhat to look for
Window (days)How many days the rolling rate coversA shorter window alerts sooner and is noisier: one bad day before the change (0.151 on 24 June) can trip it
Threshold above baselineHow far the rate must riseA higher threshold alerts later or not at all
Signal (override rate, error rate)Which series the detector watchesThe error rate never fires: the change is invisible to it

Diagram: The policy change of 1 July, seen by three signals

Each lane is one signal over the same fourteen synthetic days. Latency and errors look the same before and after the change. Only the override rate moves, and the detector fires on the second day. The detector does not know about the change; the policy_version on every record lets a person connect the two.

  1. Latency
    1. p95 flat: The slowest runs take as long after the change as before it.
  2. Errors
    1. Error rate wanders: The same detector on the error rate never fires.
  3. Overrides
    1. Override rate jumps: Baseline 0.084; the three-day window reaches 0.181 on the second day of version 2026.2.
Before AI. A rules engine in a Java underwriting system had the same problem when someone changed a rule table: the service stayed healthy and the business outcome moved. Teams that caught it watched business metrics (approval rate, referral rate) next to technical ones. The agent makes that habit mandatory, because the model's behavior can shift with a policy document, a retrieved passage or a model update, not only with a code change.
Better detectors, and their assumptions

The example's detector compares a windowed rate with a fixed baseline and a fixed margin. It assumes the baseline period is representative, and that a constant margin means the same thing at every volume. Neither holds in general.

  • Volume-aware thresholds. With d decisions in the window and a baseline rate p, the standard error of the window rate is about the square root of p(1 − p)/d. A margin of a few standard errors adapts to quiet and busy days. At p = 0.084 and d = 150, one standard error is about 0.023, so the example's 0.08 margin is about 3.5 standard errors.
  • Change-point methods (CUSUM, for example) accumulate small deviations and detect a sustained shift sooner than a window that resets.
  • Split by segment. An overall rate can hide a segment that moved: the rule that changed on 1 July applies only in wind zone C, so that is the segment to watch first (the example's series is not split by segment; this is a design note). Watch the rate per segment that a policy version can affect, within the cardinality budget of Advanced.
  • Annotate changes. Put every policy version, prompt version and model version on the chart as a marker. The detector does not need them; the person reading the alert does.
Labels arrive late: joining feedback to the run

An online label arrives seconds after the run; an override may arrive hours or days later, in a separate trace from the cockpit. The join key is run_id (and submission_id for business reporting), never trace_id, because on this site the decision is a second trace linked to the first. A quality store therefore needs to accept late updates to a run's record, or to keep labels as their own records keyed by run_id. Rates computed before the late labels arrive must be recomputed or marked provisional; a dashboard that shows today's override rate at noon is showing a partial number.

The same idea for a coding agent. For a coding-agent session such as CA-001, the reviewer's verdict on the pull request is the override: approved as is, approved with changes, or rejected. The rate of "changes requested" per week is the same kind of drift signal, and it is how a team notices that a change to its repository instructions made the agent worse. Review itself is the subject of diff-to-deploy.sammuthu.com.

Checklist#

In one breath#

A run can be fast, cheap, in a controlled state and still wrong, so quality needs its own signals: rule-based labels on every finished run, a review queue for the doubtful ones, and the underwriter's overrides counted as a rate. Watched over time and split by policy_version, the override rate showed the policy change of 1 July on the second day, while latency and errors stayed flat.

Where this leads#

You can now see when the agent's output gets worse even though the system is healthy. Seeing it is not the same as explaining it. To find out why the agent recommended what it did on one run, you want to run it again with the same inputs and watch. What that takes, and what it can never prove, is the subject of Replay.