At 14:10 on the simulated day, three questions reach the on-call engineer within a few minutes of each other.
The product owner asks: how many roof lookups failed today? The broker's account team asks: why did the quote request SUB-1006 take seven seconds? And the compliance team asks: who decided on SUB-1006, and did the person overrule the agent? The model recommends; a person decides, so that last question has a real answer somewhere.
The engineer has the log explorer from Foundations: 661 structured log lines, searchable by field. Each question can be forced through it. Count the ERROR lines of category tool: 10. Filter run_id = run-7f3a and read the gaps between timestamps. Find the cockpit's line "decision request_info, override false". All three answers come out, and all three are fragile. The count is right only if every failure wrote exactly one line and no line was sampled away. The seven seconds are a guess built from gaps between points in time. And the decision line is a diagnostic record, which the platform may sample, redact or delete after 30 days.
The problem is not that the logs are bad. A log line answers one kind of question well: what happened at this moment, in this place. The other three questions need other kinds of record. This page names the four kinds, shows what each one stores and costs in the example, and gives a rule for which to reach for. One note before any record appears: the agent here is a scripted stand-in, so these records show how the mechanism works, not how good any model is.
1. Four questions, four records#
Start with the sentences this site uses everywhere. A log records an event. A metric summarizes measurements. A trace connects the steps of one run. The fourth kind of record, the event, is a fact with a time that has a meaning of its own.
In plain words, applied to the three questions above:
- The question "how many lookups failed today?" is about many runs at once. No single record holds the answer; a number computed over all of them does. That number is a metric.
- The question "why did this one take seven seconds?" is about one run, step by step: which step took how long, and which step was waiting for which. That is a trace.
- The question "who decided?" is about a fact the business must be able to show later. That is an event, and because it must survive, an audit event.
- And "what did the worker report at 14:07:14?" is the question a log line answers.
| Record | The question it answers | In the example | Unit |
|---|---|---|---|
| Log line | What happened here, at this moment? | ERROR get_roof_assessment failed: timeout after 2000 ms, attempt_number 1 | One event, one moment |
| Metric | How often, how much, how long, across many runs? | tool_errors_total = 10 today | A number per time window |
| Trace | Where did the time go in this one run, and which step called which? | run-7f3a: 14 spans, two 2,000 ms timeouts | One run |
| Event | What fact, with what time, must be known later? | human_decision_recorded, seq 75 | One fact |
The rest of the page takes the metric first, because it is the record most backend engineers use without thinking about how it works, and because its limits explain why the others exist.
Diagram: Four questions, four kinds of record
Each column is one question about the simulated day and the record that answers it. The log line is one moment in one place. The metric is a number over many runs and names no run. The trace keeps one run whole, step by step. The audit event is a business fact that is never dropped. None of the four can stand in for another.
- What happened here?
- Log line: One record of one event, with a time, a severity and the run's ids.
- How often, how long?
- Metric: A number computed over many measurements: a counter, a gauge or a histogram.
- Where did the time go?
- Trace: A tree of spans for one run, each with a start, a length, a status and a parent.
- What must be shown later?
- Audit event: A business fact with a time, written to an append-only ledger and never sampled.
2. Metrics: numbers about many runs#
A metric is a number computed over many measurements: a count, a sum, or a distribution. A tiny example: every time get_roof_assessment times out, the worker adds 1 to a number called tool_errors_total. At the end of the simulated day the number is 10. Nobody has to read 10 log lines to know it.
A metric has a name, a few labels that say which slice of the system it describes, and a value. Here is one measurement as the example writes it to out/metrics.jsonl, the first timeout of run-7f3a:
{"ts": "2026-09-14T14:07:14.994Z", "name": "tool_errors_total", "kind": "counter", "labels": {"tool": "get_roof_assessment", "error": "timeout"}, "value": 1}
Notice what is missing: no run_id, no trace_id, no submission. The labels are tool and error, both with few possible values. That absence is on purpose, and section 5 explains why. The example uses three kinds of metric. Each kind answers a different shape of question.
Counter: how many times#
A counter is a number that only goes up: something happened, add one (or add a number of tokens). The question it answers is "how many, in this window?". To get a rate, a dashboard takes the difference between two readings and divides by the time between them.
The example's counters on the simulated day, read from telemetry.Metrics.total:
| Counter | Labels | Total on the simulated day |
|---|---|---|
tool_calls_total | tool, status | 54 calls of get_roof_assessment |
tool_errors_total | tool, error | 10 (all timeouts, all between 14:00 and 15:00) |
runs_total | status | ok 41, fallback 5, step_limit 1, denied 1 |
tokens_total | model, direction | 102,741 (96,015 in, 6,726 out) |
The first question of the morning is now one number: 10 failed lookups out of 54. Split by hour, the counter says more: 0 errors in the 13:00 hour, 10 in the 14:00 hour, 0 in the 15:00 hour. That shape is the outage of the roof-data provider, 14:00 to 14:45, seen without opening a single run.
Gauge: how much, right now#
A gauge is a number that goes up and down: the current level of something. The question it answers is "how much, at this moment?". Queue depth, memory in use and runs in flight are gauges.
The example has one, runs_in_flight, set to 1 when a worker picks up a message and to 0 when it finishes. On the simulated day it never goes above 1, because the simulator runs one submission at a time. A real worker pool would show the outage as a rising gauge: runs waiting on a provider that does not answer pile up. The gauge is in the example to show the shape, not a measured load.
A gauge must not be summed over time. Ten readings of "3 runs in flight" do not mean 30 runs. That is the practical difference from a counter.
Histogram: how long, as a distribution#
A histogram records many measurements of one thing, usually a duration, as counts in buckets. The question it answers is "how is this spread out?": how many runs finished under one second, how many between five and ten.
The example's histogram run_duration_ms uses seven bucket bounds: 100, 250, 500, 1,000, 2,500, 5,000 and 10,000 ms, plus one last bucket for anything slower. The runner prints it for the simulated day:
cd "$TMPDIR/uw-observe-signals" && /opt/homebrew/bin/python3 run_observe.py
Simulated day 2026-09-14 · seed 7 · agent: scripted stand-in (no model called)
runs 48 · log records 661 · metric points 812 · audit events 93 · security log entries 6
run status: denied 1, fallback 5, ok 41, step_limit 1
roof-provider outage 14:00:00-14:45:00: 5 runs inside, get_roof_assessment timeouts 10 (of 54 calls)
run_duration_ms: count 48 · p50 3107 · p95 7346 · buckets [0, 0, 0, 0, 0, 42, 6, 0] (bounds [100, 250, 500, 1000, 2500, 5000, 10000] ms, last = above)
anchor UW-006 · SUB-1006 · run-7f3a · trace 7540bd55d4e55ccb8f04f3d81880717a · arrived 14:07:12 · 7216 ms · request_info · decided request_info at 14:45:19
audit chain: every hash matches (93 events)
wrote out/ (96 trace files, logs.jsonl, metrics.jsonl, audit_ledger.jsonl, security.jsonl, analyses)
Read the bucket line as a table. 42 of the 48 runs took between 2,500 and 5,000 ms. 6 took between 5,000 and 10,000 ms. None was faster than 2.5 seconds, and none slower than 10.
| Bucket (ms) | up to 100 | 250 | 500 | 1,000 | 2,500 | 5,000 | 10,000 | above |
|---|---|---|---|---|---|---|---|---|
| Runs | 0 | 0 | 0 | 0 | 0 | 42 | 6 | 0 |
The histogram stores what the function Metrics.histogram_summary in telemetry.py returns: count, sum and a count per bucket. The two percentiles in the same line are computed from the raw values, which the example keeps for teaching; a production histogram keeps only the buckets.
def histogram_summary(self, name: str, **labels: Any) -> dict[str, Any]:
"""What a histogram stores: count, sum and a count per bucket. Percentiles are estimated from it later."""
vals = self.values(name, **labels)
return {"name": name, "count": len(vals), "sum": sum(vals), "bounds": list(BUCKETS_MS),
"buckets": bucket_counts(vals), "p50": percentile(vals, 50), "p95": percentile(vals, 95)}
Percentile: the value most runs stay under#
A percentile answers "how slow is slow?". The 95th percentile (p95) of the run durations is the duration that 95 percent of the runs finish within. Half the runs finish within the 50th percentile (p50, the median).
The example computes percentiles by nearest rank: sort the values, and take the one at position "q percent of the count", rounded up. This is the function in telemetry.py:
def percentile(values: list[float], q: float) -> float:
"""Nearest-rank percentile: the smallest value with at least q percent of the values at or below it."""
if not values:
return 0
ordered = sorted(values)
rank = max(1, math.ceil(q / 100 * len(ordered)))
return ordered[rank - 1]
A tiny example first. These are the durations of five steps of run-7f3a, read from its trace: plan_steps 812 ms, search_guidance 240, find_similar_decisions 150, check_eligibility 4, write_recommendation 1,310. Sorted: 4, 150, 240, 812, 1310. For p50 the rank is 50 percent of 5, rounded up: 3, so the third value, 240. For p95 the rank is 4.75, rounded up to 5: the largest value. The snippet below runs the example's function on them; a successful run proves only that the function does what the paragraph says.
from telemetry import percentile
# durations in ms of five steps of run-7f3a, read from its trace
steps = [812, 240, 150, 4, 1310]
print(percentile(steps, 50), percentile(steps, 95))
240 1310
Now the day. With 48 runs, p50 is the 24th value in order (3,107 ms) and p95 is the 46th (7,346 ms). Only the two runs at positions 47 and 48 are slower. The anchor run run-7f3a, at 7,216 ms, is the 44th: one of the five runs that took longer than 7 seconds, all five inside the outage window.
Common confusion. The average is not a percentile. The mean duration on the simulated day is 3,569 ms, above the median of 3,107, because a few slow runs pull it up. The average says nothing about how slow the slowest runs are. A team that watches only the mean sees 3.6 seconds while five quote requests waited more than 7. Watch p50 for the usual case and p95 or p99 for the tail. With 48 runs, p99 is the slowest run itself (7,663 ms): at low volume a high percentile is one run, not a trend.
Before AI. None of this is new. A Java service behind Spring has exported a request counter and a request-duration histogram per endpoint for years, and its dashboards show p50 and p99 latency. The agent changes what is worth counting: tool calls and tool errors by tool, tokens by model, runs by how they ended (ok,fallback,step_limit,denied). The statusfallbackis the new one. A run that ended in a controlled fallback did not fail, and a counter that only knows "success" and "error" hides it.
Estimating a percentile from buckets, and why percentiles do not average
A production histogram does not keep the 48 values, only the eight bucket counts. A percentile is then estimated. The 46th of 48 runs lies in the bucket 5,000 to 10,000 ms, which holds runs 43 to 48. A common estimate assumes the values in a bucket are spread evenly: 5,000 + (46 − 42) / 6 × 5,000 ≈ 8,333 ms. The exact p95 is 7,346. The error comes from the bucket width, not from the arithmetic. Two consequences follow. Bucket bounds should be dense where decisions are made: if the latency objective is 7 seconds (as on SLOs and alerts), a bound at 7,000 ms answers "how many runs missed it" exactly, while the example's bounds cannot. And bounds must be the same on every machine, or their buckets cannot be added.
Adding is the second point. Percentiles do not combine. The p95 of worker A and the p95 of worker B do not give the p95 of both, and the average of two p95 values is not a percentile of anything. Bucket counts with the same bounds do add: sum them per bucket, then estimate the percentile from the sum. That is why metrics systems ship buckets and compute percentiles at query time.
Both numbers here (8,333 and 7,346) are illustrative arithmetic over the synthetic day, not a property of any metrics product.
Counters in practice: resets, rates and what the example simplifies
The example writes every increment as its own point (812 points on the day) so that a reader can see each measurement. A metrics library in a real process keeps one running total per label set in memory and reports it at intervals. Three consequences matter. A process restart sets the in-memory total back to zero, so a query that computes a rate must treat a drop as a reset, not as negative traffic. A rate is always over a window ("errors per minute over the last 5 minutes"), and a short window on low volume is noisy: at 48 runs a day, one bad run moves a burn rate a lot, as SLOs and alerts shows. And a counter that is only incremented on failure has no value at all until the first failure; dashboards that divide errors by calls need both counters to exist from the start.
OpenTelemetry's metrics documentation, read today, describes the same instruments in its own words: a Counter is "a value that accumulates over time", an UpDownCounter one that "can also go down again", a Gauge "measures a current value at the time it is read", and a Histogram is "a client-side aggregation of values, such as request latencies". Its names are covered on OpenTelemetry.
3. Traces: where in this one run#
The broker's question was about one run: why did SUB-1006 take seven seconds? The histogram says that run-7f3a was among the slow ones. It cannot say why, because it threw the run away and kept only a count in the 5,000 to 10,000 ms bucket.
A trace is the connected record of one run across services: a tree of steps, each with a start, a length, a status and the step that called it. Each step is a span. For run-7f3a the trace has 14 spans across the gateway, the queue and the worker. Two of them answer the broker's question at once: get_roof_assessment attempt 1, 2,000 ms, error; and attempt 2, 2,000 ms, error, with 500 ms of backoff between them. Four and a half of the seven seconds were spent waiting for a provider that never answered.
A log line could not say that, because it is a point in time with no length and no parent. A metric could not say it, because it has no run. The trace is the only record that keeps the run whole. How spans are built, how the trace id crosses the queue, and how to read the waterfall is the subject of Tracing.
In our insurance agent. CaseUW-008is the strongest argument for traces. The agent reached the right recommendation without callingsearch_guidance. Its metrics are normal, its output matches a careful run, and its logs lack one INFO line that nobody would miss. Only the trace shows the missing step.
4. Events: a fact with a time#
An event is a fact with a timestamp: something that happened, at a moment, that matters on its own. That sounds like a log line, and the overlap is real. OpenTelemetry's logs documentation, read today, says "events are a special type of logs" and "Not all logs are events, but all events are logs." What makes a record an event is that it has a name and a meaning a program can rely on, not a sentence for a person. The word alone is too vague, so this site always says which kind.
A span event is a fact inside one step of a trace. The first get_roof_assessment span of run-7f3a carries one:
{"name": "retry_scheduled", "at_ms": 2994, "attributes": {"backoff_ms": 500, "next_attempt_number": 2}}
It says when, inside the span, the worker decided to try again, and how long it would wait. On the simulated day there are 5 of them, one in each run that hit the outage. The span event lives and dies with its trace. Tracing covers it.
A business event is a fact the business cares about, independent of how the software works: a recommendation was recorded for the underwriter; the underwriter decided. Other systems may react to it (a notification, a report, a billing count).
An audit event is a business event with a promise attached: it is kept complete, in order, unaltered and for years, because someone will have to prove it later. The example writes its two business events as audit events in the audit ledger. This is the answer to the compliance team, seq 75 of the ledger (hashes left out):
{"seq": 75, "ts": "2026-09-14T14:45:19.256Z", "type": "human_decision_recorded", "actor": "underwriter", "actor_type": "human", "run_id": "run-7f3a", "submission_id": "SUB-1006", "trace_id": "1fb8aad7c68d8c80d7244e4d8780b70e", "span_id": "9d96c98c958d5657", "linked_trace_id": "7540bd55d4e55ccb8f04f3d81880717a", "policy_version": "2026.2", "recommendation": "request_info", "human_decision": {"decided_by": "underwriter", "decision": "request_info", "override": false, "reason": "Agrees: ask the broker for an inspection report, because the roof-data provider did not answer."}}
It names a role, never a person; the decision; whether it overrode the recommendation (no); and the reason. The cockpit also wrote a diagnostic log line saying the same thing. The two look alike and carry different promises: a diagnostic record may be dropped; an audit event may not. Why, and how the ledger proves it was not altered, is the subject of Audit logs.
| Kind of event | Lives inside | Example | Kept as long as |
|---|---|---|---|
| Span event | One span of a trace | retry_scheduled, backoff 500 ms | The trace (14 days in the example's illustrative retention) |
| Business event | The business process | "recommendation recorded", "decision made" | Whatever the consumer needs |
| Audit event | The audit ledger | seq 75, human_decision_recorded | Years (2,555 days, illustrative) |
Common confusion. OpenTelemetry's signals page, read today, lists traces, metrics, logs and baggage as the signals it supports, and describes events as a kind of log, still in development there. Baggage is not a record of what happened: it is a set of values carried along with a request; Identifiers comes back to it. This site's "four signals" is a teaching split by question, not OpenTelemetry's list.
5. What each record costs#
Every record is written, shipped, stored, indexed and kept. The four kinds grow with different things, and that, more than anything else, decides which to use.
The example writes the whole simulated day to files. Their sizes are a fair picture of the proportions, not of a real system: the files are uncompressed JSON, and real stores compress and index them differently.
Record (file in out/) | Records on the day | Size | Grows with |
|---|---|---|---|
Traces (traces/, 94 files) | 94 traces | 693,559 bytes | Runs × steps per run × attributes and captured content |
Logs (logs.jsonl) | 661 lines | 249,842 bytes | Runs × lines per run |
Metrics (metrics.jsonl) | 812 points | 114,209 bytes | In the example, every measurement; in production, label sets × time windows |
Audit events (audit_ledger.jsonl) | 93 events | 60,207 bytes | Business facts only (two per run) |
Read the metric row with care. The example keeps every point so that the reader can see them. A metrics store keeps one value per label combination per time window, however many runs there were. run_duration_ms has 4 label combinations (one per run status) and 8 buckets: 32 numbers per window, whether the day had 48 runs or 48 million. That is why a metric is the cheapest signal at scale. It is also why it cannot name a run: it forgot the runs to get cheap.
The trace row grows the other way. One trace of run-7f3a alone is 14 spans and about 10.6 KB, more than its 15 log lines (about 5.7 KB), because each span carries attributes and the content of its step. Traces are the most expensive record per run, and the one most often sampled: keep a share of them, plus every one with an error. A trace that was never stored answers nothing, which is why the audit question must never depend on one.
The two costs that hurt are not the obvious ones. Putting a label with many values on a metric, such as run_id, turns 32 numbers per window into thousands: the example's Advanced page multiplies a metric from 8 stored counters to 1,568 by adding labels. And capturing whole prompts on spans makes traces both larger and sensitive, the subject of Sensitive data.
| Record | Can it be sampled? | Kept (illustrative, from the example) | Cheapest when |
|---|---|---|---|
| Metric | No need: it is already a summary | Not set in the example | Labels have few values |
| Log line | Yes, with care (counts from sampled logs are wrong) | 30 days | Lines are few per run and carry ids |
| Trace | Yes: head and tail sampling | 14 days | Most healthy traces are dropped, errors kept |
| Audit event | Never | 2,555 days (seven years) | Only business facts are written |
Production pitfall. Counting from logs. The first answer of the morning, "10 ERROR lines of category tool", was correct only by luck. If the platform samples diagnostic logs, or a retry loop logs each failure twice, the count drifts and nobody notices. Counts belong in counters, incremented in the same place the failure is handled, as agent_sim.py:_fetch_roof does.
A cost model for the four signals, and the trade-offs at scale
Write the cost of each signal as a product of what it grows with. Metrics: (label combinations) × (resolution: windows per day) × (retention). They do not grow with traffic, which is their strength, and they grow multiplicatively with labels, which is their failure mode (Advanced has the arithmetic). Logs: (runs) × (lines per run) × (bytes per line) × (retention), with the index adding a share on top for each indexed field (Backends compares index-heavy and label-only stores). Traces: (runs) × (sampling rate) × (spans per run) × (bytes per span) × (retention). For an agent the bytes per span are the variable to watch: a model span that captures its prompt can be tens of times larger than a tool span that does not. Audit events: (business facts) × (bytes) × (years); small per day, never sampled, never shortened, so their cost is driven by retention and by the integrity mechanism.
The trade-offs follow. To cut cost without losing answers, reduce in this order: high-cardinality metric labels (move them to traces or logs), DEBUG lines and per-iteration lines (summarize into a metric), healthy traces (sample, keep errors and slow ones), captured content (capture by decision, not by default). Never reduce audit events: if they cost too much, the business is recording facts that are not business facts.
A second, quieter trade-off is joinability. Every signal is cheaper without identifiers, and every signal is useless in an incident without them. Metrics stay id-free on purpose; logs, traces and audit events must carry the run's ids, and Identifiers is about which ones.
6. Which record to reach for#
Put the pieces together and a working rule appears. Reach for the cheapest record that can answer the question, and move to the next one only when it cannot.
- Is something wrong, and how badly? A metric. It is cheap enough to compute every minute and to alert on. On the simulated day, the counter
tool_errors_totalgoes from 0 to 10 in the 14:00 hour, and the p95 ofrun_duration_msfor that hour is 7,663 ms, against about 3.2 seconds in the hours on either side. - Where, in one affected run? A trace. Pick one slow run from the window (
run-7f3a) and read its waterfall: two 2,000 ms timeouts. - What exactly did the step say? The log lines of that run, by
run_id: the two ERROR lines, then the WARN of the controlled fallback. - What does the business have to show? The audit event: the recommendation at seq 30 and the underwriter's decision at seq 75.
This order is also the order of an incident on Incidents, and it works only if every record carries the same identifiers, so that the engineer can step from the metric's time window to a run, from the run to its trace, and from the trace to its log lines and audit events.
Diagram: From the cheapest record to the one that must be kept
Read left to right. The counter shows the failures of the 14:00 hour without naming a run. One affected run is opened as a trace, which shows where its time went. The log lines of that run say exactly what each step reported. The audit events record what the business must be able to show. Each step works only because the next record carries the same run id.
- 1 · Metric
- Something is wrong: A counter jumps from 0 to 10 in one hour; the run duration p95 rises with it.
- 2 · Trace
- Where, in one run: The waterfall of one slow run shows which steps took the time.
- 3 · Log lines
- What each step said: The run's own lines, in time order, from every service.
- 4 · Audit events
- What must be proved: The recommendation stored for the underwriter, and the underwriter's decision.
The lab below holds the four questions about the anchor run. Pick a question, then open a record: it shows whether that record can answer, and why the others cannot.
| Question | Record that answers it | Value in the example |
|---|---|---|
Why did run-7f3a take 7,216 ms? | its trace | two get_roof_assessment attempts of 2,000 ms each, 500 ms backoff between them |
| How many roof lookups failed today? | metric tool_errors_total | 10 failed lookups of 54 (5 runs x 2 attempts) |
| What did the agent-worker report at 14:07:14? | log line | ERROR get_roof_assessment failed: timeout after 2000 ms, attempt_number 1 |
| Who decided, and was it an override? | audit event human_decision_recorded | seq 75, decided_by: underwriter, request_info, override false, 14:45:19 |
Try the wrong pairings on purpose. The metric cannot say who decided: it has no run. The log line cannot say how long the run took: it is one moment. The trace of the run cannot say who decided either: the decision is a separate request, 38 minutes later, in a second trace.
What this is not. A metric here is a measurement of the running system: counts, durations, tokens. Whether a recommendation was acceptable is a judgment, and the scores, graders and rubrics that make it belong to eval.sammuthu.com. This site carries such labels as telemetry next to latency on Quality signals; it does not define them.
The same idea for a coding agent. The coding-agent sessionCA-001sorts the same way: a counter of failed test runs across sessions, a histogram of session durations, one trace of 22.5 seconds in whichrun_testsfails and then passes, and an event when the pull request is opened. How a coding agent's harness records its steps is on harness.sammuthu.com.
Checklist#
- Every count a dashboard shows comes from a counter incremented where the event is handled, not from counting log lines.
- Durations are recorded as histograms with the same bucket bounds on every machine, and a bound sits at each latency objective.
- Dashboards show p50 and p95 (or p99), not the mean alone, and say how many runs a percentile rests on.
- Run status has more values than success and error: a controlled fallback is counted as its own status.
- No metric label has unbounded values: no
run_id,trace_id,submission_idor user on a metric. - Every log line, span and audit event carries the run's identifiers, so a metric's time window leads to runs and their records.
- Traces can be sampled; nothing the business must prove depends on a trace or a diagnostic log line.
- Each business fact that must be proved later (a recommendation stored, a decision made) is written as an audit event.
In one breath#
A log line says what happened at one moment; a metric (a counter, a gauge or a histogram, read through percentiles) says how often, how much and how long across many runs; a trace keeps one run whole, step by step; an event is a named fact with a time, and an audit event is the one the business must keep. Metrics are cheap because they forget the runs, traces are expensive because they keep them, and audit events are never dropped. On the simulated day, tool_errors_total says 10 lookups failed, the trace of run-7f3a says where its 7,216 ms went, and seq 75 says the underwriter agreed with the request for more information.
Where this leads#
You can now choose the record that answers a question, and say what it costs. The rule "move from the metric to the trace to the log line to the audit event" hides a hard part: each step needs an identifier that the next record also carries, and the records do not share one. The trace of run-7f3a does not contain the underwriter's decision, and the submission SUB-1004 has two runs. Which identifier joins which records, and what goes wrong when two of them are mixed up, is the subject of Identifiers.