LLM observability is becoming an online evaluation and incident-response control plane

A conventional application trace often assumes that one HTTP request maps cleanly to one service transaction. Generative systems break that assumption. One user turn can trigger routing, multiple model calls, retrieval, reranking, tool execution, handoffs, retries and background work. The telemetry model needs an end-to-end trace plus stable session, workflow, deployment and component identities.

The identity envelope must distinguish model identity from provider endpoint, prompt template from rendered prompt, tool definition from tool invocation and evaluator definition from evaluator run. Without those distinctions, a dashboard can show that quality changed but cannot identify whether the cause was a model revision, a retrieval index, a routing policy or an evaluator update.

Trace and span IDs establish causality. Release, prompt, model, dataset and policy versions establish interpretability. Both are required before an evaluation score can become release evidence.

OpenTelemetry GenAI conventions and OpenInference provide shared names for model operations, token usage, agents, tools, retrieval and related attributes. This makes it easier to send one instrumentation stream to different backends and to compare applications built with different frameworks.

Standard fields should be treated as a transport contract, not as permission to collect every available value. A convention may define how to represent prompts or responses while the organization still prohibits exporting them. The telemetry gateway therefore needs a policy layer that can allow metadata, deny content, hash identifiers or route sensitive spans to a separate store.

Convention versions also evolve. Producers should record schema version, exporters should reject incompatible fields safely, and downstream queries should avoid assuming that an experimental attribute will remain stable.

Latency, status, token count, model name and tool type are operational telemetry. Prompts, retrieved passages, tool arguments and responses are content. Mixing both into one unrestricted trace store creates unnecessary privacy, security and retention exposure.

Content capture should be opt-in by environment, workload and purpose. Production may retain metadata for all traffic while storing encrypted payload references only for sampled, consented or incident-relevant traces. Redaction should occur before export because downstream masking cannot undo exposure to collectors, queues or third-party backends.

Hashing is not automatically anonymization. Low-entropy identifiers, repeated prompts and rare retrieved text can remain re-identifiable. The policy should specify which fields are removed, tokenized, transformed, referenced or retained in a protected enclave.

Capturing and evaluating every trace is usually too expensive and too invasive. Uniform random sampling provides an unbiased control group but can miss rare, high-impact paths. Triggered sampling catches errors, slow requests, expensive runs, explicit user complaints and policy-sensitive tools but can distort population estimates.

A production design combines both. Maintain a stable random sample for trend estimation, stratified quotas for languages and customer segments, and risk triggers for important events. Store sampling probability and rule version with each trace so aggregate metrics can be reweighted correctly.

Sampling policy is itself a production model. It can drift, be gamed by attackers or silently stop covering a newly introduced workflow. Coverage dashboards should compare evaluated volume with traffic composition and target quotas.

Many apparent AI-quality questions are ordinary software assertions. JSON validity, citation existence, tool success, allowed destination, numeric reconciliation, forbidden phrase presence and policy-state transitions should be checked with code whenever possible. These checks are cheaper, reproducible and easier to audit than model judgments.

Model judges are useful for criteria such as completeness, relevance, grounded explanation, tone or task adherence when the rubric cannot be reduced to a stable rule. They should receive only the fields needed for that criterion and return structured labels, scores and explanations.

A layered evaluator pipeline reduces cost and ambiguity. Deterministic failures can stop further judging, while uncertain or high-impact cases can be escalated to repeated judges, alternative models or humans.

Research shows that model judges can align with human preferences while still exhibiting position bias, verbosity bias, self-preference, domain weakness and sensitivity to rubric wording. A score can also change when the judge model or provider updates even though the evaluated application did not.

Calibration begins with a domain-specific gold set labeled by qualified humans. Evaluate precision, recall, confusion by class, score calibration and disagreement across languages and task types. Randomize answer order in pairwise judging and repeat a subset to estimate variance.

The evaluator record must include judge model revision, prompt, rubric, decoding settings, output schema and threshold. Judge updates should run in shadow mode against the prior evaluator before they influence release gates.

Offline evaluation uses a controlled dataset to compare candidate systems under repeatable inputs. It supports regression testing, release comparison and root-cause experiments. Its weakness is dataset staleness and incomplete representation of production behavior.

Online evaluation samples real interactions and captures current traffic, new failure modes and delayed user outcomes. Its weakness is missing references, selection bias, privacy limits and the risk of evaluating harmful outputs after they reached users.

The two systems should reinforce each other. Production failures become reviewed dataset candidates; offline fixes are tested against historical and synthetic cases; canary traffic verifies that improvements survive real workload distribution.

Immediate evaluators can check form, policy and apparent relevance, but many business outcomes are delayed. A support answer may look complete yet cause a repeat contact. A recommendation may be accepted and later reversed. An agent may finish a workflow whose downstream reconciliation fails hours later.

The observability model should allow append-only outcome annotations linked to the original trace without modifying the execution record. Examples include user feedback, task completion, refund, escalation, correction, abandonment and human audit findings.

Delayed labels change how monitoring is interpreted. Early indicators can support rapid alerts, while mature outcome windows determine whether a release genuinely improved the business or merely pleased the judge.

A quality dashboard is not an incident process. When a statistically and operationally meaningful change occurs, the system should create an incident containing the affected release, segments, evaluator versions, traffic mix, representative traces and likely changed components.

Operators need to separate application failure from evaluator failure. A score drop may come from a real prompt regression, a retrieval outage, a route shift, judge drift, sampling changes or missing telemetry. The incident workflow should test those hypotheses explicitly.

Rollback criteria should be defined before deployment. High-severity safety or policy failures may require immediate rollback from a small number of confirmed cases, while style or helpfulness changes may require larger samples and human adjudication.

Blindly adding every low-scoring trace to a dataset creates duplication, privacy risk and label noise. Promotion should require confirmation that the trace represents a meaningful failure, not an evaluator error, ambiguous request or unsupported use case.

Curators should deduplicate semantically similar examples, preserve important long-tail variants, assign taxonomy and expected behavior, remove prohibited data and document why the case matters. The resulting dataset needs versioning and coverage metrics.

Regression suites should include successes as well as failures. Otherwise a fix can optimize for known edge cases while damaging common behavior. Holdout sets and rotating fresh-production sets reduce overfitting to a public or frequently replayed benchmark.

A candidate can improve a judge score while becoming slower, more expensive or less safe. Release policy should define non-negotiable constraints plus metrics that may trade off within approved bounds. Comparisons need matched datasets, repeated runs where nondeterminism matters and confidence intervals or minimum-effect thresholds.

A global average can hide damage to a language, tenant, tool or high-risk workflow. Gates should include critical segments and maximum-regression rules even when the aggregate improves.

Evaluator and telemetry availability must not become a single point of unsafe approval. When the control plane is degraded, the system should fall back to deterministic checks, freeze risky changes or require manual authorization.

A production trace can generate storage, index, rule, judge and human-review cost. Evaluating every span with a large model can exceed the cost of serving the original request while adding little decision value.

Allocate evaluation depth according to impact and uncertainty. Low-risk common traffic may receive metadata and deterministic checks. Novel, high-value or safety-sensitive traces may receive repeated judges and human review. Cached or batched evaluation can reduce cost but must preserve version and ordering semantics.

The economic metric is not cost per score. It is cost per meaningful degradation detected, incident resolved or unsafe release prevented, balanced against false alarms and engineering time.