AI agent observability is becoming causal evidence

Most trace viewers are excellent at showing order. A model generated a tool call, the tool returned JSON, another model produced a final answer. That sequence can still leave the root cause unresolved. The wrong tenant may have been selected before the trace began. A stale policy cache may have approved the call. The tool may have committed a change after the client timed out. The assistant may then have announced failure even though the database changed.

Causal evidence adds the missing links. It identifies the runtime and release, preserves the task and delegation context, records normalized arguments, connects the policy decision, captures the execution receipt and checks the authoritative state. Each link answers a different question. Combining them is what lets an investigator distinguish model error, orchestration error, authorization failure, downstream ambiguity and misleading presentation.

The practical test is simple: can a second engineer, who did not build the agent, explain the failure and verify the final state from the retained record? If the answer requires guessing from prompt text or rerunning production traffic, the observability contract is incomplete.

W3C Trace Context gives HTTP services a portable traceparent and tracestate format. Agent workflows often travel through places where a simple parent-child request chain breaks down: message queues, scheduled work, parallel tools, resumable sessions, human approvals and subagents that continue after the initiating process exits.

Use parent-child relationships when one operation directly contains another. Use span links when a consumer processes a queued message later, when several upstream events contribute to one action, or when a retry belongs to the same logical task but not the same call stack. Keep a separate stable task or workflow identifier so trace rotation and long retention do not erase the business unit of work.

Baggage can carry correlation values, but it travels through network headers and lacks built-in integrity protection. Treat it as untrusted metadata. Tenant, principal and authorization state should come from authenticated context or signed evidence, then be recorded as protected attributes rather than accepted because a caller placed them in baggage.

Useful model telemetry includes provider, requested model, returned model, endpoint, latency, token usage, finish reason, tool schema version, output type, retry count and a reference to the exact prompt template and application release. Those fields explain many regressions without storing every user message or internal reasoning token.

Prompt and response content may be necessary for selected investigations or evaluation datasets. Capture should be explicit, purpose-bound and redacted before export. A content hash or protected object reference can prove which artifact was used while keeping the full payload in a more restricted store. Audio, images and large documents deserve the same separation because embedding them in span attributes creates cost, privacy and size problems.

Hidden chain-of-thought is not an observability requirement. It may be unavailable, unreliable as an explanation, or intentionally protected by the provider. Diagnose from externally visible inputs, model outputs, tool proposals, constraints and outcomes. That record is easier to standardize and closer to what the system actually executed.

A model first proposes an action. Deterministic code parses and normalizes its arguments. A policy or approval service decides whether the action may proceed. The tool adapter then executes against an external system. Compressing those stages into one span called tool_call hides where responsibility changed.

Record the proposed arguments and the normalized request separately. Attach tool-schema, policy and adapter versions. The authorization decision should have its own identifier, reason category and prerequisite state. Execution should record idempotency key, request digest, downstream correlation ID, transport result and commit receipt. Credentials and secret headers stay out of the trace.

Finally, reconcile the postcondition. An HTTP 200 can contain a rejected business operation. A timeout can occur after a successful commit. A queue acknowledgment may precede execution by minutes. Read the source of truth and attach what exists now: order ID, record revision, payment state, permission set or explicit absence.

A retrieval span that says vector_search returned five documents is rarely enough. Investigators need the data source, corpus or index revision, query after transformation, filters, embedding or reranker version, result identifiers and scores. Full document text can remain in the governed source system and be linked by stable references.

Memory creates a harder problem because it changes what later runs see. A write should identify the previous revision, proposed content, validation result, new revision and retention class. A read should identify which revision influenced the run. Without those links, a bad answer may look like a model regression when the real cause was a stale or poisoned memory entry created days earlier.

The same principle applies to prompt templates, tool schemas, policies and feature flags. Observability becomes far more useful when every mutable input is a versioned artifact rather than an unstructured string copied into a trace.

Head sampling decides before the request outcome is known. It is cheap and predictable, but it can discard the one run that later produces a policy denial, expensive retry storm or damaging side effect. Tail sampling waits for more of the trace and can retain errors, latency outliers or matching attributes. It also needs buffering, consistent routing and enough memory to assemble traces.

Agent workflows add business-aware conditions. Preserve every consequential write, human escalation, policy denial, evaluator failure and ambiguous postcondition. Sample ordinary successful reads more aggressively. Keep a compact all-traffic metric and event layer so low trace retention does not erase volume, latency or error-rate trends.

Retention should follow purpose. Developers may need detailed content for days, security teams may need protected decision evidence longer, and user-deletion obligations may require targeted removal. One universal retention period usually grants too much access to some data and destroys other evidence too early.

Traces answer what happened. Evaluations estimate whether the answer, retrieval, tool choice or outcome met a criterion. The evaluation must point to the exact trace and span, application version, evaluator version, rubric and judge model. Otherwise a score cannot be reproduced after the system changes.

Start with deterministic checks where possible: schema validity, allowed tool, argument constraints, citation presence, postcondition match and policy outcome. Use model-based evaluators for qualities that require judgment, then calibrate them against human labels and monitor disagreement. A judge explanation is diagnostic material, not unquestioned truth.

Reviewed production failures can become datasets. Preserve the original evidence, create a minimized test case and run candidate releases against the same acceptance criteria. This closes the loop from incident to regression test without treating all raw traffic as training data.

Prompt capture, retrieved content and tool arguments can recreate customer records, secrets and internal workflows even when the production application stores little. Auto-instrumentation defaults should be reviewed line by line. A framework may capture model or tool inputs that the central platform team never intended to export.

Redact near the source or at a trusted collector before data crosses organizational boundaries. Use allowlists for attributes, size limits for payloads and cardinality budgets for user-controlled values. Encrypt transport and storage, isolate tenants, restrict export, log access and separate broad developer traces from security or compliance evidence.

Integrity matters as much as confidentiality. Attackers can inject misleading attributes, choose high-cardinality IDs, force trace volume or tamper with mutable dashboards. Protected action evidence should carry digests, trusted timestamps or append-only controls appropriate to the risk. Do not let the agent declare its own success, severity or evaluation score.

OpenTelemetry provides the strongest common transport, context and signal model available for this work. Its GenAI conventions are still developing and have already moved into a dedicated repository. Provider and framework integrations do not all emit the same operations, attributes or content structures.

Choose a small internal contract for the fields that carry operational meaning: workflow, task, agent, release, model, prompt version, tool, normalized action, principal reference, decision ID, state revision, postcondition and evaluation. Map incoming framework spans into that contract at the collector or adapter layer. Preserve original attributes where useful, but do not make dashboards depend directly on unstable experimental names.

Version the mapping and test it with golden traces. A library upgrade that silently renames a span or stops capturing a tool result should fail an observability contract test before deployment. Export to more than one backend when portability or regulated retention justifies the cost.

A useful incident view starts from the user-visible symptom or external record, not from a model span. Operators need to search by order, case, task, tenant, downstream receipt or evaluation failure and then pivot into the trace. That requires indexed correlation fields chosen from real support and security workflows.

The evidence should show the earliest divergent state, not only the final exception. Compare the failing run with a known-good run of the same workflow and release. Highlight version differences, missing spans, altered retrieval sources, policy decisions, retries and postconditions. Root cause is often a combination: a model proposed an unusual argument, validation accepted it, policy lacked a condition and the adapter retried after an ambiguous timeout.

Measure investigator performance. Give engineers synthetic and historical incidents, then record whether they identify the correct component, authority path and final state within the target time. Observability is an operational capability, not a count of spans emitted.