AI agent runtimes are becoming durable event-sourced workflow systems

A session can preserve prior messages yet still lose the current step, repeat a tool call or forget that an approval was already granted. Durable execution must preserve control state, not only conversational context.

The minimum durable record includes workflow identity, current node, accepted inputs, activity identities, results, pending approvals, timers, external correlations and terminal status. Message history is only one field in that state.

This distinction matters because an apparently fluent resumed conversation can still be operationally wrong. The user may see the right words while the system sends a second refund or abandons an already-created artifact.

Event-sourced and checkpointed runtimes record meaningful transitions rather than relying on a mutable object inside one worker. Current state is produced by folding accepted events or loading a materialized snapshot plus later events.

A useful event envelope carries workflow ID, step ID, event type, sequence or logical time, actor, input and output digests, workflow version, tool or model version, correlation IDs and policy metadata. Payloads should be minimized and encrypted when sensitive.

Event ordering must be explicit. Remote callbacks, push notifications and retries may arrive late or more than once. The runtime needs deduplication, monotonic state transitions and a policy for events that conflict with a terminal state.

Replay re-runs orchestration code to rebuild state. Calls to models, clocks, random generators, network services and mutable databases cannot execute freely inside that replay path because they may return different results.

Durable systems therefore treat model calls and tools as activities. The first execution records an accepted result. Later replay reads that result from history and reaches the same orchestration decision without repeating the activity.

The boundary is architectural, not cosmetic. A helper function that quietly reads current time or fetches remote configuration can make replay diverge. Determinism reviews should cover libraries, callbacks, routing logic and dynamic prompt construction.

Network timeouts are ambiguous. A destination may have committed an action even though the caller did not receive the response. Retrying without identity can create duplicates; assuming failure can lose completed work.

Each side-effecting tool call should receive a durable invocation ID that is also sent to the destination as an idempotency key or business reference. The runtime records requested, dispatched, acknowledged, committed, failed, compensated or uncertain status.

When the destination does not support idempotency, recovery needs reconciliation: query by business key, inspect a ledger, or require human review. Exactly-once execution is usually an application protocol built on at-least-once delivery, not a property supplied by the queue alone.

A human may answer hours after the original worker is gone. The pending request must therefore include the exact action, arguments, resource identity, risk classification, policy version and expiry that were shown for approval.

Resumption verifies the approver and compares the stored action digest with the action about to execute. If the plan changed, credentials expired or target state drifted, the old approval should not silently authorize the new action.

Accept, decline, cancel and timeout are different outcomes. The workflow should model each explicitly, including escalation, re-planning, compensation and user-visible status.

Remote agent work increasingly returns a task identity rather than a synchronous answer. The caller may poll, stream, receive push updates, provide more input, cancel or retrieve a deferred result later.

Local workflow state must map remote states into its own lifecycle without assuming identical semantics. A remote input-required or auth-required state may pause the parent workflow; a late completion after local cancellation may require artifact quarantine or compensation.

Correlation IDs, authenticated callbacks, sequence handling and terminal-state rules become part of the agent contract. Protocol support does not automatically make the underlying execution durable; both sides must persist enough state to honor the lifecycle.

Checkpointing after every small mutation can create excessive storage and contention. Checkpointing only at the end can make long tasks expensive to recover. The useful boundary is usually a stable superstep, node completion or accepted external result.

Checkpoints must be schema-versioned and integrity-checked. A corrupt or incompatible checkpoint should fall back to an earlier snapshot plus event replay rather than becoming a single point of failure.

Compaction can summarize old conversational context while preserving operational events and effect metadata. The system should distinguish model context compaction from audit-history deletion.

A workflow started under one graph, prompt, tool schema or model may resume after several deployments. Replaying old history through changed orchestration can cause nondeterminism or reinterpret stored state incorrectly.

Production systems need a policy: pin active runs to the old worker, route by workflow version, patch only explicitly compatible changes, or migrate state through a tested transformer. Silent best-effort continuation is unsafe.

Model and tool upgrades can also alter behavior even when orchestration code is unchanged. Store version identities with accepted results, and canary resumed runs that cross a version boundary.

A cancel request may arrive while a model call, tool or remote agent is already running. The runtime can mark the workflow cancelled and attempt to stop outstanding work, but it cannot assume the external world rolled back.

Compensation is domain-specific. It may delete a draft, revoke a reservation, send a corrective message or open a manual case. Some actions are irreversible and can only be acknowledged and contained.

Cancellation races need clear precedence. Once a task is terminal, late completion events should not change its public state, though they may update an internal reconciliation record.

Distributed logs can be incomplete, reordered or sampled. A durable workflow needs a canonical execution history plus traces that explain model calls, tools, handoffs, approvals, retries, waits and costs.

Trace IDs should link to workflow, task, step, model request, tool invocation, remote task and artifact identities. The canonical state machine and the observability system should agree on terminal status and effect outcomes.

Operational dashboards should highlight stuck waits, repeated retries, uncertain effects, stale approvals, version-incompatible runs, dead letters and histories approaching retention limits.

An agent may copy the same personal or confidential value into messages, normalized state, model requests, tool arguments, checkpoints, traces, artifacts and remote tasks. Deleting one session row does not remove those derivatives.

Classify fields before persistence. Store secrets by reference, redact model traces where possible, encrypt tenant data and apply purpose-specific retention. Operational effect metadata may need to outlive raw content, but that should be an explicit policy.

Time travel and replay are powerful but can expose historical data or revive actions the user believed were deleted. Authorization should be evaluated for historical inspection and fork operations, not only live execution.

Normal integration tests rarely hit the dangerous windows: after an external commit but before event persistence, during checkpoint replacement, between approval and execution, or while a deployment changes replay logic.

A useful chaos harness can terminate workers at named boundaries, duplicate queue deliveries, delay callbacks, reorder remote updates, corrupt a checkpoint, fail a database leader and force a version migration. The expected effect ledger and terminal state should be asserted after each run.

Evaluation must also preserve task quality. A recovered agent that avoids duplicate effects but loses crucial context or changes its answer is not fully correct.