The LLM KV cache is becoming a tiered distributed storage system

The key and value tensors produced by attention summarize how every processed token participates in future decoding. Reusing them avoids repeating the prompt phase, which can be the dominant cost for long contexts.

Once state is shared across requests, workers or time, it acquires storage-system properties: identity, location, ownership, lifetime, replication, transfer, eviction and failure semantics.

The correct abstraction is therefore not merely “memory saved.” It is a distributed state object whose use must preserve the exact computation that fresh prefill would have produced.

Matching visible prompt text is insufficient. Tokenizer revision, special tokens, chat template, model checkpoint, adapter, multimodal processor, attention implementation, positional encoding, KV precision and block layout can all change the produced tensors.

Robust systems hash canonical token blocks together with parent-block identity and additional model inputs. Distributed systems also need connector, tensor-parallel and pipeline-parallel compatibility metadata.

Identity should be immutable and content-addressed. When an input cannot be represented safely in the key, reuse should be disabled rather than approximated.

Prefix reuse preserves causal position: cached tokens appear in the same order at the beginning of a later request. That makes it straightforward to skip already-computed attention work.

RAG and modular prompts may reuse chunks that appear in different positions or combinations. Systems such as CacheBlend selectively recompute portions so cached knowledge can be fused with new context.

Broader reuse can create larger savings, but it adds positional, attention and quality assumptions. It needs task-level equivalence tests and must not be treated as an ordinary exact hit.

GPU memory offers the lowest access latency but competes directly with weights, active sequences, workspaces and batch capacity. Host DRAM offers a larger nearby tier, while NVMe and remote stores trade capacity for transfer and metadata cost.

Write-through makes lower-tier durability and reuse predictable but increases write traffic. Write-back reduces immediate traffic but risks losing unflushed state. Selective policies persist only high-value blocks.

Promotion, demotion and eviction should use measured reuse value, object size, fetch cost, expiry, tenant quota and topology—not a single global least-recently-used list.

Routing a request to the worker with the largest cached prefix can save prefill, but repeated popular prefixes may overload one worker while others sit idle.

A useful router estimates both saved computation and current cost: active KV blocks, queue depth, expected decode work, topology, transfer options and service-level risk.

Cache events can provide a more accurate global index, while approximate routing reduces control-plane traffic but can become stale after eviction, failure or scaling. The chosen consistency model must be visible in metrics.

Prefill is often compute-heavy, while decode is dominated by memory bandwidth and repeated cache reads. Splitting them allows independent scaling and can protect decode latency from long prompts.

The decode worker cannot continue until compatible state is available. Transfer therefore becomes part of time to first token and depends on allocation, descriptors, topology, transport, layout conversion and synchronization.

Disaggregation is not automatically a throughput improvement. For short prefixes, low concurrency or weak networks, local recomputation can be simpler and faster. A production router should support a qualified local path.

A sender and receiver may use different tensor-parallel sizes, cache page layouts, attention types, precisions or layer groupings. Moving bytes successfully does not prove that the destination interprets them correctly.

Connectors need a handshake that covers model identity, ranks, layouts, block mapping, memory descriptors and supported transfer operations. Digests and completion signals should be checked before blocks become visible to attention.

Layer-wise transfer can overlap movement with execution, but partial visibility complicates cancellation and recovery. Failed requests must not leave reusable half-written state.

Quantization, token eviction, head-specific budgets and encoded transfer streams can reduce memory and bandwidth. They also alter the state consumed by attention or add decode work on the critical path.

Methods that perform well on retrieval or long-context benchmarks may fail on code, exact copying, multilingual prompts, adversarial needles or a different model family. Worst-case analysis also shows that not every cache is safely compressible.

Every codec or retention policy should be versioned, evaluated by task and reversible. Full-precision or fresh-prefill fallback remains necessary for unsupported models and quality alarms.

KV tensors are derived from prompt content. Even when they cannot be easily decoded into text, their presence, metadata, reuse timing and ownership can reveal that another request processed a particular prefix.

Tenant-specific namespaces or salts prevent unauthorized cross-tenant hits. Access control must apply to lookup, transfer, persistent storage and observability, not only to the original inference API.

Retention and deletion requests must remove every replica and index entry. Logs should record object identifiers and reason codes without retaining raw prompt material unnecessarily.

Cache stores, event planes, routers and transfer paths can fail independently. Metadata may say a block exists after it was evicted, or a partial object may remain after a node or link failure.

State should move through explicit pending, committed, expired and invalid states. Readers must not observe an object until its integrity and compatibility are confirmed.

When uncertainty exists, invalidate and recompute. This costs latency but preserves model meaning. Recovery tests should include stale events, index loss, split-brain metadata, partial writes and restart during transfer.

A cache can report a high hit rate while increasing tail latency because objects reside on slow tiers or affinity routing overloads workers. It can also save many tokens but few milliseconds when prefixes are short.

Useful telemetry joins lookup result, matched tokens, tier, bytes, transfer path, materialization time, queue delay, recomputation avoided, output acceptance and tenant authorization.

Metrics should expose metadata lag, duplicate objects, prefetch waste, eviction churn and invalidation backlog. Without those signals, storage and routing failures appear as unexplained inference latency.

Start with deterministic engine-local prefix caching and measure the actual reuse distribution. This establishes the identity model and a reliable no-network baseline.

Add one host tier and explicit quotas before introducing remote storage. Then add cache-aware routing with observable cache events. Only after those controls are stable should the system attempt disaggregated prefill, persistent remote stores or approximate cache blending.

Each new tier or algorithm should have a narrow workload target, canary pool, rollback switch and break-even threshold. More cache infrastructure is not inherently better than recomputation.