AI inference is becoming a distributed systems problem

The prompt phase processes many input tokens in parallel and builds the key-value cache. It tends to benefit from large matrix operations and becomes expensive with long contexts. The decode phase usually produces one token per active sequence at each step and repeatedly reads model weights and cached attention state. It is often constrained by memory bandwidth and concurrency rather than the same compute pattern as prefill.

When both phases share workers, a long prefill can interrupt ongoing token generation. Continuous batching and chunked prefill reduce that interference. At larger scale, disaggregated serving places prefill and decode in separate pools so they can use different parallelism and scale independently.

Separation is not automatically faster. The KV cache must move or become accessible to decode workers, and that transfer can dominate the first-token path. Small models, short prompts and low concurrency may remain better on a simpler aggregated server.

Autoregressive generation avoids recomputing earlier attention keys and values by retaining them for every layer. The cache expands with prompt and generated tokens, so long contexts and high concurrency consume accelerator memory even when model weights are unchanged.

PagedAttention and related block managers allocate KV memory in smaller units rather than reserving one maximum-length tensor per request. This reduces fragmentation, allows blocks to be recycled and supports sharing or offloading. The scheduler still needs a policy for reservation, eviction and pausing when demand exceeds the pool.

Cache quantization, host offload and remote cache tiers can increase apparent capacity, but each introduces conversion, transfer or quality considerations. Capacity planning should use active-token distributions and eviction behavior, not only the advertised maximum context window.

Static batches wait for a group of requests and keep them together until completion. Continuous or in-flight batching can admit and remove sequences at iteration boundaries, filling capacity as requests arrive and finish. This usually improves utilization for mixed-length traffic.

The scheduler decides which prefills and decodes run, how many tokens enter a step, whether a request pauses and which priority class receives capacity. Larger batches improve arithmetic efficiency until queueing, KV pressure or long-step interference damages latency.

Chunked prefill splits long prompts so decode work can continue between chunks. That improves fairness, but the chunk size is a policy knob: too large recreates interference; too small adds scheduling and kernel overhead.

System instructions, retrieved documents and multi-turn histories often repeat long prefixes. Prefix caching can reuse previously computed KV blocks and reduce prompt work. The benefit depends on exact tokenization, block boundaries, cache lifetime and whether a request reaches a worker holding the prefix.

Round-robin routing ignores that state. Cache-only routing can overload the worker with the best match. Production routers therefore combine estimated cache overlap with active prefill and decode load, and they need current cache events or a conservative approximation.

Shared caches also create isolation questions. Hash collisions, weak tenant partitioning or accidental reuse across incompatible adapters and model versions can produce incorrect output or information exposure. The cache key must cover every factor that changes the hidden state.

A draft model, multi-token prediction head, n-gram matcher or other proposer can suggest several next tokens. The target model verifies them in parallel and accepts a valid prefix. Correct implementations preserve the target distribution within numerical limits while reducing the number of serial decode steps.

The speedup depends on acceptance rate, draft cost, verification efficiency, batch size, request load and sampling settings. A proposer that works well at low QPS may consume useful accelerator capacity during peak traffic. Longer speculative windows can waste more work when acceptance falls.

Track accepted tokens per verification step and compare full-server behavior, not isolated single-request latency. Maintain output-quality and determinism tests because numerical behavior, batching and implementation compatibility can still alter observed responses.

Tensor parallelism splits operations across accelerators, pipeline parallelism divides layers, expert parallelism routes sparse mixture-of-experts computation and data parallelism replicates serving groups. Each method solves a different capacity problem and adds collective communication, synchronization or placement requirements.

Prefill-decode disaggregation specializes phases and can prevent long prompts from disturbing active generation. It also requires high-rate KV transfer, compatible cache formats and topology-aware placement. Network oversubscription or cross-zone routing can erase the benefit.

Use the smallest distribution topology that meets memory and SLO requirements. A configuration optimized for one prompt/output ratio may be poor when product behavior changes.

GPU replicas take time to provision, load weights, compile kernels and warm caches. Request rate alone is a weak scaling signal because one long prompt can cost more than many short requests. Active KV blocks, pending input tokens, decode sequences and projected service time provide stronger signals.

Admission control should protect the service contract by limiting work, assigning priority, enforcing token budgets or routing to a smaller model when appropriate. Unlimited queues preserve request counts while destroying tail latency and increasing cancellation waste.

Scale prefill and decode independently only when the topology is truly disaggregated. Keep warm capacity for bursts whose duration is shorter than instance startup, and test failure behavior when a worker disappears with active cache state.

Serving dashboards should separate queue, prefill, transfer, decode and post-processing. Track TTFT, inter-token latency and completion at multiple percentiles, broken down by model, input length, output length, tenant, priority and cache state.

Tokens per second is useful for hardware saturation but insufficient for product decisions. Goodput, SLO attainment, cancellations, failed requests and accepted-task cost show whether the system converts accelerator time into usable work.

Optimization changes need quality assurance. Quantization, kernel changes, speculative methods, cache formats and parallelism can affect numerical behavior or supported features. Promote changes through trace replay, quality tests, load tests, canaries and rapid rollback.