LLM serving is becoming a deadline-aware admission-control system

A traditional request-rate autoscaler can assume that requests are broadly comparable. LLM requests are not. A short prompt asking for one token, a 128K-token document analysis, a multimodal request and a multi-branch reasoning job can differ by orders of magnitude in compute, memory and duration.

Admission and scaling decisions therefore need token, phase, cache, adapter and topology estimates. Request count remains useful for business volume, but it is an unsafe sole control signal for accelerator capacity.

The estimate will always be uncertain because output length and tool behavior are not known exactly. Production controls need conservative intervals and an explicit overload margin rather than pretending to predict every request perfectly.

Time to first token is dominated by queueing and prefill. Inter-token latency is dominated by repeated decode scheduling and batch step time. End-to-end completion adds output length, cancellation and stream delivery.

A policy that maximizes TTFT attainment can still produce poor streaming quality if decode steps are repeatedly delayed by new prefills. A decode-first policy can preserve ITL while starving long prompts. The scheduler must know which objective applies to each service class.

Service objectives should be explicit ranges with measurement boundaries: arrival timestamp, admission timestamp, first byte or first token, final accepted token and client cancellation. Otherwise teams compare incompatible latency numbers.

Continuous batching can keep adding work whenever tokens or sequence slots appear available. That improves utilization, but a locally fitting request can still create future KV pressure, extend batch steps or consume the slack needed by active streams.

A production admission controller asks whether starting the request is safe for the complete service horizon, not only whether the next iteration can fit. The answer can be accept, queue, route elsewhere, use a smaller qualified model, reduce optional work or reject with a retry signal.

Failing to reject early converts a clear overload response into hidden queueing, repeated preemption and widespread tail-latency failure.

Iteration-level batching lets completed requests leave and new requests join without waiting for a static batch boundary. It is foundational to modern LLM throughput.

The shared step also creates externalities. Longer contexts can increase attention work, branch expansion can widen the batch, and a prefill chunk can lengthen the step experienced by every decode stream. Local throughput gains are not free when they consume another request’s latency slack.

The scheduler needs a prediction of marginal step cost and a rule for who may consume available slack. Fixed batch-size limits cannot express that decision across changing context lengths and hardware states.

Prefill is compute intensive and can monopolize an iteration when a long prompt is processed as one unit. Shorter requests arriving behind it may miss TTFT objectives even when total accelerator utilization looks good.

Chunked prefill improves interruptibility by dividing the prompt across scheduling steps and interleaving it with decode. Small chunks increase responsiveness but can reduce kernel efficiency and add scheduling overhead; large chunks restore efficiency but recreate blocking.

Newer work explores operator-level or event-driven preemption so responsiveness is not tied to one globally fixed chunk size. The policy still requires replay on the actual model and hardware because the best granularity depends on shapes and load.

A request that fits compute budgets can still exhaust KV capacity as generation continues. When memory is tight, runtimes may pause, evict, swap or recompute requests. Those actions have different latency and bandwidth costs.

Conservative schedulers reserve enough state to complete admitted work without eviction. Aggressive policies pack more requests for utilization but risk pauses when predictions are wrong. Watermarks and headroom reduce oscillation near the capacity boundary.

Admission should include expected KV growth, uncertainty, prefix-sharing behavior and recovery reserve. A policy that repeatedly admits and preempts the same long sequence is usually worse than delaying it once.

Priority scheduling can protect interactive or contractual traffic, but an always-busy high-priority class can prevent lower classes from ever running. First-come-first-served avoids explicit preference while still allowing a few very long requests to dominate.

A governed scheduler needs authenticated classes, quotas, aging, maximum queue residence and tenant isolation. Priority must be assigned by policy, not trusted from an arbitrary client field.

Fairness should be measured in delivered service relative to entitlement and request cost. Counting completed requests can make a tenant with many tiny prompts appear over-served or under-served incorrectly.

When a high-urgency request arrives or an instance becomes overloaded, the system can pause another request, recompute its prefix later, move its KV state or migrate it to a different instance.

These controls can improve tail latency and isolation, but they introduce serialization, network transfer, destination uncertainty and state-consistency obligations. Multi-model systems may additionally need to reload weights or adapters, making preemption cost strongly model dependent.

The decision should compare avoided SLO loss with full intervention cost and collateral delay. A request should carry a migration generation or lease so two workers cannot continue the same state concurrently.

Traditional autoscaling often reacts to CPU utilization or requests per second. LLM latency depends on prompt length, output length, active sequences, KV pressure, phase mix and model topology, so the relationship between load and replicas is not linear.

Aggregated serving needs workload-aware replica estimates. Disaggregated serving needs separate prefill and decode capacity because their resource curves differ. Scaling also has startup, weight-loading and cache-warming delays that can exceed the burst being handled.

Trace-driven simulation and measured forward-pass models can explore configurations before consuming a large cluster. Final decisions still require real hardware verification because kernel, network and cache effects can invalidate simplified models.

A shared service may host several base models, quantizations, adapters and speculative draft models. The scheduler must decide not only when a request runs but where the required weights and state are resident.

Offloading and reload cost are nonlinear across models and hardware. A policy optimized for one model can create severe stalls when another requires weight movement or different parallelism. Adapter-heavy traffic adds its own cache and batching constraints.

Qualification should include the actual model mix, transition frequency and residency policy. A single-model benchmark cannot justify a multi-model admission controller.

An attacker does not need to break model safety to exhaust an inference service. Long prompts, large maximum-output requests, branch expansion, expensive adapters and repeated cancellations can occupy compute, KV memory and queue slots.

Admission must authenticate service classes, enforce token and concurrency quotas, cap speculative or branch work and charge for admitted resource reservations. Cancellation should release resources promptly without allowing cheap churn to destabilize batching.

Per-tenant telemetry needs bounded cardinality and privacy controls. Queue position, cache hits and detailed timing can reveal information about other tenants if exposed carelessly.

Average latency cannot explain why a scheduler succeeded or failed. Useful evidence links arrival, class, predicted work, admission decision, queue age, each batch membership, preemption, cache state, delivered tokens and completion outcome.

A sanitized trace can replay the same workload through alternative policies to compare SLO goodput, fairness and cost before live rollout. The replay must preserve request timing and shape correlations; randomly shuffled requests hide burst and head-of-line behavior.

Live canaries should shadow the candidate policy or limit it to one pool, retain the previous configuration and define automatic rollback thresholds for deadline misses, starvation, oscillation and cost regression.