New scale-up and scale-out architectures are moving collective operations, congestion control, routing intelligence and failure recovery into switches, NICs and interconnect protocols. For large AI systems, the network is no longer plumbing beneath the model; it is part of the runtime.
Why it is movingAI infrastructure is crossing an architectural boundary: the network is starting to perform work that once belonged almost entirely to endpoints and host software. UALink Common 2.0 explicitly adds In-Network Compute for accelerator scale-up; OCP ESUN 1.0 defines lossless Ethernet behavior and compact framing for tightly coupled accelerator domains; Ultra Ethernet reached specification 1.0.3 on July 16, 2026; AMD is building programmable DPUs, UALoE scale-up and 800G-class AI NICs into Helios; NVIDIA’s NVLink 6 offloads collectives inside the scale-up fabric while Spectrum-6 applies adaptive traffic management at scale-out. The common thread is not one protocol winning. It is that collective latency, routing, congestion, resiliency and even numerical reductions are becoming first-order parts of model execution. Peak accelerator FLOPS increasingly describe only one component of delivered AI throughput.
Evidence confidence96%
Benchmark the fabric as part of the model runtime, not as a bandwidth line item29 min read
As frontier and specialized model artifacts grow, startup speed increasingly depends on where weights are cached, how shards are read, how many ranks repeat the same bytes, and whether tensors can move directly from storage or ready peers toward GPU memory.
Why it is movingModel loading is becoming a first-class inference subsystem. NVIDIA Dynamo now documents separate paths for node and shared-storage caching, concurrent model streaming, ModelExpress peer distribution, InstantTensor direct loading and worker snapshots. KServe 0.18 exposes Local Model Cache as an installable serving component and describes node-local NVMe as a way to cut LLM startup time. Google Cloud Storage FUSE recommends file caching plus parallel downloads for large model and checkpoint reads. Recent systems work such as HydraServe and ServerlessLLM treats model placement, fetch parallelism, checkpoint locality and overlapped initialization as explicit cold-start scheduling problems.
Evidence confidence97%
Optimize bytes-to-ready, not download time alone46 min read
Qwen3.5, Nemotron 3 and the Mamba line are making recurrent and linear-attention layers a practical part of large language models. The serving consequence is not simply “less KV cache”: production runtimes must now manage fixed-size recurrent state and sequence-length-dependent attention state together, with different precision, checkpointing, branching, batching and recovery rules.
Why it is movingSeveral current model families no longer use full softmax attention in every layer. Qwen3.5 documents a 3:1 stack of Gated DeltaNet linear-attention layers and full attention. NVIDIA Nemotron 3 Super and Ultra use hybrid Mamba-attention backbones, while Nemotron-H documentation describes Mamba layers with constant computation and constant memory per generated token. Mamba-3 advances the recurrent side further with a more expressive state update and an inference-oriented MIMO formulation. These systems do not erase context state; they change its shape. A hybrid runtime must carry recurrent or linear-attention state for some layers and ordinary KV tensors for others, so memory capacity, prefix reuse, pause/resume, branching and batching depend on the layer mix and state representation rather than token count alone.
Evidence confidence98%
Benchmark the state/KV mix, not context length alone49 min read
Long-running agents do not behave like independent chat requests. They make repeated model calls around tools, tests, browsers and subagents, carrying large reusable prefixes while GPU work pauses between turns. Current NVIDIA Dynamo, vLLM, SGLang and LMCache work is exposing session identity, serving hints, hierarchical cache offload, cache-aware routing and program-level scheduling so inference systems can retain, move, prefetch or discard KV state according to the trajectory rather than generic recency. The opportunity is lower end-to-end agent latency and less repeated prefill. The risk is turning temporary model state into a long-lived, cross-worker resource without clear correctness, fairness, isolation and deletion rules.
Why it is movingAgent-aware serving has moved from research prototypes into current infrastructure interfaces. NVIDIA Dynamo now documents session IDs, agent hints, trace replay, priority scheduling, speculative prefill and an experimental ThunderAgent program scheduler that pauses and resumes at tool boundaries. Its March 2026 agentic-inference analysis shows why generic LRU is a poor fit: repeated system prompts and conversation prefixes have high reuse value, while reasoning tokens and dead subagents often do not. vLLM now exposes native multi-tier KV offloading and selective offload controls. SGLang has active RFCs for agent-aware and programmatic KV cache directives, while HiCache extends reusable state through GPU, host and distributed tiers. Recent systems papers such as ThunderAgent, CacheWise, Leyline, IntentKV and AgentKVShift are converging on the same conclusion: the inference scheduler needs lifecycle information from the agent harness if it is going to manage memory efficiently across long, interrupted trajectories.
Evidence confidence98%
Schedule the trajectory, not isolated requests48 min read
Choosing a model is no longer a static application setting. Production gateways now filter candidates by capability, residency, retention and safety policy; predict quality and cost; account for latency, cache locality and provider health; explore alternatives; and execute traceable fallbacks. The difficult part is proving that each decision remained compatible, calibrated and economically rational after models, prices and workloads changed.
Why it is movingModel routing is converging from two directions. Cloud platforms and gateways expose configurable routers, capability filters, regional and retention constraints, health-aware provider selection, sticky sessions, weighted rollouts and fallbacks. Research systems learn which model or model-and-budget combination is most likely to satisfy a request at acceptable cost. New benchmarks show real model complementarity, but also reveal weak calibration, routing collapse, limited gains over simple baselines and persistent distance from an oracle. The operational shift is from selecting a model name to operating a policy engine whose decisions affect quality, privacy, latency, cache reuse, spend and incident behavior.
Evidence confidence96%
Treat every routing decision as versioned policy with evidence, not an invisible cost heuristic59 min read
Reasoning effort is no longer a cosmetic model setting. Production systems increasingly estimate task difficulty, allocate serial or parallel inference compute, generate candidate trajectories, score them with process or outcome verifiers, stop when marginal value falls and preserve a typed boundary between hidden reasoning, visible explanations and accepted answers.
Why it is movingMajor APIs now expose reasoning effort, thinking levels or token budgets, while open serving runtimes parse reasoning streams separately from final answers. At the same time, test-time-scaling work has moved beyond “let the model think longer” toward adaptive compute allocation, best-of-N sampling, tree or beam search, process and outcome verifiers, budget forcing, early stopping and per-task cost controls. The operational shift is from running one completion with a larger token limit to running a versioned policy that decides how much reasoning to buy, how to explore alternatives, which evidence to trust and when to stop.
Evidence confidence96%
Govern reasoning compute as an adaptive decision policy, not a fixed token allowance55 min read
Guaranteed JSON is only the visible edge of a larger serving subsystem. Production structured generation now has to normalize schemas, negotiate backend support, compile grammars against tokenizers, cache masks, coordinate reasoning and tool-call regions, validate semantics, limit hostile complexity and preserve a safe fallback when the requested contract cannot be enforced.
Why it is movingStructured output support has moved from application-side retries into the decoding path of major APIs and inference engines. vLLM, TensorRT-LLM, SGLang, TGI, llama.cpp and provider APIs expose JSON Schema, regular-expression, grammar or structural-tag controls. XGrammar, LLGuidance, Outlines and related engines compile those contracts into parser state and token masks. Newer work focuses on dynamic structures, cross-grammar caches, reasoning-before-constraining, environment-dependent grammars and the quality loss that can occur when syntactic validity is mistaken for semantic correctness. The operational shift is from asking a model to format text to running a schema compiler and policy engine inside every generation request.
Evidence confidence97%
Treat every output schema as executable serving policy, not prompt decoration53 min read
Sharing one base model across many lightweight adapters reduces duplicated weights, but it creates a new production control plane for adapter identity, placement, loading, batching, cache compatibility, tenant isolation and rollback. The hard problem is no longer whether LoRA is parameter-efficient. It is whether the correct signed adapter can be resident, scheduled and removed without corrupting another tenant’s latency, memory or model behavior.
Why it is movingCurrent inference stacks support per-request LoRA selection, multiple simultaneously active adapters, CPU-side adapter pools, dynamic resolver plugins, runtime load and unload operations, slot limits and adapter eviction. At the same time, systems research is moving from heterogeneous batching alone toward adapter–KV co-management, merge/unmerge decisions, request–adapter co-migration, rank-aware scheduling, cross-adapter cache reuse, disaggregated LoRA execution and online residency control. The operational shift is from treating an adapter as a small weight file to treating it as versioned executable state with placement, compatibility, security and service-level obligations.
Evidence confidence96%
Govern adapters as versioned executable dependencies, not filenames51 min read
Modern inference runtimes can continuously batch, chunk prefills, prioritize requests, pause work and scale separate prefill and decode pools. The difficult production problem is no longer filling the GPU. It is deciding which heterogeneous requests may enter, how much shared latency slack they may consume and when overload must produce an explicit queue, fallback or rejection instead of hidden SLO failure.
Why it is movingCurrent serving stacks expose first-come-first-served and priority policies, chunked prefill, token and sequence budgets, KV-cache watermarks, request pausing, custom capacity schedulers, prefill/decode planners and workload-aware simulation. Recent systems work is increasingly framed around deadline slack, SLO goodput, head-of-line blocking, branch externalities, adaptive preemption, request migration and heterogeneous multi-model work. The operational shift is from asking how many requests fit in a batch to asking whether each admitted request can complete without violating the latency, fairness, memory and recovery contracts of work already in flight.
Evidence confidence96%
Optimize SLO goodput, not raw batch occupancy50 min read
Long-context serving increasingly depends on whether reusable attention state can be identified, isolated, placed, transferred, prefetched and invalidated across GPU memory, host DRAM, NVMe and remote stores. The KV cache is therefore moving from an engine-local allocation detail toward a governed storage and routing plane with database-like identity, locality, consistency, security and lifecycle obligations.
Why it is movingCurrent serving systems now expose automatic prefix caching, cache-aware routing, KV connectors, disaggregated prefill, GPU-to-GPU transfer, CPU and filesystem offload, remote cache stores, hierarchical write policies and global cache metadata. Research is simultaneously treating KV state as a content-delivery object, a compressed stream, a database record and a placement signal. The operational change is that a cache hit is no longer merely a local memory event: it is a decision about exact model state, token identity, tenant boundaries, topology, transfer cost, quality and retention.
Evidence confidence97%
Qualify cache identity and movement before counting cache hits52 min read
Sparse activation reduces the expert arithmetic used by each token, but it does not remove the full model’s memory, token-routing, all-to-all communication, straggler and failure obligations. Modern MoE serving is therefore shifting from a fixed sharding decision toward a governed runtime that measures expert hotness, places or replicates experts, selects phase-specific communication and survives changing traffic without altering model meaning.
Why it is movingCurrent MoE serving stacks expose expert parallelism, redundant experts, expert-placement strategies, periodic load balancing, phase-specific all-to-all backends and elastic membership. At the same time, 2026 systems research is treating expert placement as an online optimization problem, studying proactive migration, least-loaded execution, portable communication and partial-rank recovery. The practical shift is from asking whether a model activates few experts to asking whether the complete router–placement–network–kernel system remains balanced, correct, affordable and recoverable under real traffic.
Evidence confidence96%
Benchmark the router, network and placement policy together48 min read
Modern inference speed increasingly comes from generated graphs, fused kernels, autotuned schedules and cached machine artifacts. That makes the compiler output part of the deployable product: it needs a precise hardware identity, numerical qualification, provenance, cold-start controls, performance acceptance and rollback—not just an impressive microbenchmark.
Why it is movingPyTorch now exposes ahead-of-time compilation, portable cache artifacts and provenance tracking; vLLM treats torch.compile as a default serving component with explicit cache keys and pre-serving compilation; Helion and Triton are expanding portable kernel authoring and autotuning; TensorRT-LLM, CUTLASS and CuTe DSL package hardware-specific kernels; StableHLO, MLIR and IREE provide portable intermediate layers; and current systems research is moving toward megakernels, automated superoptimization and AI-assisted tuning. The trend is no longer simply to compile a model. It is to release a qualified model–graph–compiler–kernel–hardware bundle.
Evidence confidence97%
Ship compiled artifacts, not benchmark screenshots55 min read
Speculative decoding can preserve a target model's output distribution while reducing serial decode steps, but speedup is conditional. Production gains depend on accepted tokens, proposer cost, verification kernels, batching, context length, tokenizer alignment, traffic shape and feature compatibility—not on draft depth alone.
Why it is movingCurrent serving stacks support multiple speculative families: separate draft models, feature-level drafters such as EAGLE, multi-token prediction heads, Medusa-style heads, recurrent drafters, n-gram and suffix retrieval, self-speculation and lookahead decoding. vLLM now positions speculative decoding primarily for medium-to-low-QPS memory-bound workloads and exposes workload-specific method selection. TensorRT-LLM, SGLang, Transformers, llama.cpp and OpenVINO provide production implementations, while recent evaluations show that verification overhead, acceptance variation and batch expansion can erase theoretical gains. The engineering trend is toward qualification by target, proposer, runtime, hardware and traffic class.
Evidence confidence97%
Measure accepted tokens, not draft depth52 min read
FP8, FP4, INT4 and quantized KV caches are moving from specialist tricks into mainstream serving stacks. The winning practice is not choosing the smallest number of bits; it is qualifying one exact model, tensor map, scale rule, kernel, device and workload as a controlled release.
Why it is movingNVIDIA TensorRT-LLM now exposes FP8, FP4, AWQ, GPTQ and quantized KV-cache recipes; PyTorch torchao documents float8, int8, int4, MXFP4 and NVFP4 inference workflows; vLLM maintains hardware-specific compatibility across AWQ, GPTQ, FP8, FP4 and compressed-tensor backends; Intel, AMD, Microsoft and Hugging Face provide parallel deployment paths. The market signal is no longer that quantization exists. It is that precision selection, calibration, kernel support, mixed-precision fallback, cache policy and acceptance testing have become part of the production release process.
Evidence confidence98%
Qualify the format, kernel and workload together52 min read
Model quality no longer determines serving quality by itself. Queueing, KV-cache memory, batching, routing, speculation, parallelism and data transfer increasingly decide whether an AI product is fast and economical.
Why it is movingvLLM, SGLang, TensorRT-LLM and NVIDIA Dynamo now expose continuous batching, paged KV memory, prefix reuse, speculative decoding, cache-aware routing and prefill-decode disaggregation as production serving controls. Kubernetes inference routing, distributed KV stores and phase-specific schedulers are moving these decisions above the individual model server. Current documentation and systems research also show that gains depend strongly on prompt length, output length, arrival rate, cache locality, interconnect topology and latency objectives.
Evidence confidence96%
Serving architecture now shapes the product40 min read
Hardware-isolated execution and remote attestation can protect models and data while they are in use. The technology narrows infrastructure trust; it does not make the application, model or output correct.
Why it is movingConfidential CPU VMs are now established across major clouds, while confidential H100 and newer GPU deployments, composite CPU–GPU attestation, Kubernetes confidential containers and policy-bound model-key release are moving into production architectures. NIST’s 2026 draft guidance specifically applies hardware-enabled confidential computing to cloud AI workloads, while current vendor advisories and attack research show that firmware, policy and application assurance remain active requirements.