Mixture-of-experts serving is becoming an online expert-placement problem

A mixture-of-experts layer activates only a subset of expert feed-forward networks for each token. That can increase model capacity without multiplying arithmetic by the total number of experts.

The serving system still has to store or access the complete expert set, route every token, move hidden states to the selected experts, execute irregular batches and combine the results. Those obligations shift cost from pure compute toward memory capacity, network topology, scheduling and reliability.

The correct comparison is therefore not total parameters versus active parameters. It is end-to-end accepted-task performance for a specific checkpoint, router, workload and cluster.

In a dense transformer, each token follows the same feed-forward path. In a token-choice MoE, router scores select one or more experts, so the input itself determines which devices receive work.

Training-time load-balancing objectives can improve average utilization, but they do not guarantee balanced production traffic. Languages, domains, prompt templates, retrieval context, speculative drafts and long-running conversations can create persistent or sudden expert hot spots.

Router telemetry must preserve enough context to distinguish stable specialization from a transient burst, while avoiding retention of raw tenant content when aggregate expert counts are sufficient.

Static expert parallelism maps experts to ranks at startup. That is simple and reproducible, but it assumes the placement remains suitable for future traffic and that every rank stays healthy.

Current serving stacks and research increasingly expose redundant experts, periodic placement updates, expert-weight movement and elastic membership. These features turn placement into a versioned state transition rather than a command-line constant.

Every transition needs an input telemetry window, a policy version, a proposed mapping, memory validation, migration plan, acceptance threshold and rollback target. Otherwise a load balancer can become an unreviewed production optimizer.

Prefill processes many tokens per request and can create large expert batches. High-throughput dispatch, hierarchical aggregation and contiguous grouped GEMMs can amortize setup cost.

Decode often processes one new token per active sequence. The same expert set may receive small irregular groups, making launch overhead, synchronization and low-latency communication dominant.

A single backend or placement policy can be suboptimal across both phases. Qualification should report phase-specific latency and throughput, and any automatic mode switch must be observable and deterministic enough to reproduce failures.

Token dispatch and combine can use all-to-all, all-gather with reduce-scatter, hierarchical collectives or device-initiated point-to-point transfers. The best option depends on token volume, topology, GPU generation, NIC, process layout and whether the workload prioritizes throughput or latency.

High-bandwidth scale-up domains such as NVLink reduce the penalty of expert parallelism, while cross-node traffic introduces RDMA, congestion and topology sensitivity. Hierarchical designs aggregate locally before using scale-out links.

Backend selection must be qualified on the actual cluster. A library name is not evidence that the chosen mode, message sizes and link paths are correct for the service.

After dispatch, tokens are permuted into expert groups and processed by expert matrices. Large groups map efficiently to grouped or fused GEMMs; tiny and uneven groups can underfill the GPU and amplify launch overhead.

Prefill commonly uses contiguous expert segments. Decode may require masked layouts because the CPU does not know each expert count when CUDA graphs are captured. Precision and weight layout also constrain the available kernels.

Kernel benchmarks should be stratified by the real expert-token histogram. Reporting only a large balanced shape hides the exact cases that cause tail latency in online serving.

Moving a hot expert to another rank changes placement but may simply transfer the bottleneck. Replicating the expert increases capacity at the cost of memory and synchronization. Sharding the expert spreads its matrices but adds communication. Rerouting tokens can alter model semantics unless the method is designed and validated for that behavior.

The least invasive control is usually to improve placement within existing semantics. Replication can absorb stable hot experts. Expert sharding is useful when one expert cannot fit or cannot be served fast enough on one device. Router modification requires the strongest quality evidence.

A planner should compare these alternatives under capacity, topology and quality constraints instead of assuming expert migration is always the answer.

An MoE service can show high average utilization while one rank repeatedly becomes the straggler. It can also show low utilization because communication, synchronization or tiny expert batches dominate.

Useful telemetry joins request scheduling, router decisions, token permutation, dispatch, expert kernels, combine, collective synchronization and output acceptance into one trace. The trace should identify the critical rank and the exact phase that extended the step.

Metrics need bounded cardinality. Per-expert counters are manageable; per-token labels tied to user content are usually not. Sampled traces and aggregate histograms provide detail without turning observability into a privacy or cost failure.

When one request depends on experts distributed across many ranks, a single rank failure can invalidate communicator membership, expert coverage, routing metadata and captured execution graphs.

Restarting the entire instance is simple but creates a large outage and reload cost. Emerging approaches model membership as mutable state, restore lost expert coverage, repair communication and reintegrate ranks without rebuilding every healthy worker.

Production qualification should define the supported degraded modes: reject affected requests, use redundant experts, shrink the group, fall back to a smaller model or restart. Silent approximation is not an acceptable recovery policy.

A placement-only change should not alter expert selection or router weights. Yet implementation errors in permutation, capacity handling, quantization, top-k indexing or combine weights can produce plausible but incorrect outputs.

Differential tests should compare router selections, expert outputs, combined hidden states and final accepted tasks against a qualified reference. Test duplicate experts, empty experts, ties, boundary token counts and mixed precision.

When a technique intentionally reroutes tokens or substitutes replicas, document the equivalence contract. A throughput gain is not acceptable if it quietly changes model behavior beyond the approved tolerance.

An attacker does not need to know expert names to search for prompts that concentrate work on a subset of ranks. Repeated hot-spot traffic can increase tail latency, trigger migrations or consume redundant-expert capacity.

Rate limits, admission control and tenant isolation should operate before router pressure becomes a shared-cluster incident. Placement telemetry must not expose tenant prompts or sensitive domain specialization.

Expert weight movement and dynamic communicator changes also belong to the software supply chain. Authenticate control-plane commands, verify weight digests and limit which processes can change placement or membership.

First qualify a static placement with one communication backend and one expert kernel path. Without that baseline, later automation has no trustworthy comparison.

Next collect activation and critical-path telemetry without changing placement. Use offline replay to estimate candidate mappings and validate memory, topology and migration cost.

Introduce periodic placement changes in shadow mode, then canary one hardware pool. Add redundant experts or elastic membership only after placement governance, quality checks and rollback are reliable.