Low-bit inference is becoming a qualified deployment layer

Two files described as four-bit models can use different integer or floating-point formats, group sizes, scale precision, zero-point rules, clipping, protected layers and packing layouts. One runtime may execute native low-bit matrix multiplication; another may unpack or dequantize to a higher precision before computing. The visible label therefore says little about latency, quality or compatibility by itself.

The release identity should include the source model revision, tokenizer, adapter set, quantizer and version, recipe, calibration fingerprint, tensor precision map, scale metadata, packing format, runtime, kernel build, device capability and sampling configuration. Changing any of these can change outputs or performance.

This is why pre-quantized community checkpoints require the same supply-chain discipline as compiled binaries. The artifact must be reproducible and tied to the hardware and runtime that qualified it.

Weight-only quantization primarily reduces model storage and memory bandwidth. Activations remain higher precision, which often protects quality but limits arithmetic savings. Weight-and-activation formats can use native low-bit tensor cores and reduce more traffic, yet activation outliers make them harder to qualify.

Embeddings, first and last layers, normalization, output heads, attention projections, mixture-of-experts routers and small modules may be disproportionately sensitive. A mixed-precision map can keep these components in BF16, FP16, FP8 or INT8 while compressing the bulk of linear weights more aggressively.

KV-cache quantization is a separate decision. It increases concurrent context capacity and reduces transfer cost, but errors accumulate across layers and long token histories. The correct cache format may differ between short chat, retrieval-heavy workloads and very long generation.

INT8 and INT4 use discrete integer levels plus scales and sometimes zero points. They can be symmetric or asymmetric and use per-tensor, per-channel or groupwise parameters. Smaller groups usually preserve more local detail but add metadata and can reduce kernel efficiency.

FP8 formats retain sign, exponent and mantissa, giving wider dynamic range than an integer with similar storage. FP4 and microscaling formats use very small elements with block-level shared exponents or multiple scale levels. These formats depend strongly on native hardware and precise scale handling.

Accumulation precision remains important. A system may multiply low-bit operands while accumulating in FP16, BF16 or FP32. The format of weights, activations, scales, accumulators and outputs should all be explicit.

Post-training quantization is attractive because it avoids full retraining. Simple round-to-nearest can work for tolerant models and higher bit widths. GPTQ, AWQ, SmoothQuant, OmniQuant, rotations and related methods use weight statistics, activations or reconstruction objectives to preserve more quality.

Calibration data determine which ranges and outliers the quantizer sees. A small English text sample may not represent code, math, rare languages, multimodal tokens, tool schemas, long prompts or production system messages. The sample should cover the token and activation distribution that matters, while remaining isolated from protected evaluation sets.

Calibration is also a governance artifact. Record dataset identity, selection logic, preprocessing, sample count, sequence lengths and random seeds so the scales can be regenerated.

QAT inserts fake quantization during training so the model learns to operate with rounding and clipping. It can recover quality when low-bit activations, embeddings or sensitive layers fail under post-training methods. It can also combine with LoRA or other parameter-efficient updates.

QAT adds data, optimization and reproducibility risk. A successful training loss does not prove that the final packed runtime kernel matches the fake-quantized numerics. Conversion and serving must still be tested end to end.

Use QAT when a measured quality gap justifies it. Preserve the PTQ baseline, training data rights, optimizer state, fake-quant configuration and final conversion recipe.

Quantized inference only accelerates when the device and runtime execute efficient low-bit kernels for the actual matrix shapes. Packing, alignment, batch size, sequence length, fused operations, graph compilation and expert layout all affect the result. A runtime may silently dequantize unsupported layers or switch to a generic kernel.

Weight-only methods often help decode because generation repeatedly streams weights. Prefill can remain compute-heavy, especially at large batches, so a weight-only format may provide less benefit. FP8 or FP4 weight-and-activation paths can accelerate more arithmetic when hardware support is native.

Kernel coverage should be measured, not assumed. Trace operator dispatch, fallback count, graph breaks, dequantization time and achieved bandwidth. Benchmark cold loading, steady state, small batches and peak concurrency.

KV cache grows with active tokens, layers, heads and cache element size. Reducing cache precision can increase concurrency or context length without changing weight format. It can also reduce the bandwidth cost of cache transfer in disaggregated serving.

Cache errors interact with attention, position, retrieval and generation length. A format that preserves short-answer quality may degrade long-context retrieval, copying, code continuation or multilingual generation. Per-channel or asymmetric designs can improve accuracy but require compatible kernels and scale storage.

Cache policies must include eviction, prefix reuse, tenant isolation and cache-format identity. Reusing cache blocks produced under a different model, adapter, tokenizer, rope scaling or quantization recipe can produce incorrect output.

Aggregate perplexity can miss failures in code syntax, arithmetic, JSON generation, tool selection, refusal behavior, rare languages and long-context evidence use. Quantization may preserve average accuracy while changing confidence margins or increasing instability on specific prompts.

Acceptance should combine model-level metrics with application outcomes. Compare exact task success, structured-output validity, safety decisions, hallucination rate, calibration, subgroup performance and consistency across seeds. Use paired tests against the same high-precision outputs where appropriate, but retain human or authoritative ground truth.

Define tolerances before seeing results. A memory-saving candidate should not be accepted because its losses seem small after the fact.

A quantized model is not a one-time conversion. New runtime versions, kernels, drivers, adapters, rope settings or model revisions can change numerical behavior and performance. Each combination needs canary evidence before broad rollout.

Production telemetry should record release identity, tensor format map, fallback kernels, memory, cache occupancy, TTFT, token cadence, retries and task-quality proxies. Sudden fallback rates or latency changes can indicate that an optimized path stopped executing.

Keep the high-precision or higher-bit release available for rollback and difficult workloads. Routing can send sensitive tasks, very long contexts or unsupported hardware to the safer artifact while low-bit candidates handle qualified traffic.