Small AI models are moving onto ordinary laptops

A current laptop may offer a CPU, an integrated or discrete GPU, and an NPU. That sounds like plenty of choice, but software support decides what is actually useful. The runtime has to understand the model's operations, precision and memory layout. If it cannot place the full graph on the preferred accelerator, parts of the job will run elsewhere.

The CPU remains the dependable fallback. Small, well-quantized models can be perfectly usable there, especially for extraction, classification and short-form assistance. A GPU usually improves throughput because it has more parallel compute and bandwidth, but memory capacity becomes the limit. With an integrated GPU, the model competes with the operating system and every other application for the same pool.

NPUs are attractive because they are designed for efficient machine-learning workloads. Even so, decoder-style language models are demanding customers. Operator coverage, static-shape constraints, cache handling and vendor-specific execution providers all matter. In practice, you have to inspect logs and measure the full request. A claimed NPU path is not proof that the whole model stayed on the NPU.

Most laptop deployments compress weights that were originally stored in 16-bit form. Eight-, six-, five- and four-bit versions need less storage and move less data through memory. That helps most during token generation, when the runtime repeatedly reads the weights.

The loss in quality is uneven. Some models tolerate four-bit quantization well; others lose instruction following, factual accuracy or structured-output reliability on the very tasks you care about. Two files described as '4-bit' may use different block sizes, calibration methods and mixed-precision rules, so the label alone does not tell you how they will behave.

Packaging mistakes can be just as damaging as aggressive compression. The tokenizer, special tokens, chat template, rope settings and any multimodal projector must match the weights. When those pieces drift apart, the model may repeat itself, ignore instructions or stop early. The file still loads, which makes the problem easy to misdiagnose.

The model first reads the prompt. This prefill phase can process many tokens in parallel, so it makes good use of accelerators. A long report or a large retrieval bundle can still delay the first word because every input token must pass through the model before generation begins.

Generation works differently. Each new token depends on the previous one, so the decode phase is largely sequential. The runtime keeps revisiting the weights and the growing KV cache. On many laptops, memory bandwidth matters as much as raw compute.

This is why a single tokens-per-second figure tells an incomplete story. A one-line chat prompt, a fifty-page document and four simultaneous users place very different demands on the same model. Thermal behavior also changes the picture. A laptop may look excellent for the first minute on AC power and slow down after ten minutes, on battery, or under a quiet fan profile.

A model does not get the entire memory specification printed on the box. Windows or macOS is already using part of it. So are the application, retrieval index, browser, video driver and background services. Once the KV cache and runtime buffers are added, a machine that looked comfortable on paper may be close to swapping.

A 16 GB system is generally a better home for small models and modest contexts. Some 7–8B four-bit models will run, but there may be little room left for a large document set or several sessions. At 24 or 32 GB, 7–14B-class quantized models become more practical, particularly when the product also needs retrieval and a sizeable context. A discrete GPU can be faster, yet its VRAM creates a separate ceiling unless partial offload is supported.

These are planning ranges, not guarantees. Model architecture, quantization, context length and backend buffers can move the requirement by gigabytes. The only defensible release number is the peak working set measured on each supported device class, with enough headroom for the rest of the user's machine.

Keeping prompts on the device removes one obvious data path: routine transmission to a remote API. It also creates a collection of local assets that need protection. Weights, document extracts, embeddings, chat history, temporary files, crash dumps and telemetry can reveal far more than teams expect.

Retrieval brings another problem. A document may contain text designed to override the application's instructions or manipulate a tool call. The model cannot be relied upon to separate trusted policy from hostile content on its own. Retrieved text should be labelled as untrusted, tools should be narrowly scoped, and any action with a real side effect should require authorization.

The model itself is part of the software supply chain. Use approved repositories, pin revisions and hashes, scan what you download, and avoid runtimes that execute arbitrary remote code during model loading. Drivers and inference libraries belong in the same update and provenance process as the weights.

A small model can sort requests, redact sensitive fields, search private files and handle routine drafts. A larger remote model may still be better for difficult reasoning, broad knowledge or a request that exceeds the local context window. Splitting the work this way is often more useful than forcing every task through one model.

The hard part is deciding what happens at the boundary. When the network disappears, does the product keep working? When the local answer is uncertain, does it ask a question, send the task elsewhere, or bluff? If policy forbids cloud use, does the application stop safely? Users experience those choices as product behavior, not as model configuration.

Versioning also becomes more important. The local model, remote model, router and runtime may change on different schedules. Every material update should trigger tests for routing accuracy, answer consistency, privacy rules and fallback behavior.

A demo prompt can make almost any model look good. Product testing has to be less forgiving. Fix the model revision, quantization, runtime, backend, driver, power mode, prompt set and output limit. Then judge speed and quality together. A faster answer is useless if it drops required facts or breaks the JSON schema.

Build tests from the work people will actually do: extract fields from known documents, generate constrained JSON, write code that must pass a test suite, summarize without losing specified facts, refuse unsafe requests, and resist hostile retrieved text. Keep some cases hidden from prompt tuning so the benchmark does not become a rehearsed performance.

MLPerf Client follows the same broad principle at industry scale: define the model, workload, quality threshold and tested configuration. An internal benchmark can be much smaller, but it should be just as clear about what was measured.