A Model Without a GPU: When the CPU Is Actually Enough

Most guides to local AI start with one question: "How much VRAM do I need?" That is the right question if you want speed. But it is not the only one. There is a second way of doing local inference that has become fully legitimate in 2026: running the model entirely on the CPU, with no dedicated GPU at all. It is no longer a fallback for people who lack a graphics card. It is a reasoned choice that works well for a precise range of use cases.

This article tells you when the CPU is genuinely enough, what speed you can realistically expect, how to measure your own hardware in five minutes, and which levers to pull to squeeze the most out of a processor. We are not repeating the GPU-vs-CPU theory (there is already a dedicated article on that): this one is about practice.

Why CPU inference is no longer a fallback

The reason is simple and measurable. Decoding a token on a CPU is a memory-bandwidth problem, not a raw-compute problem: for every token, the processor must read the model's weights from RAM. If your RAM is fast enough and the model (quantized) fits entirely in memory, the CPU produces tokens at a steady, predictable rate.

Let's do the arithmetic. A modern workstation with dual-channel DDR5 has roughly 60–70 GB/s of bandwidth. A mid-range GPU, the RTX 3060 12GB, has 360 GB/s on its VRAM. Take an 8-billion-parameter model quantized to 4 bits, which is about 5 GB of weights:

• On a CPU at 64 GB/s, decoding sits around 10–12 tokens per second.
• On the RTX 3060, with the same model, you are between 60 and 90 tokens per second.

The ratio is roughly the bandwidth: 64 against 360. The CPU is not losing because it is slow; it is losing because it reads the weights over a narrower bus. The gap is huge, but it is structural and predictable, not random. And for a certain kind of use, that gap changes nothing.

There is also a second phase, the prefill, where the model reads your prompt all at once. There the CPU loses far more, because prefill is a parallel-compute problem and the GPU wins outright. If your prompts are long, the wait before the first token is the CPU's weak point. Keep it in check (short prompts, reduced context) and the limit becomes manageable.

When the CPU is actually enough

These are the cases where, for the same result, the CPU costs you nothing.

1. Small models, under 4B. A model from 0.5B to 4B quantized to 4 bits weighs between 0.4 and 2.5 GB. On a CPU you run it at conversational speed: 20–40 tokens per second for the smallest. For summarization, classification, reorganizing text, extracting fields, and short completions, that is more than enough. This is where the CPU does not merely "suffice" — it wins on everything else: zero cost, zero setup, works on any recent machine.

2. Low-frequency conversation. If you use the model for a handful of questions a day, with medium-to-short answers, and do not care whether the first answer arrives in one second or three, the CPU is the cheapest and most reliable option. It is not for anyone who wants a real-time experience, but it is perfectly adequate for people who use local AI as a work tool, not a snappy chatbot.

3. Batch and overnight jobs. Translating a corpus, generating metadata, running large-scale classification, producing summaries of hundreds of documents: here speed is not a problem, it is the cost you are happy to pay over a night. The CPU works at 8–15 tokens per second, and if the job is a thousand documents you launch it in the evening and wake up to finished work. No subscription, no cost spike.

4. Edge and always-on devices. A Raspberry Pi 5, a 40–80 euro mini-PC, an old laptop with 16GB of RAM: these are the places where the CPU is not the only option, it is the option. A 1–3B quantized model runs there, slowly but it runs, and it lets you have an assistant that is always available without sending any data to any server.

5. Reliability before speed. In a small production context (an internal bot, a triage workflow, a summarizer over files received by email), the most expensive variable is not response time but the fact that the system is always there, with no external dependency. The CPU, on a silent, 8-watt mini-PC, is the simplest local infrastructure that exists.

When the CPU is not enough: real-time conversation, models above 14–20B, image generation, high throughput. In those cases you need a GPU, or at least a CPU+GPU hybrid that covers part of the bottleneck.

What speed you can realistically expect

These are reference numbers on a typical CPU workstation (8 cores / 16 threads, dual-channel DDR5, enough RAM). Yours will differ: the factor that matters is your RAM's bandwidth.

Model (4-bit)	Decode	Prefill	Use
0.5B – 1B	25–45 tok/s	100–300 tok/s	chat, classification, completion
3B – 4B	12–25 tok/s	50–150 tok/s	light conversation, text reorg
8B (dense)	8–15 tok/s	20–60 tok/s	acceptable chat, summaries
8B – 14B (MoE)	15–35 tok/s	30–80 tok/s	MoE: fewer active weights → faster
14B – 30B	3–8 tok/s	10–30 tok/s	batch, non-interactive use
30B+	under 5 tok/s	very slow	impractical for interactive use

Three important notes. First: decode numbers are bandwidth-limited, so they scale almost linearly with your RAM's bandwidth. Moving from dual-channel DDR4 (40 GB/s) to dual-channel DDR5 (64 GB/s) gains you 50–60% more tokens per second. Second: MoE (mixture-of-experts) models are a pleasant surprise on CPU, because for each token they activate only a fraction of the total weights and therefore read less memory: a 30B MoE with 3B active can beat a dense 8B. Third: prefill is where the CPU suffers most; if your prompts are long, that is where you lose.

How to measure your hardware in 5 minutes

Do not trust other people's numbers: measure your own. The standard procedure is this.

Download llama.cpp (compile it or grab a binary), download a reference GGUF model (Qwen3 8B Q4_K_M is the de facto standard), and run the built-in benchmark:

./llama-bench -m model.gguf -p 512 -n 128 -t <thread>

The benchmark gives you three numbers: pp512 (prefill of 512 tokens) and tg128 (decode of 128 tokens). Those are your references. Run it again with -t 4, -t 8, -t 16 to find the peak-decode point: it usually coincides with your physical core count, not the logical count.

Then check the real bottleneck during use:

perf stat -e cache-misses,cycles ./llama-cli -m model.gguf

If cache-misses is high and the cores are saturated but RAM is not, the bottleneck is memory bandwidth and there is not much you can do: a smaller model or faster RAM. If the cores do not reach 100%, you have too few threads or an SMT (hyperthreading) limit: try lowering the thread count. If physical RAM is exhausted (swapping to disk), the model does not fit: drop the quantization or the size.

The golden rule: always measure your hardware with your model and your quantization. An 8B Q4_K_M on a machine with 64 GB/s does 11 tok/s; the same model on a machine with 40 GB/s does 7. That is a 33% gap, and you will not find it on any blog.

The levers to squeeze the CPU

This is where most of the performance comes from. Four levers, in order of impact.

1. Threads and cores. The optimal thread count for decoding is almost always the number of physical cores, not logical ones. On an 8-core / 16-thread CPU, -t 8 beats -t 16 in most cases: SMT doubles the count, not the bandwidth. Test it and verify with llama-bench.

2. Quantization. 4-bit quantization (Q4_K_M) is the sweet spot: still-high quality, half the size of 8-bit, double the speed. Dropping to Q3 or Q2 buys speed but loses quality in a noticeable way; going up to Q5 or Q8 makes no sense on CPU because the bottleneck is bandwidth. The GGUF format was designed exactly for this: one file, pre-quantized weights, no dependencies.

3. Flash attention. Flash attention reduces the memory read during decoding and prefill, with a gain that is especially marked on long contexts. In llama.cpp you enable it with the -fa flag: it is a single thing to add to the command line and the benefit is measurable. If your version is recent, turn it on and run a before/after with llama-bench: on your hardware the improvement is more predictable than any number reported on a blog.

4. Speculative decoding. The most elegant technique: a small model (the draft) produces a run of tokens quickly, and the large model verifies them all in a single prefill pass. If the draft guesses well — which, for most everyday tasks, it does — the result is a 2–3x multiplier on decode speed with no loss of quality. On CPU, where decode is the bottleneck, this is the most powerful lever.

And a fifth, for large models: mmap. Mapping the weights into memory (--mmap) lets the OS load only the weights it needs into RAM and keep the rest on disk. That means a model that does not fit in RAM can still run, at reduced but non-zero speed. The trade-off is that every disk access is much slower than a RAM access: mmap is a way out, not a solution.

For serving, Ollama already handles all of this in the background: a single ollama serve command, and your models run with the right configuration for your machine. The v0.35.0 release also added support for Jev-style decision models, which produce structured outputs for classification and evaluation — exactly the kind of load that suits a CPU well. For batch or serving workloads, vLLM also supports CPU (via OpenMP), even though its heart is the GPU: it is the right choice if one day you want to serve multiple users in parallel.

The picture in 2026

The point is this: CPU inference in 2026 is not a niche case. llama.cpp keeps investing specifically in the CPU side: the October run of releases (the latest is b11337) widened the CPU and edge backend matrix — Snapdragon with Adreno and Hexagon, OpenVINO, SYCL FP16, CUDA 13, Android — and models keep getting more efficient: MoE reduces active memory, quantizations get more aggressive without losing quality.

And the movement is going the right way on the model side too: Cloudflare made clef public, a 27B model that turns a state and a set of typed questions into structured decisions — one probability per option in a single pass, with no free-form text generation — alongside a smaller, faster variant (clef-flash) built for low-latency workloads. It is not a chatbot: it is exactly the class of models — structured outputs, short inference, minimal latency — that works well on CPU. The line between a "GPU model" and a "CPU model" is moving upward, and compact and MoE models with low active weights are becoming the most interesting class for local CPU inference.

The question is no longer "can I run a model without a GPU?" The question is: "does my specific use need more than 15 tokens per second?" If the answer is no, the CPU does not just suffice: it is the best choice.

In short

• The CPU is sufficient for models under 4B, low-frequency conversation, batch jobs, edge, and reliability.
• Decode on CPU is bandwidth-limited: speed depends on your RAM's bandwidth, not the processor's clock.
• MoE models are the surprise: fewer active weights → faster at equal total parameters.
• Always measure prefill and decode for your own setup with llama-bench.
• The levers, in order of impact: threads = physical cores, Q4_K_M quantization, flash attention, speculative decoding.
• For serving: Ollama for single use, vLLM for batch and multi-user.
• The line between "CPU model" and "GPU model" is moving toward 20–30B: by 2027 the CPU will run models you would today consider GPU-only.

Article generated with AI