Nemotron 3.5 Bolt: NVIDIA's 74B MoE with ~9B active — what changes on your desktop

A 74-billion-parameter model running on your 24GB GPU? With MoE, that's not a contradiction: of those 74 billion, only a fraction works per token — here, about 9. Nemotron 3.5 Bolt is an NVIDIA MoE built for local agentic work, and its 2-bit version is designed to fit in 24GB of VRAM. Here's what changes on your desktop.

MoE, in two lines

With dense models, every token activates all the weights: a 30B model always computes 30B. With a Mixture-of-Experts (MoE), each token passes through a router that picks a handful of "experts" and activates only those. Nemotron 3.5 Bolt has 74B total parameters and about 9B active per inference. The difference that matters for you: the VRAM must hold all 74B (they're heavy, they take up space), but the speed is set by the 9B active (fewer computations per token). It's like an office with 74 desks where 9 are occupied: the work runs at speed 9, but the lease covers all 74.

The architecture is a Mamba-2/Transformer hybrid with a LatentMoE router, plus a C-RADIO v4-H vision encoder: the model reads text, images, and video, with context up to 256K tokens (model card).

How much VRAM the weights eat

The weights are the fixed cost of your VRAM: params × bytes per weight + KV cache (depends on context length and tokens generated). The fact that few parameters are active doesn't reduce the bytes: all 74B sit in memory; quantization is what compresses them.

Version	Bytes/weight	74B weights	Where it runs
BF16 (full)	2	~148GB	multi-GPU server
Q8	~1	~74GB	80GB workstation
Q4_K_M	~0.57	~42GB (estimate)	dual 24GB
IQ2 (24GB version)	~0.3	~24GB	one RTX 4090 / 5090, alongside other apps

The math is simple: 74 × 10^9 weights × 2 bytes = 148GB for the BF16, which NVIDIA published for post-training, not direct inference. The 24GB version is a 2-bit (IQ2) quantization built exactly for your card: NVIDIA states it runs at 24GB of VRAM while coexisting with your other open applications. Quantization is the point that keeps a big model on your hardware — if you want to understand the names and the trade-offs, I'll leave you with Quantization: get more from the same hardware.

The numbers: what a 2-bit version actually does

The numbers declared in the model card for the IQ2-24GB version, against the other MoE that fit in the same 24GB:

Benchmark	Nemotron 3.5 Bolt (IQ2)	Qwen3.6-35B-A3B	Gemma 4 26B-A4B	Nemotron-Lightning-30B-A3B
Terminal-Bench 2.1	50.98	44.38	37.22	23.46
GDPVal-AA-V2	28.47	25.75	15.33	18.26
GPQA Diamond	86.43	83.40	79.61	75.57
BrowseComp	66.75	48.74	26.30	36.81

The reading that matters: on Terminal-Bench 2.1 — the model runs commands, navigates the file system, solves tasks — the Bolt beats Qwen3.6-35B by ~7 points and Gemma 4 by ~14. Same story on GDPVal (real work tasks), BrowseComp (web navigation), and GPQA (reasoning). Declared numbers, always: these are NVIDIA's in the model card, not tested by us. The context: the full-precision BF16 scores 53.93 on Terminal-Bench and 88.83 on GPQA, so the 2-bit version stays very close to the original model.

How to run it

Three commands, depending on your stack:

• Ollama — the simplest: ollama run hf.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF
• llama.cpp — if you want to control the flags: llama serve -hf nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF
• vLLM — if you're serving the model to agents or scripts: vllm serve nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB

Ollama if you want to open and chat; vLLM if you want to serve multiple requests in parallel. The vLLM recipes NVIDIA publishes include speculative decoding with a LiLiCorr drafter (7 speculative tokens): more speed without more hardware.

And the limits?

Honest, because they matter:

• 2 bits is not free. Quality is a step below the BF16: the same table shows it (Terminal-Bench 50.98 vs 53.93). For serious work, expect a step back from the full model.
• The BF16 is not for the desktop. 148GB of VRAM means a multi-GPU server: it's the version for post-training, not your machine.
• English is the reference language. The model covers English + coding, Spanish, French, German, Italian, and Japanese: Italian is there, but it's not the strongest language.
• The Mamba-2/Transformer hybrid is new. Backends must support it: use current versions of Ollama or llama.cpp.

The test I'd recommend: 5-10 commands you actually use on one of your projects, 5 questions about your own documents (with images), 10 questions in Italian — then compare against the models you already have. The benchmarks say it's the best MoE in your 24GB; your work says whether it's the right model for you.

If you want to go further

• Aleph Alpha's Kolibri: the 78B MoE that runs (almost) locally — the other way to bring big MoE to local
• When a big model isn't needed: choosing by size — the opposite criterion: when an 8B is enough
• Hardware for local AI — which GPU to buy for a model like this

In summary: Nemotron 3.5 Bolt puts a 74B MoE (9B active) into 24GB of VRAM thanks to an IQ2 quantization built for exactly that. The declared numbers put it ahead of Qwen3.6-35B, Gemma 4 26B, and Nemotron-Lightning-30B on terminal, web, and reasoning. Three commands to run it (Ollama, llama.cpp, vLLM), text-image-video support, 256K context, and an OpenMDW-1.1 license that holds up in business. The limit to keep in mind: 2 bits is not 16, and Italian is not the headline language. Test it on your own tasks before moving everything over.

Sources

• Model card NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 (Hugging Face)
• Model card NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF (Hugging Face)

Article generated with AI