Nemotron 3.5 Bolt Squeezes a 74B Agent Model Into 24GB by Dropping Its Experts to 2 Bits

NVIDIA’s Nemotron 3.5 Bolt 74B-A9B, whose Hugging Face model card is dated October 15, 2026, is a vision-language model built to run everyday agent work on a single 24GB GPU. It has 74 billion total parameters and 9 billion active per token, a 256K-token context window, and text, image and video inputs with text output. The weights ship under OpenMDW-1.1, a permissive model license, and the cards say the model is ready for commercial use.

A Super VL model, compressed by search

Bolt is not trained from scratch. The model card says it was compressed from Nemotron 3.5 Super VL 120B-A12B, whose BF16 and NVFP4 checkpoints are also on the Hub, using an NVIDIA technique called Star Elastic. A budget-conditioned router searches the architecture layer by layer, choosing how many routed experts each layer keeps and how wide each MoE block is. The search target was a footprint under 24GB at batch size 1 and 128K context, already accounting for later quantization. Experts activated per token drop from 22 in the parent to 9.

The card gives the token budget in detail. The router converged within the first 8 billion tokens. Recovery training then ran to 72 billion tokens at an 8K context, followed by 48 billion tokens at 128K, for 120 billion in total. NVIDIA reattached the parent’s vision tower, a C-RADIO v4-H encoder, retrained only the vision-to-language projector, and then ran vision-heavy supervised fine-tuning, agentic training and RLHF. An end-to-end reinforcement learning stage on a simulated MCP server lifted MCP-Atlas from about 48 to about 56 on the 500 public tasks, according to the card.

Where the 2-bit budget goes

The 24GB figure comes from aggressive, uneven quantization. Every routed and shared expert is stored as IQ2-XXS at 2.0625 bits per weight. Mamba layers stay at NVFP4, and the embedding and output layers stay at Q8_0. NVIDIA used quantization-aware distillation through its ModelOpt toolkit to recover accuracy. It cites prior research suggesting that, for fully connected layers, a model twice as large at 2 bits can beat the original at 4 bits. The card also notes that at small batch sizes, kernel latency improves roughly in line with weight compression.

The architecture stays a Mamba-2–Transformer hybrid LatentMoE, the same family as the rest of Nemotron 3.5. That matters for long contexts on consumer hardware, because Mamba layers do not grow a KV cache the way attention layers do.

Four repositories, two runtimes

NVIDIA split the release by job. NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB is the packed model for vLLM. A matching -GGUF repository targets llama.cpp. A BF16 checkpoint is meant for fine-tuning, RL and making other quantizations, not production serving. A small LiLiCorr drafter provides DFlash speculative decoding, with seven speculative tokens in the vLLM recipes and six in the llama.cpp example. NVIDIA lists test hardware from H100, H200 and GB200 down to the GeForce RTX 5090, DGX Spark and the new RTX Spark. The 24GB card’s software section names vLLM on Linux, targeting Blackwell SM120 and SM121 GPUs.

English and programming languages are the primary targets, with Spanish, French, German, Italian and Japanese also supported. Post-training data runs through May 2026.

The benchmark picture, in NVIDIA’s numbers

All of the scores below are NVIDIA’s own. Against Qwen3.6-35B-A3B, Gemma4 26B-A4B and its own Nemotron 3.5 Lightning 30B-A3B, the 2-bit Bolt leads on most agent rows. It scores 50.98 on Terminal-Bench 2.1 versus 44.38 for Qwen, 66.75 on BrowseComp versus 48.74, and 50.86 on vision WebArena. The full-precision BF16 checkpoint scores slightly higher: 53.93 on Terminal-Bench and 70.00 on BrowseComp.

It does not win everywhere. On AA-Omniscience non-hallucination, Bolt 24GB scores 61.27, behind Lightning at 68.81 and Qwen at 65.85. Qwen edges it on PinchBench. NVIDIA also footnotes that Qwen reports 62.8 on MCP-Atlas while NVIDIA measured 51.2 on the public task subset. Readers should keep that gap in mind when comparing cross-vendor tables.

For local-agent builders, Bolt’s pitch is a 74B-class agent model that fits beside a desktop workload on one card, with open weights, an open license and recipes for both vLLM and llama.cpp. The license matters as much as the size. OpenMDW-1.1 grants broad rights to the model materials, requires keeping the license and notices when redistributing, and places no restrictions on outputs.