Nemotron 3.5 Bolt's pitch is a 74B-parameter agent model that fits in 24GB of VRAM, and the 24GB part checks out: NVIDIA quantized the checkpoint natively, not as an afterthought, and the weights really do land at 22.01GB. What doesn't check out automatically is the follow-on assumption that any 24GB card can serve it. NVIDIA's own vLLM recipe for that checkpoint pins kernels built for Blackwell, which an RTX 4090 doesn't have no matter how much VRAM it carries. That one line in the model card changes which instance you should actually rent.
TL;DR: What Does It Take to Deploy Nemotron 3.5 Bolt?
Nemotron 3.5 Bolt's 24GB checkpoint needs one 24GB+ GPU for weights, but NVIDIA's documented vLLM path requires Blackwell hardware specifically.
- Model: Nemotron 3.5 Bolt is NVIDIA's 74B-A9B LatentMoE agent model under the OpenMDW-1.1 license, with a 128K context window.
- VRAM: The 24GB checkpoint's weights total 22.01GB, plus about 0.87GB of KV cache at batch 1 and 128K context.
- GPU catch: NVIDIA's vLLM recipe pins Blackwell SM120/SM121 b12x kernels, ruling out an RTX 4090 for this serving path.
- Spheron lists the RTX 5090 GPU rental (Blackwell, 32GB) for the vLLM path and the RTX 4090 (24GB) for Ollama/llama.cpp, both on-demand per minute.
- Fine-tuning: QLoRA fits DGX Spark's ~119.68GB pool; standard LoRA likely doesn't at this parameter count.
What Nemotron 3.5 Bolt Actually Is: 74B-A9B LatentMoE, Native 2-Bit Quantization, 128K Context
Nemotron 3.5 Bolt is a 74B-total, 9B-active-per-token model, built as a hybrid of mixture-of-experts routing and Mamba-style state-space layers rather than a pure transformer stack. NVIDIA ships it under the OpenMDW-1.1 license, which permits commercial use, the same posture NVIDIA has taken with the rest of the Nemotron line.
The headline feature is the checkpoint itself. Most quantized releases are a community conversion of a full-precision model, done after the fact with whatever accuracy loss falls out. NVIDIA's own 24GB model card documents a checkpoint native to the 24GB target: expert layers quantized to IQ2_XXS (a 2.0625-bit format), Mamba layers quantized to NVFP4, and attention, embeddings, and the output layer held at Q8_0. The model supports text, images and video, tool calling and a 128K context window.
Ollama's own announcement of the build is blunt about what the quantization buys you: native 2-bit quantization that fits in 24GB of memory and runs alongside other applications. That's the pitch for a desk-side workstation. It's also exactly the pitch for a single rented GPU instead of a multi-GPU cloud node, which is the comparison the rest of this post runs the numbers on.
VRAM Math: Why 24GB Is Enough, What You Lose vs BF16, and the Blackwell Kernel Catch
Here's the direct answer: the 24GB checkpoint's weights fit in 22.01GB, leaving about 2GB of headroom on a 24GB card before cache and activations, and NVIDIA's official serving recipe for that checkpoint only runs on Blackwell GPUs.
| Component | Precision | Size |
|---|---|---|
| Expert layers (MoE) | IQ2_XXS (2.0625-bit) | part of 22.01GB total |
| Mamba layers | NVFP4 | part of 22.01GB total |
| Attention, embeddings, output | Q8_0 | part of 22.01GB total |
| Total weights | mixed | 22.01GB |
| KV cache + activations (batch 1, 128K context) | fp8 | ~0.87GB |
| Headroom on a 24GB card | - | ~1.1GB |
That headroom number is tight enough that batch size 1 is the realistic ceiling for a single 24GB card; pushing concurrency or context further means moving to a GPU with more VRAM, not a bigger batch on the same card.
NVIDIA's model card publishes benchmark scores for the 24GB checkpoint itself, not a side-by-side comparison against the full BF16 release, so treat these as the quantized model's own numbers rather than a measured delta:
| Benchmark | 24GB checkpoint score |
|---|---|
| GPQA Diamond | 86.43 |
| BrowseComp | 66.75 |
| WebArena (vision) | 50.86 |
| Terminal-Bench 2.1 | 50.98 |
| SciCode | 41.02 |
| PinchBench | 87.82 |
NVIDIA states there's little to no degradation in text accuracy from the compression, but doesn't publish the BF16 side-by-side needed to verify that claim directly. Treat it as a starting point to check against your own eval set, not a guarantee, especially for a task like browsing or tool-heavy agent work that leans on the benchmarks above specifically.
The part that catches people is architectural, not a VRAM number at all. NVIDIA's own vLLM deployment guide for the 24GB checkpoint states the build is deployed for Blackwell SM120 and SM121. Put plainly: a 24GB Ada card has the capacity to hold these weights, and it will run them through llama.cpp or Ollama's GGUF path without issue, but it cannot run NVIDIA's documented vLLM recipe for them, because that recipe's kernels don't exist on Ada silicon. The 24GB number answers "does it fit." It doesn't answer "which GPU," and those are two different questions for this specific model.
Deploy Nemotron 3.5 Bolt With vLLM on Spheron: Step-by-Step
This walks through NVIDIA's documented recipe on a Blackwell card, which is the path that gets you the b12x kernels and FlashInfer attention/Mamba backends the model card specifies.
- Provision a Blackwell GPU instance. The b12x kernel requirement rules out Ada cards for this recipe, so provision an RTX 5090 instance (32GB, SM120) on Spheron, billed per minute with a 20-minute minimum and no contract. Attach at least 60GB of persistent storage; the 24GB checkpoint plus the drafter model and any cached FlashInfer kernels need more than the weight footprint alone.
- Confirm the driver and CUDA version. SSH in and run
nvidia-smi. Blackwell consumer and data-center cards both need CUDA 12.8 or newer for the kernels this recipe depends on. - Install the nightly vLLM build. NVIDIA's model card specifies a nightly vLLM image rather than the latest stable release, since the b12x backend and the Mamba FlashInfer support it depends on haven't landed in a tagged release yet. Follow the exact nightly wheel or container tag listed on the model card at deploy time, since nightly builds move fast.
- Download the 24GB checkpoint from Hugging Face. Pull
nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GBdirectly; it ships pre-quantized, so there's no separate quantization step to run. - Launch vLLM with NVIDIA's documented flags. The model card's own configuration sets
--tensor-parallel-size 1,--max-model-len 262144,--kv-cache-dtype fp8,--linear-backend b12x,--moe-backend b12x,--attention-backend FLASHINFER, and--mamba-backend flashinfer. Don't substitute a different attention or MoE backend to "see if it's faster"; the model card pins these specifically because the Mamba and MoE layers are what the generic vLLM backends don't yet handle correctly for this architecture. - Add the speculative decoding drafter for throughput. NVIDIA pairs the checkpoint with a 1.13B drafter. Loading it alongside the main checkpoint is what turns the 24GB deployment from merely fitting into something fast enough for interactive agent calls.
- Test with a real agent-style call, not a single-turn chat prompt, since Bolt's accuracy and speed profile is tuned for multi-step tool use, which is the workload the GPQA Diamond and BrowseComp scores above are measuring in the first place.
If you'd rather skip the Blackwell requirement entirely, the GGUF path through Ollama or llama.cpp runs the same checkpoint on an RTX 4090 on Spheron (24GB, Ada), the exact ollama run nemotron-3.5-bolt command NVIDIA's own release announcement uses. You lose vLLM's continuous batching and the FlashInfer-accelerated Mamba path, which matters for concurrent agent sessions, but for a single-session development box it's the simpler setup and the cheaper instance.
For the general 2-bit and sub-4-bit quantization landscape this checkpoint sits in, including why Blackwell-only kernels keep showing up across NVIDIA's newer releases, see our MXFP4 microscaling quantization guide.
DGX Spark Fine-Tuning vs Renting a GPU for Bolt: Where Each Wins
DGX Spark's unified memory pool reports roughly 119.68GB usable to the OS. That's the number to fine-tune against, and it's the same figure our dedicated DGX Spark fine-tuning guide for 128GB unified memory uses for its full LoRA and QLoRA sizing formula. Rather than re-deriving that formula for Bolt specifically, the useful move is to place Bolt's 74B total parameter count against the closest bracket that post already worked out: a 70B dense model.
At that bracket, QLoRA needs roughly 52GB, which leaves close to 67GB of headroom inside DGX Spark's 119.68GB pool, comfortable enough to run alongside the OS and the rest of a dev box without the job competing for the same shared memory it needs. Standard LoRA is the harder case: the same formula needs roughly 159GB at this parameter count, more than one DGX Spark unit holds, full stop, regardless of batch size or offloading tricks. If your fine-tuning plan for Bolt is QLoRA, DGX Spark is a real option. If it's a standard LoRA run or anything closer to a full fine-tune, the job needs to move to a rented multi-GPU instance before you start, not after it runs out of memory partway through.
Inference is the other half of this comparison, and it's where DGX Spark's bandwidth ceiling, not its capacity, becomes the limiting factor. DGX Spark's GB10 superchip delivers roughly 273 GB/s of memory bandwidth, and a GB10-class unit's bandwidth ceiling bites harder the bigger the model you push through it, independent of whatever quantization scheme got the weights to fit.
For a workload that needs more than single-session, single-user throughput, renting a Blackwell GPU wins on both axes DGX Spark is weak on: raw bandwidth and concurrent request handling. Our DGX Spark vs cloud rental break-even guide runs the hours-per-month math for exactly this decision against H100, H200, and B200 on-demand rates, and the same framework applies here: prototype the fine-tune locally on DGX Spark with QLoRA, then move production serving to a rented instance once you need more than one concurrent agent session.
Cost: One GPU vs an H100 Node for a 74B-Class Model
The entire point of a 24GB checkpoint is that it changes the hardware comparison from a multi-GPU node to a single card, so run that comparison explicitly before you provision anything.
A 74B-class model at full BF16 precision needs enough combined VRAM across multiple GPUs that the realistic starting point is an 8-GPU H100 node, the same tier NVIDIA lists as a supported architecture for the un-quantized Bolt release. On Spheron, an 8x H100 SXM5 node totals $17.60/hr on spot ($2.20/hr per GPU) as of 28 Oct 2026, or scales from the $2.65/hr on-demand per-GPU rate if you need guaranteed capacity instead of spot. That's the bill for serving the full-precision model, the version NVIDIA doesn't include in the benchmark table above since it only publishes scores for the 24GB checkpoint itself.
Nemotron 3.5 Bolt's 24GB checkpoint replaces that node with one GPU. A single RTX 5090 on Spheron runs $0.84/hr on-demand as of 28 Oct 2026, for the Blackwell vLLM path with the full drafter speedups. For the Ollama/llama.cpp path on Ada hardware, an RTX 4090 on Spheron runs $0.48/hr on-demand. Either number is a small fraction of an 8-GPU node's hourly cost, and for an agent workload where NVIDIA's "little to no degradation" claim holds up against your own eval set, that's the entire argument for choosing the 24GB checkpoint over the BF16 release in the first place.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 28 Oct 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.
The caveat worth repeating from the VRAM Math section: cost only drops this far if you can actually use the vLLM path, which means the GPU has to be Blackwell-architecture, not just 24GB-or-larger. Pricing out an RTX 4090 against this model's official serving recipe and finding it doesn't run the b12x kernels is a worse outcome than paying slightly more for the RTX 5090 that does.
Nemotron 3.5 Bolt is a case study in reading a model card past the headline VRAM number. Our Nemotron 3 Nano Omni deployment guide covers the same NVIDIA's-own-quantization pattern at a smaller 30B-A3B scale, if you want the comparison across the Nemotron lineup before picking which model and which GPU tier to commit to.
Nemotron 3.5 Bolt's 24GB checkpoint still needs the right GPU architecture underneath it, not just the right amount of VRAM, and that's exactly the kind of spec Spheron's per-minute billing lets you test before committing to a longer run.
Frequently Asked Questions
Nemotron 3.5 Bolt is NVIDIA's 74B-A9B LatentMoE agent model, released under the OpenMDW-1.1 license, which permits commercial use. It's built as a hybrid of mixture-of-experts and Mamba layers and ships with a native 2-bit quantized checkpoint small enough to run in 24GB of VRAM, alongside the full BF16 release for teams that need the un-quantized accuracy.
The 24GB checkpoint's weights take up 22.01GB (expert layers at IQ2_XXS, Mamba layers at NVFP4, attention and embeddings at Q8_0), leaving about 2GB of headroom on a 24GB card. Measured KV cache and activation overhead at batch size 1 with a 128K context adds roughly 0.87GB on top of that, which is why 24GB is the floor, not a card with room to spare.
It depends on which serving path you use. NVIDIA's own vLLM recipe for the 24GB checkpoint pins Blackwell-specific b12x kernels and is documented as deployed for SM120/SM121 architectures, meaning an RTX 5090 or a B200/B300-class card, not an RTX 4090. If you serve through llama.cpp or Ollama instead of vLLM, a 24GB RTX 4090 runs the same checkpoint, because neither of those runtimes depends on the Blackwell-only kernel path.
QLoRA fits comfortably. DGX Spark's unified pool reports about 119.68GB usable, and this site's general fine-tuning formula puts a QLoRA run on a 70B-class model at roughly 52GB, which is the same parameter-count bracket Bolt's 74B total sits in. Standard LoRA is the riskier call: the same formula needs roughly 159GB for a model this size, more than DGX Spark holds, so a LoRA run is a candidate for renting a GPU instead.
Yes, but only through NVIDIA's documented recipe, which requires a nightly vLLM build, tensor-parallel-size 1, max-model-len 262144, kv-cache-dtype fp8, linear-backend and moe-backend set to b12x, attention-backend FLASHINFER, and mamba-backend flashinfer. That recipe is listed as deployed for Blackwell SM120 and SM121 GPUs specifically, which is the detail that rules out an RTX 4090 for this particular serving path.
