vLLM's v0.32.0 release quietly removed a hardware requirement that's been attached to NVFP4 since NVIDIA introduced the format: NVFP4 KV cache quantization now runs on SM8x (Ampere, including A100) and SM90 (Hopper, including H100 and H200), not just Blackwell's SM100. Every writeup of NVFP4 so far, including Spheron's own, has treated it as a Blackwell weight format. This release makes it something else too: a KV cache lever that works on hardware already sitting in most fleets.
TL;DR: What Changed With vLLM NVFP4 KV Cache in 0.32?
- Release: vLLM v0.32.0 widened NVFP4 KV cache support from Blackwell (SM100) to SM8x (Ampere) and SM90 (Hopper).
- Format: NVFP4 packs values into 16-element blocks with an FP8 (E4M3) scale, about 0.5 bytes per cached element versus 2 bytes for FP16 and 1 byte for FP8.
- Spheron rents A100, H100 and H200, the SM8x and SM90 hardware this release now covers, at $2.65/hr on-demand as of 29 Oct 2026.
What Is NVFP4 Quantization, and Why KV Cache Is a Different Target Than Weights
NVFP4 is a 4-bit floating point format: each value is stored as E2M1 (1 sign bit, 2 exponent bits, 1 mantissa bit), grouped into 16-element micro-blocks, with every block carrying its own FP8 (E4M3) scale factor plus a second-level FP32 scalar per tensor, per NVIDIA's own description of the format. That fine-grained, per-block scaling is what lets a format this narrow stay usable: a 16-element block is small enough that outlier values don't drag the whole group's dynamic range around the way they would in a coarser 32-element block.
Model weights and KV cache get quantized at different points in a model's lifecycle. Weights are static: a calibration pass runs once, offline, computing per-block scales against sample inputs and writing out a checkpoint that stays NVFP4 from then on. KV cache has no such luxury. It's a tensor that grows one key and value vector per token per request, for as long as that request is in flight, so quantizing it means computing a block's scale the moment it's written, not ahead of time.
That's also why cache-side NVFP4 doesn't need the hardware weight-side NVFP4 does. Our NVFP4 vs MXFP4 explainer covers that weight-side format comparison in full. Dequantizing a cached key or value vector before an attention lookup is a far smaller operation than a full matrix multiply, which is the gap vLLM 0.32 just exploited.
What Changed: NVFP4 KV Cache Support Widens in vLLM 0.32
vLLM's SM-generation naming maps directly to the GPUs this matters for: Ampere is SM 8.0/8.6, Hopper is SM 9.0, and Blackwell is SM 10.0 (SM100) for data center parts, per vLLM's own quantization documentation. The v0.32.0 release notes list NVFP4 KV cache now running on SM8x and SM12x, and on SM90, alongside a separate addition of XQA decode support for SM12x and MiniMax-M3 sparse attention for SM100. In plain terms: A100 (Ampere), H100 and H200 (Hopper) can now use the NVFP4 KV cache path, despite none of them having a single FP4 tensor core. The same release shipped from 814 commits across 363 contributors, 121 of them first-time contributors to the project.
That release also broadened cache quantization well beyond NVFP4. GLM-5.x now defaults to an FP8 KV cache on SM100, halving KV memory per rank for a 2.3-5.5% throughput gain at a 7-11% time-to-first-token cost. fp8_ds_mla on SM90 gives GLM-5.3-Flash up to 31% more output throughput in the same release. NVFP4's jump to Hopper and Ampere is one line in a release that treated cache-precision tuning as a lever across nearly every model family and GPU vendor it touched, a theme our vLLM 0.31 weight-cache daemon writeup covers from the restart-speed side of the same release cadence.
Measuring the Memory Savings: Concurrent Sequences Before and After on the Same GPU
KV cache memory scales with 2 x L x H_kv x D x S x B x bytes_per_element, where L is transformer layers, H_kv is KV heads, D is head dimension, S is context length, B is batch size, and bytes-per-element runs 2 for FP16, 1 for FP8, and 0.5 for FP4/NVFP4, as derived in Spheron's own H100 KV cache sizing guide.
Dropping bytes-per-element to NVFP4's 0.5 doesn't change that 56GB budget, since weights stay at FP16 either way. It changes what each user's slice of it costs, which is a straight division by the same ratio the format gives you:
| KV cache precision | Bytes/element | KV cache per user at 4K context | Concurrent users on 56GB budget |
|---|---|---|---|
| FP16 | 2 | ~0.52GB | ~107 |
| FP8 | 1 | ~0.26GB | ~214 |
| NVFP4 | 0.5 | ~0.13GB | ~428 |
The same ratio holds at longer context, where the gap matters more in absolute terms:
| Context length | FP16 users | FP8 users | NVFP4 users |
|---|---|---|---|
| 4,096 tokens | ~107 | ~214 | ~428 |
| 32,768 tokens | ~13 | ~27 | ~53 |
| 131,072 tokens | ~3 | ~6 | ~13 |
These are derived figures from the formula above with vLLM's default memory settings, not a measured benchmark, and they say nothing about whether your P95 latency holds at that concurrency. Treat them as a sizing ceiling to validate against real traffic, the same caveat the sizing guide above applies to its own FP16 and FP8 rows.
Accuracy Tradeoff: What NVFP4 KV Cache Costs You vs FP16 and FP8
vLLM's 0.32 release notes don't publish a dedicated accuracy delta for NVFP4 KV cache itself, so don't borrow a number that wasn't measured for this target. Weights are calibrated offline; cache is quantized live, per block, so that figure is a rough proxy at best, not a substitute for testing the cache path directly.
What the same release does quantify at the cache level is a coarser format than NVFP4: GLM-5.x's FP8 KV cache default on SM100 trades a 7-11% time-to-first-token cost for a 2.3-5.5% throughput gain. FP8 stores twice as many bits per cached value as NVFP4, so an 8-bit cache carrying a measurable TTFT cost suggests a 4-bit cache carries at least as much, until vLLM or NVIDIA publishes a number specific to it. For how much further cache compression can erode quality, our KIVI 2-bit KV cache guide covers a published, peer-reviewed 2-bit scheme that costs roughly 1-2 points of CoQA and GSM8K accuracy against FP16, well past NVFP4's 4-bit depth, against vLLM's native FP8 ceiling of around 54% of baseline cache size. Run your own evaluation against NVFP4 KV cache specifically before a production rollout.
Buying Consequence: How Many Fewer GPUs This Buys Back
Translate the concurrency table above into GPU count and the NVFP4 KV cache flag looks like a capacity decision, not just a memory optimization. A target of 400 concurrent 4,096-token-context Llama 3.1 8B sessions needs about 4 H100s at FP16 KV cache (107 each, rounding up) and about 1 at NVFP4 (428 capacity): same model, same weights precision, same GPU generation, with the only change a --kv-cache-dtype flag. That's before the accuracy work from the section above and before validating against real traffic, but it's the order of magnitude a software release just moved without anyone buying new hardware.
What that's worth depends on what's already in your fleet. Spheron's own catalog spans A100 (Ampere, SM8x), H100 and H200 (Hopper, SM90), and Blackwell's B200 and B300 on one marketplace with per-minute billing and no cross-GPU lock-in, so a team can test --kv-cache-dtype nvfp4 on an A100 or H100 node this week instead of waiting on a Blackwell allocation, then weigh the result against an H200's larger memory pool before committing to either tier. Spheron doesn't set or validate the vLLM version or kernel flags running inside a rented instance, that stays on the renter's own serving stack, and spot capacity can be reclaimed without notice, which matters for a concurrency target that assumes the GPU stays up: weigh on-demand or reserved over spot if uptime is the constraint, regardless of how much headroom NVFP4 frees. For the rest of the levers NVFP4 KV cache sits alongside, GQA, prefix caching, and other quantization options, our KV cache optimization guide covers the full set.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 29 Oct 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.
NVFP4 KV cache no longer needs a Blackwell waitlist to buy back concurrency, it needs Hopper or Ampere hardware most serving stacks are already renting.
Frequently Asked Questions
It's vLLM storing cached key and value vectors in NVIDIA's NVFP4 format, 4-bit values grouped into 16-element blocks with an FP8 (E4M3) scale per block, instead of FP16 or FP8. That cuts KV cache memory to roughly 0.5 bytes per cached element versus 2 bytes for FP16 and 1 byte for FP8, which frees VRAM for more concurrent sequences on the same GPU.
As of vLLM v0.32.0, NVFP4 KV cache runs on SM8x (NVIDIA Ampere, including A100), SM90 (NVIDIA Hopper, including H100 and H200), and SM12x (consumer Blackwell), per vLLM's own release notes. Weight-side NVFP4, by contrast, still needs Blackwell's native FP4 tensor cores (SM100, B200 and B300) to run at full speed.
vLLM's 0.32 release notes don't publish a dedicated accuracy number for NVFP4 KV cache specifically. The nearest reference points are a weight-side NVFP4 figure from NVIDIA (1% or less degradation versus FP8 on DeepSeek-R1-0528) and the same release's FP8 KV cache default for GLM-5.x, which trades a 7-11% time-to-first-token cost for 2.3-5.5% more throughput. Treat both as directional, not a substitute for running your own eval on NVFP4 KV cache specifically before a production rollout.
The block format is the same, 16-element groups with an FP8 E4M3 scale, but the target tensor isn't. Model weights are quantized once offline through a calibration pass before serving starts. KV cache is quantized continuously at serving time, one new key and value vector per token per request, which is also why it doesn't need the same FP4 tensor core hardware a weight matrix multiply does.
