KV cache size grows linearly with sequence length and batch size, and at long context it becomes the thing that runs out before compute does. vLLM's PagedAttention already fixed the fragmentation problem; the remaining lever is shrinking each cached value, and that's what KV cache quantization is for. KIVI is the 2-bit method with a published, peer-reviewed asymmetric scheme worth understanding here, and it comes with a catch worth stating up front: it is not a vLLM feature, it's a Transformers patch you run alongside vLLM, not inside it.
TL;DR: How Do You Set Up KV Cache Quantization With KIVI on vLLM?
- Not inside vLLM's engine. KIVI runs as a separate Transformers serving path installed alongside vLLM, not a flag inside it.
- vLLM's native ceiling is FP8: cache shrinks to as low as 54% of BF16, no custom kernel.
- KIVI quantizes to 2-bit: keys per-channel (group 32), values per-token, last 128 tokens kept in fp16.
- Accuracy cost: about 1 point of CoQA and GSM8K loss versus FP16 on Llama-2-7B and Mistral-7B.
- Setup needs a compiled CUDA kernel: two
pip install -e .steps, package then quant extension, on Python 3.10. - Reproduce it on an A100 80GB, the GPU class KIVI's paper used; see Spheron's A100 80GB rental page.
Why KV Cache Is the Memory Bottleneck at Long Context
Model weights are fixed once a model is loaded. KV cache is not: it grows with every token generated, per sequence, per layer, per attention head. At a long enough context window or a wide enough batch, the cache outgrows the weights it's attached to, and that's the wall teams hit long before compute saturates.
This is also why the fix has multiple, non-competing layers. One approach moves KV cache off the GPU entirely once it's no longer hot, tiering it to CPU DRAM and NVMe; our NVIDIA ICMSP guide covers that path in detail. Quantization is a different lever: it shrinks what stays resident in HBM in the first place, so the two techniques stack rather than substitute for each other. The vLLM production deployment guide walks through the multi-GPU serving side of this; here we're narrowing in on just the cache compression question.
KIVI: Per-Channel Key, Per-Token Value Quantization Explained
KIVI, published as "KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache", starts from an empirical observation: keys and values don't have the same error distribution once you try to compress them, so quantizing them the same way wastes accuracy on both.
The paper's authors put it directly:
"The key cache should be quantized per-channel, i.e., group elements along the channel dimension and quantize them together. In contrast, the value cache should be quantized per-token."
The reasoning is structural. Key vectors have channel-wise outliers, meaning certain dimensions consistently carry larger magnitudes across tokens, so grouping and quantizing along the channel axis keeps each group's dynamic range tight. Value vectors don't show that same channel-wise skew; they're closer to uniformly distributed, so per-token quantization (grouping along the token axis instead) is the better fit there. Applying the wrong axis to either tensor blows up quantization error, which is why most earlier KV cache compression schemes that used a single uniform quantizer underperformed KIVI's asymmetric one.
The full paper specifies two defaults that matter for reproducing its numbers:
- Group size G=32 for per-channel key grouping: 32 channel elements are quantized together per group.
- Residual length R=128: the most recent 128 tokens of the KV cache stay in full fp16 precision, unquantized, because recent tokens are read most often and are most sensitive to compression error during ongoing generation.
Everything older than the residual window gets compressed to 2-bit. The paper's own benchmark tables cover Llama-2 (7B, 13B), Falcon-7B, Mistral-7B, Llama-3-8B, and LongChat-7B-v1.5-32K; there's no 30B+ model in the published results, so treat anything above roughly 13B as unverified territory for this specific method until you test it yourself.
Where vLLM's Native FP8 KV Cache Stops Short of INT2
Before reaching for a third-party patch, it's worth being clear about what vLLM already does natively, because for a lot of workloads it's enough. vLLM ships FP8 KV cache quantization as a first-class feature: no custom kernel, no alternate serving stack, just a dtype flag inside the engine you're already running.
The vLLM blog's own benchmarking reports FP8 KV cache reducing per-token cache cost to as low as 54% of the BF16 baseline in the best cases. That's roughly half the memory and it costs you nothing operationally: FP8 stays inside vLLM's PagedAttention scheduler, continuous batching, and everything else the engine already does well.
What FP8 doesn't get you is INT2's footprint. 8 bits per value versus 2 bits per value is a real gap, and if you're memory-bound enough that 54% of baseline still isn't enough headroom for the batch size or context length you need, FP8 has nowhere further to go inside vLLM today. So the native options in vLLM right now are FP16 (no compression) or FP8 (moderate compression), full stop. Going below that means leaving vLLM's engine, which is what KIVI actually requires.
KV Cache Quantization Setup for vLLM: Installing and Running KIVI on a 13B Model
The official repo specifies Python 3.10 for the conda environment, and the install is a two-step build:
# clone and install the KIVI package itself
git clone https://github.com/jy-yuan/KIVI.git
cd KIVI
pip install -e .
# build the custom CUDA quantization kernel
cd quant
pip install -e .That second step compiles a real CUDA extension against your installed toolkit, which is why this isn't something you can run on a locked-down managed notebook: it needs a build toolchain with write access, matching CUDA headers, and a GPU present at build time. If you're provisioning fresh hardware for this, an instance with full root access and a standard CUDA toolkit removes that friction; Spheron's on-demand and spot A100 80GB instances come with full root SSH, and per-minute billing means you're not committing to an hourly minimum just to compile a kernel and run a benchmark pass. Current rates are covered below, since marketplace prices move often enough that they're worth quoting separately from the setup steps.
Once the kernel is built, KIVI works by swapping in its patched model class, for example LlamaForCausalLM_KIVI in place of the standard LlamaForCausalLM, and running inference through Transformers exactly as you would with any other from_pretrained load, with the quantization config (group size, residual length) passed alongside it. This is the point where you're fully outside vLLM: no --kv-cache-dtype flag, no PagedAttention, no continuous batching scheduler. It's a Transformers-native serving path.
For a 13B model, that means the same generate loop the paper benchmarks: load the KIVI-patched Llama-2-13B checkpoint, set group size 32 and residual length 128 (the defaults above), and run your prompts through Transformers' standard generation API rather than vLLM's OpenAI-compatible server. If your production stack is built around vLLM's serving interface, that gap is the real cost of adopting KIVI, separate from the accuracy tradeoff below.
VRAM Before/After and What It Buys You in Batch Size
The core value proposition is straightforward arithmetic: fp16 stores 16 bits per cached value; KIVI's 2-bit path stores roughly 2 bits per value outside the residual window. That's close to an 8x reduction in the portion of the cache eligible for compression, before accounting for the residual buffer and quantization metadata (scale and zero-point per group) that add a small amount back.
KIVI's own throughput and memory experiments were run on a single NVIDIA A100 80GB GPU, per the paper. That's the same GPU class you'd provision to reproduce these numbers directly, since matching the paper's hardware means matching its memory bandwidth and capacity ceiling rather than approximating it on a different card. Whatever headroom KIVI frees up in the KV cache translates directly into either a longer context window per sequence or more concurrent sequences in the same batch: the two things KV cache capacity always trades against each other.
Where this actually pays off is the workloads that are KV-cache-bound rather than weight-bound: long multi-turn conversations, RAG pipelines with long retrieved contexts, or high-concurrency serving where dozens of sequences are each holding their own cache simultaneously. If your bottleneck is model weights rather than cache (a 70B+ model that barely fits regardless of batch size), KIVI's savings compound with weight quantization methods like AWQ rather than replacing them; see the AWQ quantization guide for the weight-side complement to this cache-side technique. They target different tensors, so stacking them is additive, not redundant.
The Accuracy Tradeoff: CoQA and GSM8K at 2-Bit
Compression at this ratio isn't free, and KIVI's own published results are the honest place to check the cost. Two tasks, CoQA (conversational QA) and GSM8K (grade-school math reasoning), show the pattern clearly:
| Model | Task | FP16 | KIVI 2-bit | Delta |
|---|---|---|---|---|
| Llama-2-7B | CoQA | 63.88 | 63.05 | -0.83 |
| Llama-2-7B | GSM8K | 13.50 | 12.74 | -0.76 |
| Mistral-7B | CoQA | 67.40 | 66.35 | -1.05 |
| Mistral-7B | GSM8K | 38.36 | 36.01 | -2.35 |
CoQA holds up close to fp16 on both models, roughly a single point of degradation. GSM8K is more sensitive on Mistral-7B specifically, a 2.35-point drop, which is worth flagging if your workload leans on multi-step arithmetic or chain-of-thought reasoning rather than retrieval-style QA. The pattern matches what you'd expect from any aggressive KV cache compression: tasks that depend on precise intermediate reasoning steps propagated through a long context are more exposed to cache-quantization error than tasks answering from a shorter, well-defined context window.
Weigh that against vLLM's own FP8 numbers from earlier: 1-2 points of degradation on reasoning tasks at roughly 54% of baseline memory, with zero setup cost beyond a flag. KIVI buys a much smaller footprint (2-bit vs 8-bit) at a comparable or slightly worse accuracy cost on reasoning-heavy tasks, plus the operational cost of leaving vLLM's engine entirely. That's the real decision: how much of your memory ceiling is the actual constraint, versus how much operational simplicity you're willing to give up to push past it.
Provisioning the GPU: A100 80GB Cost to Reproduce This
If you want to reproduce KIVI's published numbers rather than take them on faith, matching the paper's hardware is the fastest way to get a comparable baseline. As of 5 September 2026, Spheron's A100 80GB is $1.43/hr on-demand and $1.15/hr on spot, with per-minute billing and no minimum rental period, so a short build-and-benchmark session (compile the kernel, run a handful of CoQA/GSM8K passes) doesn't lock you into an hourly commitment. Deploy time is under 2 minutes and instances come with full root access, which matters specifically here since the CUDA kernel build needs a writable toolchain, not a managed notebook environment.
Pricing fluctuates based on GPU availability. The prices above are based on 5 Sep 2026 and may have changed. Check current GPU pricing → for live rates.
One honest caveat: Spheron's pricing page doesn't document CUDA toolkit or driver specifics for compiling third-party kernels like KIVI's quant extension, and that verification (matching CUDA headers, confirming your driver supports the compute capability KIVI's kernel targets) is on you on any GPU host, not something the platform promises to solve out of the box. And if your actual goal is production serving rather than reproducing a research paper, vLLM's native FP8 path is the simpler operational choice regardless of which GPU cloud you're running on, since it needs no custom kernel and stays inside the serving stack you already run. Whether KIVI is worth the extra setup step comes down to whether your bottleneck is genuinely past what FP8's roughly 54%-of-baseline ceiling can clear.
For a broader look at the sub-8-bit landscape beyond KV cache, our MXFP4 quantization guide covers the weight-side counterpart to KIVI's cache-side approach: microscaling formats and their current vLLM support status.
KIVI's asymmetric 2-bit scheme is a real ceiling past vLLM's native FP8 KV cache, reproducible on the same A100 80GB class the paper used.
Frequently Asked Questions
No. KIVI ships as a patch to Hugging Face Transformers model classes (for example LlamaForCausalLM_KIVI), not as a vLLM plugin or scheduler backend. Running it means installing the jy-yuan/KIVI repository and its custom CUDA quantization kernel, then serving through Transformers directly, separate from vLLM's PagedAttention engine and continuous batching.
On Llama-2-7B, CoQA accuracy drops from 63.88 (FP16) to 63.05 (KIVI 2-bit), and GSM8K drops from 13.50 to 12.74. On Mistral-7B, CoQA drops from 67.40 to 66.35 and GSM8K from 38.36 to 36.01. Each is roughly a 1-point drop, which the KIVI paper attributes to its asymmetric per-channel key, per-token value quantization scheme rather than a naive uniform quantizer.
vLLM's FP8 KV cache quantizes both keys and values to 8-bit floating point uniformly, cutting per-token KV cache size to as low as 54% of the BF16 baseline in the best cases, and it runs natively inside vLLM's serving engine with no custom kernel. KIVI quantizes to 2-bit integers with an asymmetric scheme, key quantized per-channel and value quantized per-token, which reaches a much smaller footprint but requires stepping outside vLLM into a Transformers-based serving path.
