Does quantization speed up inference? The honest answer is: sometimes, and the condition that decides it has nothing to do with the bit count on the label. Real benchmarks show INT4 models running slower than the FP16 or BF16 baseline they were supposed to replace, on the same model, same GPU, same framework. The gap isn't a benchmarking fluke. It shows up whenever a workload stops being memory-bound and dequantization overhead moves onto the critical path instead of hiding behind memory latency.
TL;DR: Does Quantization Speed Up Inference?
- Not always: Red Hat AI's async benchmark ran an INT4 8B model at 5.3 QPS on docstring generation versus 5.6 QPS for BF16, on the same A6000, a real regression.
- The variable is compute-bound vs. memory-bound, not bit-width. On an H100, decode turns compute-bound at roughly 74 concurrent tokens for INT4 versus roughly 295 for BF16.
- The kernel matters as much as the format. A naive GPTQ kernel hit 276 tok/s against a 461 tok/s FP16 baseline on Qwen2.5-32B; Marlin's optimized kernel hit 712 tok/s on the same weights.
- Spheron bills H100 SXM5 by the minute, no minimum term, at $2.64/hr on-demand as of 18 Sep 2026, so testing a format costs a few dollars, not a procurement cycle.
The Assumption: Lower Precision Always Means Faster Inference
The logic behind "quantize it and it'll run faster" sounds airtight on paper. INT4 weights are a quarter the size of BF16 weights. A GPU serving an LLM spends a large share of its time reading those weights out of HBM, so fewer bytes should mean less time waiting and more tokens out the other side. It's the same instinct behind assuming doubling batch size doubles throughput, and it holds in exactly the same narrow set of conditions.
The assumption is correct precisely when a workload is memory-bound: the GPU is idle waiting on data, not busy doing math. Our memory wall post covers why single-request decode lands there by default. A 70B model at FP16 moves roughly 140 GB of weights per token generated, and that transfer time sets a floor on latency no amount of extra compute touches. Shrink those weights to INT4 and the floor drops with them, because there's genuinely less to move.
What the assumption skips is that "less to move" and "less work to do" are different claims. Quantization changes the first number. It does nothing to the second, and once a workload's bottleneck shifts, the second number is the one that decides speed.
Does Quantization Speed Up Inference? Not in These Benchmarks
Two independent benchmarks, on different hardware and different frameworks, land on the same result: INT4 running behind FP16 or BF16 on the same model.
An 8B Model on an A6000, Serving Async: INT4 Loses to BF16
In Red Hat AI's asynchronous serving benchmark of an 8B model on a single A6000, INT4 weight-only quantization (W4A16) came in slower than the BF16 baseline on every one of three tasks measured, while INT8 (W8A8) beat both:
| Task | BF16 | INT8 (W8A8) | INT4 (W4A16) |
|---|---|---|---|
| Docstring generation | 5.6 QPS | 7.7 QPS | 5.3 QPS |
| RAG | 4.4 QPS | 6.1 QPS | 4.1 QPS |
| Instruction following | 11.8 QPS | 16.5 QPS | 11.2 QPS |
INT4 isn't close to BF16 here, it's behind it, by a small but consistent margin across every task in the table. INT8 does what the "lower precision, faster inference" intuition predicts. INT4 doesn't. Same model, same GPU, same serving setup, and the format that shrinks the weights the most is the one that runs the slowest of the three.
The Kernel Problem: Naive Quantized Kernels Run Below the FP16 Baseline
A second data point makes the picture sharper: how much of "INT4 is slower" is actually about the bit-width, and how much is about which kernel executes it.
That's not a small implementation detail. It's the difference between a quantization format that looks like a regression and one that looks like a 1.5x-plus win, using identical weights. Jarvislabs measured the Marlin kernel delivering a 2.6x speedup for GPTQ over the naive path in the same comparison. If a vLLM deployment quantizes a model to GPTQ or AWQ and doesn't confirm which kernel is actually serving those weights, "INT4 is slower than FP16" is a very plausible outcome, and it has nothing to do with 4-bit math being inherently worse.
Why It Happens: Dequantization Sits on the Critical Path
Both benchmarks point at the same mechanism. Weight-only INT4 quantization shrinks the number of bytes moved from HBM, but it doesn't touch the number of floating-point operations the GPU has to perform. Weight-only INT4 quantization reduces memory traffic without reducing FLOPs, and during prefill, which runs at arithmetic intensities of hundreds to thousands of FLOPs per byte, shrinking the weight bytes buys nothing while still paying the cost of unpacking them.
The Architectural Mismatch: 4-Bit Weights, 16-Bit Tensor Cores
The unpacking cost is the part the "fewer bits, less work" framing leaves out entirely. GPU Tensor Cores don't execute 4-bit matrix multiplication as a native op. INT4 weights get dequantized, expanded back out to BF16, FP8, or INT8, before the matmul that actually produces a token runs. That dequantization step is real work sitting directly on the critical path.
When a kernel is memory-bound, that extra step is close to free: the GPU has spare compute cycles sitting idle while it waits on the next HBM read anyway, so slotting dequantization into that gap costs almost nothing, and the bandwidth saved on the read dominates. When a kernel is compute-bound, there's no spare capacity left to absorb it. Dequantization competes directly with the matmul for the same Tensor Core cycles, and the bytes saved on the read no longer matter because the GPU was never waiting on the read in the first place. That's the exact condition prefill sits in almost by default, and it's the condition high-concurrency async serving drifts into as more requests share the same batch.
Compute-Bound vs Memory-Bound: The Variable That Actually Decides It
This is the one number worth checking before trusting any bit-width's marketing. Our roofline model post walks through the mechanics in full, but the short version: every GPU has a ridge point, an arithmetic intensity (FLOPs per byte) above which it's compute-bound and below which it's memory-bound. Quantization doesn't remove that ridge point. It moves where a given workload sits relative to it, and not always in the direction the intuition expects.
On an H100, the crossover to compute-bound during decode lands at roughly 74 concurrent tokens for INT4 weights, versus roughly 295 for BF16 weights. INT4's smaller weight footprint means the GPU's compute units catch up to the memory system sooner as concurrency rises, so INT4 crosses into compute-bound at a lower batch size than BF16 does on the identical GPU. Push concurrency past that point and INT4 is now paying its dequantization tax with no spare compute cycles to absorb it in, which is the exact mechanism behind the Red Hat AI numbers above and behind our batch size myth post, which shows the same compute/memory ceiling deciding throughput independent of quantization.
Bit-width and concurrency aren't separate levers here. They're two inputs to the same crossover point, and INT4 shifts that point closer, not further away.
When INT4 Does Speed Things Up (and When It Reliably Doesn't)
Given a real workload rather than a slide, INT4 quantization is worth trusting in some conditions and worth testing carefully in others:
- Reliably helps: single-stream or lightly batched decode, where the request stays memory-bound and a smaller weight read genuinely cuts wait time. Interactive, low-concurrency serving is the common case this covers.
- Reliably helps for a different reason: fitting a model onto fewer or smaller GPUs. A 70B model that needs two GPUs at BF16 can fit on one in INT4. That's a capacity win independent of throughput, and it holds even in the cases below where speed doesn't improve.
- Depends on the kernel, not the format: AWQ and GPTQ both ship pre-quantized checkpoints, but a naive kernel serving either one can land below FP16 throughput, as the Marlin comparison above shows. The same caution applies to GGUF's K-quant formats: the compression ratio on disk says nothing about which kernel actually executes the matmul.
- Rarely helps: prefill-heavy workloads. Prefill's arithmetic intensity is already well above almost any GPU's ridge point, so it's compute-bound regardless of weight format, and INT4 adds a dequantization step with no bandwidth win to offset it.
- Often hurts: high-concurrency async serving. This is where the Red Hat AI numbers land, and where the crossover math above explains why: concurrency alone can push a workload compute-bound, and quantization gets there sooner than a higher-precision baseline would.
The Buying Consequence: Test Your Own Workload Before Trusting the Bit-Width
The honest takeaway isn't "avoid INT4." It's that the speedup is conditional on a property of your specific deployment, not a property of the format, and that property is cheap to check before you commit a fleet to it. Profile your model at your real concurrency, on your real serving stack, with the kernel you actually plan to run, and compare it against the FP16 or BF16 baseline on the same hardware. If the workload is memory-bound at that concurrency, INT4 should win. If it isn't, don't be surprised when it loses.
That test is exactly the kind of thing per-minute billing is good for. Spheron rents H100 SXM5 at $2.64/hr on-demand ($2.13/hr on spot) as of 18 Sep 2026, aggregated across 5+ providers with no minimum rental period, so a same-GPU, same-model comparison between a naive and an optimized kernel, or between BF16 and INT4 at your actual concurrency, is an afternoon's rental, not a hardware purchase. Bare-metal access also matters here specifically: swapping in a Marlin-built kernel to compare against a naive one, the difference that decided the Jarvislabs result above, needs the ability to touch the serving stack directly, not just call an API.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 18 Sep 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.
What Spheron doesn't do is run that benchmark for you or pick the quantization format on your behalf. The compute-bound crossover point on your model, at your concurrency, with your kernel, is a property of the GPU and the code path, not of whoever you rented the GPU from. Run the comparison yourself before the bit-width gets to decide anything for you.
Quantization's speedup is a bet on your workload staying memory-bound, and the only way to know if that bet holds is to measure it on your own model and concurrency.
Frequently Asked Questions
No. Quantization only speeds up inference when the workload is memory-bound, meaning the GPU is waiting on weight reads rather than saturating its compute units. In Red Hat AI's asynchronous serving benchmark, an INT4 8B model ran at 5.3 QPS on docstring generation against 5.6 QPS for the BF16 baseline, a real slowdown, not a rounding error. Once concurrency pushes a workload toward compute-bound, the bytes quantization saves stop mattering and the cost of unpacking those bytes back to a usable format starts to dominate.
Because GPU Tensor Cores don't execute 4-bit matrix multiplication natively. INT4 weights have to be dequantized back to BF16, FP8, or INT8 before the matmul runs, and that unpacking step sits on the same critical path as the computation it's meant to speed up. When the model is already compute-bound, that extra step is pure overhead with no bandwidth savings left to offset it.
A GPU step is memory-bound when it spends more time moving weights and KV cache from HBM than doing math on them, and compute-bound when the Tensor Cores are the bottleneck instead. On an H100, the crossover to compute-bound during decode happens at roughly 74 concurrent tokens for INT4 weights versus roughly 295 for BF16, so quantization narrows the concurrency window where it actually helps rather than removing the ceiling entirely.
At low concurrency, single-stream or lightly batched decode, where the workload stays memory-bound and a smaller weight read genuinely means less time waiting on HBM. It also reliably helps when the goal is fitting a larger model on fewer GPUs rather than raw throughput. It stops helping, and can actively hurt, once batch size or concurrent requests push the same workload into compute-bound territory, or when the serving stack runs a naive quantized kernel instead of an optimized one like Marlin.
