GPTQ vs AWQ vs GGUF: Speed, Accuracy & Cost Compared (2026)

Every "GPTQ vs AWQ" post you'll find describes the Hessian math behind one and the activation-aware scaling behind the other, then declares a winner without ever touching a GPU. That's backwards. The two benchmarks behind this post, run independently on an H200 and an RTX 5090 through vLLM, show something neither format's paper mentions: swapping the serving kernel underneath the same GPTQ or AWQ weights moves throughput by up to 10.9x, while the algorithm choice accounts for a fraction of that. If you're deciding between GPTQ, AWQ, and GGUF for a production deployment, the kernel your inference engine dispatches to is the bigger lever, and this post walks through the actual numbers instead of restating each format's abstract.

TL;DR: GPTQ vs AWQ vs GGUF, Which 4-Bit Format Wins

• Kernel beats algorithm: Marlin AWQ hit 741 tokens/sec on Qwen2.5-32B-Instruct on an H200 (vLLM, 2026), a 10.9x jump over the default kernel's 68 tokens/sec.
• GPTQ's kernel gap is smaller: Marlin-GPTQ hit 712 tokens/sec against the default kernel's 277, a 2.6x jump that converges near AWQ's Marlin speed.
• Quality: same 2026 benchmark, GGUF Q4_K_M scored 6.74 WikiText-2 perplexity and 51.8% HumanEval Pass@1, best of the three quantized formats versus FP16's 6.56 / 56.1%.
• Cost: at Spheron's H200 on-demand rate of $4.80/hr (as of 06 Sep 2026), the 10.9x AWQ kernel gap is roughly a 10.9x cost-per-million-token swing.
• Verdict: pick GPTQ or AWQ by whichever's Marlin path your vLLM version favors. Compare H200 GPU rental pricing.

GPTQ vs AWQ vs GGUF: What Actually Got Benchmarked, and on What GPU

Two independent, reproducible benchmarks anchor this comparison, and this post converts both into a cost-per-token view that neither original write-up covered.

The first is a 2026 vLLM quantization benchmark run on a single NVIDIA H200 serving Qwen2.5-32B-Instruct. It compares GPTQ and AWQ across two kernel paths each, vLLM's non-Marlin default and the Marlin-accelerated path, and it also quantizes the same model to GGUF Q4_K_M and reports perplexity and HumanEval scores against an FP16 baseline, all on the same weights and the same hardware. It's the source for the complete vLLM quantization guide and benchmark data cited throughout this post.

The second is a community benchmark on an RTX 5090 running Qwen3-14B and Qwen3-8B under vLLM 0.11.0, comparing GPTQ against AWQ standalone throughput and file size on a consumer Blackwell card, without a GGUF leg.

Between them, that's one 32B model on a datacenter Hopper card and two smaller models on a consumer Blackwell card, both routed through vLLM, so the kernel effect isolates cleanly from the noise you'd otherwise get from different hardware, different model sizes, and different frameworks all changing at once.

How Each Format Actually Quantizes Weights

The three formats solve the same problem, shrinking 16-bit weights down to roughly 4 bits each, with three different ideas about which weights can afford to lose precision.

GPTQ quantizes layer by layer using approximate second-order (Hessian) information: for each layer, it solves for the INT4 weight values that minimize reconstruction error against that layer's original output, correcting each remaining weight for the error already introduced by the ones quantized before it. In the original GPTQ paper, Frantar et al. write that "GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline." The same paper reports end-to-end inference speedups of roughly 3.25x on an A100 and 4.5x on the cheaper A6000 versus FP16. It's a one-shot, calibration-set-driven method: no retraining, but the reconstruction step is sensitive to which calibration data you feed it.

AWQ starts from a different assumption. Instead of correcting for quantization error after the fact, it decides in advance which roughly 1% of weight channels matter most, by observing activation magnitudes during a calibration pass rather than the weights' own magnitudes, then scales those channels up before quantizing everything to INT4. In the AWQ paper, Lin et al. write that "protecting only 1% salient weights can greatly reduce quantization error," and that "AWQ does not rely on any backpropagation or reconstruction, so it generalizes to different domains and modalities without overfitting the calibration set." That last property is why AWQ has a reputation for holding up better on data it wasn't calibrated on: there's no reconstruction loop to overfit in the first place. For the full AutoAWQ install and calibration walkthrough this post skips, see our AWQ quantization guide.

GGUF isn't a quantization algorithm at all, it's a file format, the one llama.cpp and Ollama use, and its most common 4-bit scheme (K-quants, like Q4_K_M) works differently from either GPU-native format. Rather than one flat INT4 scale per weight matrix, llama.cpp's own quantization documentation describes K-quants grouping weights into small super-blocks that carry more scale and minimum values per matrix than a plain INT4 grid does. That's part of why, as the numbers below show, GGUF's perplexity can beat plain GPTQ or AWQ at a similar nominal bit-width, at the cost of a larger average file size. For the quantization steps and llama.cpp server setup, see our GGUF Dynamic Quantization guide.

GPTQ vs AWQ vs GGUF: Tokens/Sec and Perplexity, the Actual Trade-off Table

Throughput, by kernel (Qwen2.5-32B-Instruct, H200, vLLM):

Quality, by format (same model, same hardware):

Why the Serving Kernel Moves the Needle More Than the Algorithm

Look at the throughput table again: AWQ's default vLLM kernel runs at 68 tokens/sec, worse than GPTQ's default kernel at 277. Swap both to Marlin and they land within 4% of each other, at 741 and 712. The gap between formats nearly disappears once the kernel is held constant, while the gap between kernels for the same format is enormous.

vLLM's own quantization documentation explains why: the Marlin kernel accelerates GPTQ, AWQ, FP8, and FP4 formats by fusing dequantization into the matrix multiply itself, and it supports GPU architectures from Turing through Hopper. Without that fusion, the default path pays for unpacking INT4 weights back toward full precision as a separate, memory-bound step before the matmul can even start. That's the step Marlin removes, and it's why AWQ's 10.9x jump is larger than GPTQ's 2.6x: AWQ's non-Marlin default in this benchmark had further to climb.

The RTX 5090 benchmark shows the same lever from a different angle, and it flips the ranking. On Qwen3-14B, GPTQ ran 6-19% faster than AWQ under vLLM 0.11.0 (74.71 tokens/sec for GPTQ against 65.92 for AWQ standalone), attributed to the gptq_marlin kernel path being more mature than awq_marlin in that specific vLLM release. That's the opposite ranking from the H200 result above, and the flip is the whole point: kernel maturity changes from one vLLM release to the next, it isn't a fixed property of either algorithm. Whichever format "wins" a benchmark today can lose the same comparison after the next vLLM upgrade patches the other kernel.

One more data point from that same RTX 5090 test worth knowing before you pick a format by file size: GPTQ's quantized weight file was larger than AWQ's at 14B (9.37 GiB versus roughly 7-8 GiB) but roughly tied, and if anything slightly smaller, at 8B (5.71 GiB versus roughly 6 GiB). End-to-end VRAM used during serving converged to within about 1 GB of each other once KV cache was added regardless. File size on disk is a weak proxy for how much VRAM you'll actually need once the server is running.

Perplexity and Task-Accuracy Retention (WikiText-2, HumanEval)

None of the perplexity gaps in the quality table above (6.56 to 6.97) will be visible in a chat transcript. HumanEval is where the picture changes: GPTQ's 45.7% (Marlin) to 46.3% (default kernel) Pass@1 is roughly 5-6 points behind GGUF and AWQ's shared 51.8%, and about 10 points behind the FP16 baseline's 56.1%. If your workload leans on code generation, that gap is the one number in this section worth taking seriously before you standardize on GPTQ for a coding assistant, independent of anything the throughput table says.

Cost Per Million Tokens: Converting the Benchmark Into a Rental Decision

Tokens per second and cost per million tokens are the same number seen from two ends. Once you know a GPU's hourly rate and its measured throughput for a given format and kernel, cost per token falls straight out of the arithmetic, and the format with the higher throughput always wins proportionally.

The Cost Formula

cost_per_million_tokens = (gpu_price_per_hour / (tokens_per_second * 3600)) * 1,000,000

Feed in an on-demand rate and a measured tokens/sec figure, and a 10.9x throughput gap becomes a 10.9x cost gap. There's no rounding to be found once you're at this stage; the kernel effect from the section above carries straight through to the invoice.

Cost Table on Spheron H200 and RTX 5090 Pricing

Neither cited benchmark ran on Spheron's own pricing, so here are the same tokens/sec figures from above run against Spheron's on-demand rates for the exact GPUs each benchmark used. Because a cost-per-token figure needs both a price and a throughput number at the same instant, and only one of those two can be a live token, the rates below are frozen to a single date rather than resolved live.

Pricing fluctuates based on GPU availability. The on-demand rates in the table above are frozen to 11 Sep 2026 because the cost-per-token figures are derived from them; live H200 and RTX 5090 rates as of 06 Sep 2026 are $4.80/hr and $0.86/hr respectively. Other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.

H100 doesn't appear in either cited benchmark, but it's Spheron's most commonly requested H100 GPU rental, so it's worth a word on why there's no extrapolated number for it here: H100 and H200 share the same Hopper streaming multiprocessors and identical Marlin kernel support, but H200 ships with substantially higher HBM3e memory bandwidth than H100's HBM3, and a dequant-fused kernel like Marlin leans on memory bandwidth. Expect H100 to land somewhat below H200's Marlin numbers on the same model, not above them, but there's no measured figure to put next to that expectation, so none is printed here. For teams on a tighter budget, L40S GPU rental is Spheron's cheapest catalog card that still runs Marlin-accelerated AWQ or GPTQ comfortably for 8B-14B-class models; neither cited benchmark tested it either, but the formula above applies the moment you have your own tokens/sec measurement.

Which Format to Pick: A Decision Framework by Constraint

Spheron's own LLM quick-guides cover vLLM, llama.cpp, LMDeploy, and four other serving stacks end to end if you want deployment steps to go with the format decision below.

• Serving on vLLM, SGLang, or TensorRT-LLM on a rented datacenter GPU: pick GPTQ or AWQ, whichever your vLLM version's Marlin path is more mature for right now, not whichever paper you read first. Check your engine's release notes for gptq_marlin and awq_marlin support before committing. For how the serving engine itself changes throughput independent of quantization format, see our vLLM vs TensorRT-LLM vs SGLang benchmarks.
• On Blackwell hardware (B200, B300, RTX 5090, RTX PRO 6000): hardware-native FP4 usually beats all three INT4 formats in this post outright, since it runs on Blackwell's FP4 tensor cores instead of a dequant-fused kernel. See our NVFP4 vs MXFP4 decision guide before defaulting to GPTQ or AWQ on this hardware.
• On H100, H200, or Blackwell where near-lossless matters more than squeezing every dollar: FP8 gets most of INT4's speedup with a much smaller accuracy hit; our FP8 quantization explainer covers when that trade beats going to 4-bit.
• Running community-quantized checkpoints across multiple formats without standardizing on one engine: Aphrodite Engine serves EXL2, GGUF, and GPTQ from a single stack, useful if you're not ready to commit to one format.
• On InternLM, Qwen3, or DeepSeek, and you want AWQ without going through vLLM at all: LMDeploy's TurboMind engine has its own native AWQ kernel path, a fourth serving option worth a bake-off.
• Unsure how much VRAM your target model needs before picking a format: start with our LLM VRAM requirements guide, then come back to this comparison once you know your tier.
• Not sure which physical GPU to rent at all: our GPU selection guide for AI inference covers the broader hardware picture beyond H100, H200, L40S, and RTX 5090.
• CPU-only, Apple Silicon, or edge deployment: GGUF is the only format of the three that runs there at all. Skip the cost-per-token math in this post entirely.

Common Mistakes When Comparing These Formats

Conflating the kernel with the algorithm. The single biggest mistake this post exists to correct: a "GPTQ vs AWQ" number that doesn't name the serving kernel isn't comparable to anything, including a different benchmark of the same two formats. Always ask which kernel path produced the number before you compare it to another one.

Reading a batch-1 number as production throughput. The standalone figures cited above measure latency for a single request. They say very little about aggregate throughput once vLLM's continuous batching is serving 32 or 64 concurrent requests at once, where GPU utilization curves look different across formats and kernels. Benchmark your own batch size before trusting any single-request number, including the ones in this post.

Comparing GGUF-via-vLLM to native llama.cpp. vLLM has its own GGUF loader, and it is not the same code path as Marlin, and not the same code path as native llama.cpp's GPU offload either. That's why this post's GGUF numbers are quality-only (perplexity, HumanEval): the cited benchmark measured GGUF's accuracy on the same weights as GPTQ and AWQ, but not its vLLM throughput, and lining up a native llama.cpp tokens/sec figure against a Marlin-accelerated GPTQ number would be comparing two different serving engines, not two quantization formats. If your target is native llama.cpp GGUF throughput specifically, get that number from our llama.cpp server deployment guide rather than assuming it lines up with the Marlin figures above.

Renting the exact hardware behind these numbers is the fastest way to stop guessing which format wins for your own model. Spin up an H200 or an RTX 5090 by the minute, run your own default-versus-Marlin kernel comparison, and settle the GPTQ vs AWQ question for your workload instead of someone else's.

RTX 5090 GPU pricing →