If you searched "vllm vs tensorrt-llm," you're almost certainly choosing between the two engines most teams shortlist for serving open-weight LLMs in production. Both are free, both speak the OpenAI-compatible API, and both run on the same NVIDIA hardware, so the decision comes down to three things: raw throughput, how much setup pain you're willing to absorb, and what it costs per million tokens once the meter is running. We ran both on the same H100 80GB with the same model to answer all three.
This is the tight, two-engine version of our broader vLLM vs TensorRT-LLM vs SGLang benchmark, which adds a third engine for teams whose workloads have heavy prompt-prefix reuse. If that's not you, this post has everything you need.
Quick Verdict: vLLM vs TensorRT-LLM at a Glance
TensorRT-LLM wins on throughput and latency once its engine is compiled. vLLM wins on time-to-production and hardware flexibility. Neither wins on cost per token outright, it depends on your concurrency and how often you change models.
| Metric | vLLM | TensorRT-LLM |
|---|---|---|
| Throughput (50 concurrent req) | 1,850 tok/s | 2,100 tok/s |
| TTFT p95 (100 concurrent req) | 1,450 ms | 1,280 ms |
| Cold start | ~62 seconds | ~28 minutes (one-time compile) |
| Hardware support | NVIDIA, AMD ROCm, Intel, TPU, AWS Neuron | NVIDIA only |
| Setup complexity | Low, one Docker flag for FP8 | High, quantize then compile pipeline |
| Best for | Multi-model fleets, fast iteration, mixed hardware | Fixed model in long-term production, max throughput |
- Pick vLLM if you swap models often, run on non-NVIDIA hardware anywhere in your fleet, or need to be serving traffic within minutes.
- Pick TensorRT-LLM if one model is staying in production for months and you're willing to run a compile pipeline for the throughput and latency edge.
vLLM vs TensorRT-LLM Architecture: PagedAttention vs Compiled Engines
The throughput and latency numbers below come from two fundamentally different engineering bets. vLLM optimizes the memory manager around a fixed compute graph; TensorRT-LLM optimizes the compute graph itself for one specific hardware and workload shape.
How vLLM's PagedAttention Works
vLLM's core mechanism, PagedAttention, treats the GPU's KV cache the way an operating system treats virtual memory: instead of reserving one large contiguous memory block per request up front, it allocates KV cache in small fixed-size pages on demand, tracked with a lookup table per sequence (vLLM's original PagedAttention writeup). Older serving systems that pre-reserve a worst-case contiguous block waste 60-80% of KV cache memory to fragmentation and over-allocation; PagedAttention gets that waste under 4%.
Paired with continuous batching, which slots new requests into the running batch as older ones finish rather than waiting for a fixed batch to complete, this is what let vLLM claim up to 24x higher throughput than raw Hugging Face Transformers and up to 3.5x over Hugging Face TGI in its original benchmarks. For the deeper mechanics of both techniques, see our continuous batching and PagedAttention deep dive.
The practical upshot: vLLM's memory manager works the same way regardless of GPU, batch size, or sequence length, which is exactly why it needs no compilation step. You load weights and serve.
How TensorRT-LLM's Ahead-of-Time Compilation Works
TensorRT-LLM takes the opposite bet. Instead of a general-purpose runtime that adapts to whatever request shape shows up, it compiles your model into a kernel graph tailored to a specific GPU, a specific max batch size, and a specific max sequence length, using NVIDIA's TensorRT compiler. The build pipeline runs the model through a quantization pass (converting weights to FP8 or another target precision), then trtllm-build generates the fused, hardware-specific kernel graph and writes it to disk as a compiled engine binary.
Naveen Rao, VP of Engineering at Databricks, described the tradeoff after using it in production at MosaicML: "TensorRT-LLM is easy to use, feature-packed with streaming of tokens, in-flight batching, paged-attention, quantization, and more, and is efficient. It delivers state-of-the-art performance for LLM serving using NVIDIA GPUs and allows us to pass on the cost savings to our customers" (NVIDIA developer blog). That efficiency is real, NVIDIA's own H100 benchmarks with in-flight batching showed an 8x total throughput increase over A100 on GPT-J-6B and a 4.6x speedup on Llama 2 70B versus A100. But it's paid for with a build step you have to plan your deployment pipeline around, and a kernel graph that's tied to the exact configuration you compiled for.
TensorRT-LLM v1.0 and later also ship a PyTorch backend, now the default, that loads Hugging Face weights directly and skips the compiled-engine step entirely. It trades some peak throughput for a cold start closer to vLLM's, which is worth knowing if the 28-minute compile is the dealbreaker keeping you on vLLM. For the full build pipeline including the quantize and compile commands, see our TensorRT-LLM production deployment guide; the equivalent multi-GPU walkthrough for vLLM is in vLLM Production Deployment 2026.
Throughput and Latency Benchmarks on the Same GPU
We ran both engines on a single Spheron H100 SXM5 80GB bare-metal instance, serving meta-llama/Llama-3.3-70B-Instruct at FP8 precision (vLLM v0.18.0, TensorRT-LLM v1.2.0 with a compiled FP8 engine). Load was generated with an async client across 200 prompts, 512 average input tokens and 256 average output tokens, at four concurrency levels. Full methodology, plus SGLang's numbers on the same run, is in the three-way benchmark post.
Output Throughput at Increasing Concurrency
| Concurrency | vLLM | TensorRT-LLM | TRT-LLM advantage |
|---|---|---|---|
| 1 req | 120 tok/s | 130 tok/s | +8% |
| 10 req | 650 tok/s | 710 tok/s | +9% |
| 50 req | 1,850 tok/s | 2,100 tok/s | +13% |
| 100 req | 2,400 tok/s | 2,780 tok/s | +16% |
TensorRT-LLM's compiled engine led at every concurrency level we tested, and the gap widened as load increased, from 8% at a single request to 16% at 100 concurrent requests. That's the compiled kernel graph doing its job: it's most efficient exactly where GPU scheduling gets hardest, at high batch occupancy.
Time to First Token (TTFT) p50/p95
| Concurrency | vLLM p50 | vLLM p95 | TRT-LLM p50 | TRT-LLM p95 |
|---|---|---|---|---|
| 1 req | 45 ms | 68 ms | 38 ms | 55 ms |
| 10 req | 120 ms | 195 ms | 105 ms | 170 ms |
| 50 req | 380 ms | 720 ms | 340 ms | 620 ms |
| 100 req | 740 ms | 1,450 ms | 680 ms | 1,280 ms |
TTFT is what a user actually feels as "responsiveness." At 100 concurrent requests, TensorRT-LLM's p95 TTFT of 1,280ms beats vLLM's 1,450ms by 170ms, enough to notice in an interactive chat product under load. At low concurrency the gap shrinks to single-digit milliseconds, so if you're rarely running more than a handful of simultaneous requests, this line item shouldn't drive your decision.
Setup Complexity and Hardware Support
The benchmark table tells you which engine is faster on paper. It doesn't tell you what it costs to get there, and for a lot of teams that's the number that actually decides this.
NVIDIA-Only vs Broader Hardware
TensorRT-LLM's support matrix is NVIDIA GPUs, full stop: Ampere, Ada Lovelace, Hopper, Blackwell, and Grace Hopper/GB200 NVL72, on Linux x86_64 or aarch64 only (official support matrix). There's no ROCm, TPU, or CPU path, and there won't be, since it's built as a compiler around NVIDIA's own TensorRT toolchain.
vLLM's installation docs confirm NVIDIA CUDA, AMD ROCm (MI200/MI300/MI350 series and Radeon RX GPUs), and Intel GPUs as supported backends (vLLM installation docs). The PyTorch Foundation's own announcement extends that list further, describing vLLM's "Comprehensive Hardware Compatibility: Runs on NVIDIA GPUs through Blackwell, with official support for AMD, Google TPU, AWS Neuron, Intel CPU/XPU/HPU, and ARM" (PyTorch Foundation announcement). vLLM became a PyTorch Foundation-hosted project in May 2025, with the foundation citing over 46,500 GitHub stars and more than 1,000 contributors at the time, and that PyTorch backbone is exactly what gives it hardware reach TensorRT-LLM structurally can't match. If you're weighing whether to standardize on CUDA or keep an AMD option open across your fleet, our ROCm vs CUDA GPU cloud guide goes deeper on that tradeoff.
In practice: if every GPU you'll ever run on is NVIDIA, this isn't a differentiator. If there's any chance you diversify hardware later, either for cost or for capacity reasons, vLLM keeps that door open and TensorRT-LLM closes it.
Cold Start and Engine Compilation Time
This is where the two engines diverge hardest. TensorRT-LLM's compiled-engine path took about 28 minutes to build our Llama 3.3 70B FP8 engine on a single H100, a one-time cost per model version that's reused on every subsequent restart (reloading the saved engine takes roughly 90 seconds). vLLM went from a cold container to serving its first request in about 62 seconds, no compile step at all.
That 28-minute tax isn't a flaw, it's the price of the compiled kernel graph's throughput advantage. But it changes how you have to operate: blue-green deploys, scale-to-zero autoscaling, and frequent model swaps all get harder when every new model version needs a 28-minute build before it can take traffic. Teams running one stable model for months absorb this cost once and never think about it again. Teams that update models weekly, or scale workers up and down with demand, feel it constantly. If the compile step is what's holding you back from TensorRT-LLM's throughput numbers, the PyTorch backend (default since v1.0) skips the build entirely and lands closer to vLLM's cold-start time, at some cost to peak throughput.
Cost Per 1M Tokens at Scale
Cost per token is throughput divided into GPU-hourly rate, so the formula is: (GPU $/hr) / (output tokens/sec x 3,600) x 1,000,000. Pricing on Spheron for a single H100 SXM5 currently starts from $5.01/hr on-demand and $2.91/hr on spot per GPU; H100 PCIe starts from $3.30/hr on-demand and $2.20/hr on spot per GPU (live rates checked via Spheron's GPU pricing API on 13 Aug 2026).
Using our 50-concurrent-request throughput numbers on an on-demand H100 SXM5:
| Engine | Throughput (50 req) | GPU cost | Cost per 1M output tokens |
|---|---|---|---|
| vLLM | 1,850 tok/s | $5.01/hr | $0.75 |
| TensorRT-LLM | 2,100 tok/s | $5.01/hr | $0.66 |
At sustained high concurrency, TensorRT-LLM's throughput edge translates to roughly 12% cheaper output tokens on the same hardware. That gap holds or widens as concurrency climbs toward 100 requests, where TensorRT-LLM's throughput lead grows to 16%. Two things this table doesn't capture: the one-time 28-minute compile isn't free engineering time, and if your concurrency rarely gets past 10-20 requests, the cost gap between the two engines shrinks toward the low single digits, at which point engineering overhead is the bigger line item, not GPU-seconds. For the fuller economics picture, spot pricing strategy, and a cross-provider cost comparison, see AI Inference Cost Economics in 2026; if you're benchmarking against a managed API instead of self-hosting, NVIDIA NIM pricing vs self-hosted vLLM cost uses the same H100 FP8 baseline for that comparison.
Pricing fluctuates based on GPU availability. The prices above are based on 13 Aug 2026 and may have changed. Check current GPU pricing → for live rates.
Which One to Choose by Team Size and Deployment Target
- Small team, multiple models, fast iteration: vLLM. No compile pipeline to maintain, broadest model support, and you can swap a model in production without a 28-minute wait.
- One model, stable for months, throughput-critical: TensorRT-LLM. Absorb the compile cost once, then take the 8-16% throughput edge and lower cost per token for as long as that model stays in production.
- Mixed or non-NVIDIA hardware fleet: vLLM, since it's the only one of the two that runs on ROCm, TPU, or Neuron alongside CUDA.
- Auto-scaling from zero or frequent blue-green deploys: vLLM, or TensorRT-LLM's PyTorch backend if you want to stay on NVIDIA's compiler stack without the build tax.
- Squeezing out the last few points of latency at 100+ concurrent requests: TensorRT-LLM's compiled engine path, full stop.
Engine choice isn't the only lever, either. Both frameworks support speculative decoding as an add-on layer that can compound with whichever engine you pick for another 2-5x latency reduction, see our speculative decoding production guide for the vLLM and TensorRT-LLM configuration details. And if SGLang's RadixAttention prefix caching is relevant to your workload (chatbots, RAG, multi-turn agents with shared context), the vLLM vs SGLang 2026 benchmark covers that third option in the same depth as this post covers these two.
Both engines deploy the same way on Spheron: provision an H100 instance, SSH in, and follow the vLLM or TensorRT-LLM quick-guide to get either serving in minutes.
Whichever engine you land on, the GPU underneath it decides your real throughput ceiling. Spheron runs both vLLM and TensorRT-LLM on bare-metal H100 capacity with no hypervisor tax, billed per minute.
Frequently Asked Questions
Not on raw throughput. On the same H100 running Llama 3.3 70B at FP8, TensorRT-LLM's compiled engine beat vLLM at every concurrency level we tested: 8% faster at 1 request and 13% faster at 50 concurrent requests. vLLM wins on time-to-production, since it needs no compilation step and starts serving in about a minute.
At 100 concurrent requests on our H100 benchmark, TensorRT-LLM's p95 TTFT was 1,280ms versus vLLM's 1,450ms, a 170ms gap. At low concurrency the difference shrinks to single-digit milliseconds and rarely matters for user-perceived latency.
No. TensorRT-LLM only runs on NVIDIA GPUs (Ampere, Ada Lovelace, Hopper, Blackwell, and Grace Hopper/GB200 NVL72) on Linux x86_64 or aarch64. vLLM supports NVIDIA CUDA plus AMD ROCm, Intel GPUs, Google TPU, AWS Neuron, and Intel Gaudi HPUs, which matters if you want to avoid being locked to one vendor's hardware roadmap.
Compiling a Llama 3.3 70B FP8 engine for TensorRT-LLM on a single H100 takes about 28 minutes, a one-time cost per model version that's reused on every restart. vLLM has no compilation step and reaches its first served request in roughly 62 seconds from a cold container.
TensorRT-LLM's higher throughput per GPU-hour usually produces a lower cost per million output tokens at sustained, high-concurrency load, once you amortize the one-time compile cost. vLLM often wins on total cost when you count engineer time, since there's no compilation pipeline to build and maintain, and no re-compile tax every time you swap models.
