LLM inference optimization is not one lever you pull. It's a three-way trade between throughput, latency, and memory, and every serving engine makes a different bet on how to balance them. vLLM, TensorRT-LLM, and SGLang are the three that matter for production LLM serving in 2026, and picking the "fastest" one off a leaderboard is the wrong question. The right question is which trade-off matches your traffic: bursty and diverse, repetitive with shared context, or a single model you'll run unchanged for months.
This post is the framework for making that call, not a fourth benchmark run. For the actual H100 numbers behind every claim below, throughput, TTFT, VRAM, and cold start across all three engines on the same hardware and model, see our vLLM vs TensorRT-LLM vs SGLang H100 benchmarks. If you've already ruled one engine out, the two-way deep dive on cost-per-token between the compiled and uncompiled paths goes further, and so does the RadixAttention-versus-PagedAttention breakdown later in this post.
What LLM Inference Optimization Actually Means: Throughput, Latency, and Memory
LLM inference optimization means allocating a fixed GPU's compute and memory across three goals that pull against each other: throughput (tokens served per second across all requests), latency (how fast any single user gets a response), and memory efficiency (how many concurrent sessions fit in VRAM before you run out). No engine maximizes all three simultaneously. Every configuration choice, batch size, KV cache layout, quantization format, decides which two you get closer to and which one you give up.
The Throughput vs Latency Tradeoff: Why You Can't Max Both
Batching is the core mechanism behind every serving engine's throughput, and it's also the reason latency degrades as load increases. Group more requests into a single forward pass and the GPU spends more of its cycles on compute instead of waiting on memory bandwidth, which raises tokens-per-second. But every request in that batch now waits for the others to finish their share of the pass, which raises the time any individual user waits.
You can see this directly in Spheron's own H100 testing on Llama 3.3 70B FP8. vLLM's throughput went from 120 tok/s at 1 concurrent request to 2,400 tok/s at 100 concurrent requests, a 20x gain from batching alone. But TTFT p50 went from 45ms to 740ms over the same range. That's the tradeoff in one dataset: more concurrency buys you more aggregate throughput at the cost of individual response time. If your product is a single-user coding assistant, you're optimizing for the left side of that curve. If it's a batch summarization pipeline processing a document queue overnight, you're optimizing for the right side. If your workload is already slow and you're not sure which side of that curve you're stuck on, our guide to diagnosing slow LLM inference walks through the most common root causes: no KV cache reuse, static instead of continuous batching, and running the wrong engine for the traffic shape.
Where Memory (KV Cache) Becomes the Real Bottleneck
Past a certain context length and concurrency, the constraint stops being raw compute and becomes VRAM. Every token generated adds an entry to the KV cache, and that cache scales with sequence length times concurrent sessions times model size. A 70B model at FP8 already occupies roughly 70GB of an 80GB H100 for weights alone, which leaves a narrow band for KV cache before you have to reduce max-model-len or max-num-seqs to avoid an out-of-memory crash mid-request.
This is why PagedAttention and RadixAttention exist in the first place: both are KV cache memory managers before they're anything else. Neither adds compute; both reclaim wasted memory so more concurrent sessions fit in the same VRAM budget. For the full math on how context length and batch size translate into GPU memory requirements, see our KV cache optimization guide.
The Optimization Levers Every Engine Pulls: Batching, Paging, Quantization, Compilation
Strip away the marketing and vLLM, TensorRT-LLM, and SGLang all pull from the same small set of levers. They just pull them in different combinations and at different points in the request lifecycle:
- Continuous (in-flight) batching: add and remove requests from the running batch every iteration instead of waiting for a full batch to form, so a fast-finishing request doesn't block a slow one.
- Paged or radix KV cache management: allocate cache memory in fixed-size blocks on demand (PagedAttention), and, in SGLang's case, reuse blocks across requests that share a prefix (RadixAttention).
- Quantization: run weights and KV cache at FP8, INT4, AWQ, or GPTQ precision to cut memory footprint and increase effective batch size. All three engines support FP8 natively; see our FP8 quantization and inference performance guide for the accuracy and throughput tradeoffs across formats.
- Speculative decoding: use a small draft model to propose multiple tokens per step, verified by the full model in one pass, cutting the number of sequential forward passes needed. All three engines support it as an add-on layer; see the speculative decoding production guide for configuration.
- Ahead-of-time compilation: TensorRT-LLM's distinguishing lever, compiling the model into a kernel graph tuned to a specific GPU, batch size, and sequence length before serving a single request, trading a one-time build cost for lower per-token overhead at runtime.
vLLM: Strengths, Weaknesses, and Ideal Use Cases
vLLM describes itself as "a high-throughput and memory-efficient inference and serving engine for LLMs," and the GitHub numbers back the description: over 89,000 stars and 2,000-plus contributors, easily the largest community of the three engines. It supports NVIDIA, AMD, and Intel GPUs, Google TPU, Intel Gaudi, and Huawei Ascend through a plugin architecture, which makes it the only one of the three not locked to a single hardware vendor.
PagedAttention and Continuous Batching Explained
PagedAttention is vLLM's core contribution: it manages the KV cache the way an operating system manages virtual memory, allocating fixed-size blocks on demand rather than reserving a contiguous chunk sized for each sequence's theoretical maximum length. That eliminates the memory fragmentation and pre-allocation waste that plagued earlier serving systems. Combined with continuous batching, this is what lets vLLM sustain high GPU utilization on bursty, unpredictable traffic without manual batch-size tuning. The peer-reviewed evaluation behind PagedAttention (Kwon et al., SOSP 2023) reports a 2-4x throughput improvement over FasterTransformer and Orca at the same latency level, the two strongest serving systems available at the time of publication.
Where vLLM Wins: Model Breadth, Zero-Compile Deploys, Multi-Vendor Hardware
vLLM's deploy path requires a single Docker run command and no compilation step. On Spheron's H100 benchmark, vLLM reached its first served request in about 62 seconds from a cold container, on par with SGLang and dramatically faster than TensorRT-LLM's compiled path. Combined with the widest model catalog of the three (hundreds of architectures, including multimodal and MoE families) and multi-vendor hardware support, vLLM is the safest default when you don't yet know your exact production traffic shape or expect to swap models. For the full multi-GPU production setup, see our vLLM production deployment guide.
Where vLLM Loses: Peak Throughput Ceiling vs a Compiled Engine
The cost of skipping compilation is a lower throughput ceiling. In Spheron's own H100 testing, TensorRT-LLM led vLLM at every concurrency level tested, from 8% faster at a single request up to 13% faster at 50 concurrent requests. That gap is the price of vLLM's flexibility: a general-purpose runtime cannot extract quite as much hardware efficiency as a kernel graph compiled specifically for your GPU, batch size, and sequence length.
TensorRT-LLM: Strengths, Weaknesses, and Ideal Use Cases
TensorRT-LLM is NVIDIA's compiler-based inference library, and it's explicitly NVIDIA-only: H100, H200, B200, GH200, L4, Jetson, and GeForce RTX hardware, nothing else. Its GitHub footprint is smaller than vLLM's, around 14,400 stars, reflecting a narrower, more specialized user base of teams already committed to NVIDIA hardware and willing to run a compilation pipeline.
Compiled Engines and In-Flight Batching Explained
Instead of running model weights through a general-purpose runtime, TensorRT-LLM compiles the model into an optimized CUDA kernel graph tuned to your specific GPU, batch size, and sequence length configuration ahead of time. That compiled engine is then served with in-flight batching at runtime, NVIDIA's term for the same continuous-batching concept vLLM and SGLang implement. NVIDIA reports that speculative decoding on top of this compiled path gives up to 3x throughput on Llama 3.3 70B, and that Llama 4 throughput exceeds 40,000 tokens/sec on a single B200, figures from NVIDIA's own benchmarks rather than independent third-party testing, so treat them as a ceiling to validate on your own hardware rather than a guaranteed number.
Where TensorRT-LLM Wins: Max Tokens/Sec on Fixed NVIDIA Hardware
On Spheron's H100 benchmark, TensorRT-LLM delivered both the highest throughput and the lowest TTFT at every concurrency level tested. The p95 gap is what shows up most in interactive applications: at 100 concurrent requests, TensorRT-LLM's p95 TTFT was 1,280ms versus vLLM's 1,450ms, a 170ms difference in the tail latency users actually notice. If you have one model in long-term production on NVIDIA hardware and throughput per GPU-hour is the number that matters to your unit economics, this is the engine built for exactly that case. For the full build pipeline, see our TensorRT-LLM production deployment guide.
Where TensorRT-LLM Loses: Compile Time, NVIDIA Lock-In, Deploy Pipeline Overhead
The tradeoff is upfront and ongoing cost. Compiling a Llama 3.3 70B FP8 engine on a single H100 took roughly 28 minutes in Spheron's testing, versus about 62 seconds for vLLM and 58 seconds for SGLang to reach their first served request. That 28 minutes is a one-time cost per model version, but it means blue-green deploys, auto-scale-from-zero, and frequent model swaps all need a compilation step built into your pipeline, not just a container pull. Add the NVIDIA-only hardware requirement, and TensorRT-LLM is the engine you choose after you've already committed to a stable model and a single hardware vendor, not while you're still experimenting.
SGLang: Strengths, Weaknesses, and Ideal Use Cases
SGLang joined the PyTorch ecosystem as an official project, a signal of how far its adoption has moved beyond its original research lab origins. Its production users include xAI, which serves Grok 3 on it, and Microsoft Azure, which uses it to serve DeepSeek R1 on AMD GPUs. AMD, NVIDIA, LinkedIn, Cursor, Baseten, Nebius, RunPod, and several universities are also listed adopters. The PyTorch engineering blog puts it plainly: SGLang "can often significantly outperform other state-of-the-art frameworks in terms of serving throughput and latency."
RadixAttention and Prefix-Sharing Explained
RadixAttention is SGLang's core mechanism, and it's best understood as PagedAttention with a reuse layer on top. Cached KV blocks are organized in a radix tree keyed by token sequence, so when two requests share a prefix, a system prompt, a few-shot example set, a RAG document, SGLang computes the attention for that shared prefix once and serves it to every request that shares it, instead of recomputing it from scratch each time. SGLang's core feature set beyond RadixAttention includes a zero-overhead CPU scheduler, continuous batching, standard paged attention, speculative decoding, tensor parallelism, chunked prefill, structured output generation, and FP8, INT4, AWQ, and GPTQ quantization.
Where SGLang Wins: Chatbots, RAG, Multi-Turn, Agentic Workloads
The prefix-sharing payoff is largest exactly where you'd expect: workloads with repeated context. In Spheron's own prefix-heavy testing (shared system prompts and RAG context), SGLang delivered a 37% lower TTFT p50 and a 41% lower p95 than vLLM at 50 concurrent requests. Independent benchmarking backs the same pattern in a different setting: RunPod's vLLM vs TensorRT-LLM comparison measured SGLang at roughly 16,200 tokens/sec against vLLM's roughly 12,500 on Llama 3.1 8B with prefix-heavy traffic, a 29% gap attributed almost entirely to prefix cache reuse. For chatbots with long system prompts, RAG pipelines that reuse retrieved context across queries, and multi-turn agent loops, that's a real, measurable win, not a marginal one. See our vLLM vs SGLang 2026 benchmark for the full prefix-heavy TTFT breakdown, including what it takes to route only the prefix-heavy slice of your traffic to SGLang rather than moving your whole fleet.
Where SGLang Loses: Benefit Disappears on Unique-Prompt Traffic
RadixAttention has nothing to reuse when there's nothing shared. On Spheron's unique-prompt H100 benchmark, SGLang's throughput advantage over vLLM narrowed to roughly 3.8%, well within the noise of normal benchmark variance. The same RunPod benchmark shows the identical collapse on unique prompts: without shared prefixes, vLLM and SGLang perform nearly identically, with SGLang ahead by only 1-4% at lower concurrency. If your traffic is one-shot creative generation, unique search queries, or independent summarization jobs with no repeated context, SGLang's headline advantage simply doesn't apply to you, and you should evaluate it on raw throughput and TTFT alongside the other two, not on its prefix-caching reputation.
Choosing an Engine for LLM Inference Optimization: A Decision Framework
Throughput, TTFT, and Cold-Start Summary
At 50 concurrent requests on Spheron's H100 benchmark (Llama 3.3 70B FP8, unique prompts), the three engines land like this:
| Engine | Throughput (50 req) | TTFT p50 (50 req) | Cold Start |
|---|---|---|---|
| vLLM | 1,850 tok/s | 380 ms | ~62 sec |
| TensorRT-LLM | 2,100 tok/s | 340 ms | ~28 min |
| SGLang | 1,920 tok/s | 360 ms | ~58 sec |
That's a summary, not the full picture. The complete concurrency curve (1, 10, 50, and 100 requests), TTFT p95, and peak VRAM usage across all three engines are in the full H100 benchmark post linked at the top of this article, which also covers how these numbers shift with prefix-heavy versus unique-prompt traffic.
A Decision Framework: Match the Engine to Your Traffic Pattern, Not the Leaderboard
None of these engines is wrong. Each is tuned for a workload shape, and the fastest one in a benchmark table is not necessarily the fastest one for your actual traffic.
| Your situation | Best fit |
|---|---|
| Diverse, mostly unique prompts; need broad model support and fast iteration | vLLM |
| One stable model in long-term production; every last token/sec matters | TensorRT-LLM |
| Shared system prompts, RAG context, or multi-turn agents (60%+ prefix overlap) | SGLang |
| Need AMD, TPU, or other non-NVIDIA hardware | vLLM |
| Need to be serving requests in under 2 minutes from a cold container | vLLM or SGLang |
| Mixture-of-experts models (DeepSeek, Llama 4) with expert parallelism | vLLM or SGLang, see our MoE inference optimization guide |
| Prototyping or evaluating a model before committing to a production stack | vLLM, or a lighter path like Ollama for local testing |
The working rule of thumb: measure your prefix overlap before you measure anything else. Above roughly 60% shared-prefix traffic, SGLang's RadixAttention pays for itself. Below it, vLLM's --enable-prefix-caching flag closes most of the remaining gap without a framework switch, and TensorRT-LLM is worth the compile-time cost only once your model and traffic pattern have both stabilized.
Can You Mix Engines by Workload?
Yes, and it's increasingly the practical answer rather than an edge case. Teams running mixed traffic, high-throughput batch jobs alongside an interactive chat or RAG path, get better results running two engines behind a router than forcing one engine to serve both shapes well. A common split is TensorRT-LLM or vLLM for the batch and background workload, with SGLang handling the prefix-heavy interactive path, and a routing layer in front deciding which backend a request hits based on its shape rather than a fixed assignment. Our LLM inference router guide covers how to build that routing layer on GPU cloud infrastructure.
None of this changes if you're already deep into speculative decoding or FP8 quantization on top of any of these three; those levers compound with the engine choice rather than replace it. A compiled TensorRT-LLM engine with speculative decoding enabled still outperforms an unoptimized vLLM deployment, but a well-tuned vLLM deployment with speculative decoding on can beat a default TensorRT-LLM engine that hasn't turned it on.
Whichever engine you land on, it runs on the same GPU either way. H100 SXM5 instances on Spheron start at $2.65/hr on-demand.
Pricing fluctuates based on GPU availability. The price above is based on 19 Aug 2026 and may have changed. Check current GPU pricing → for live rates.
All three engines deploy the same way on Spheron: provision an H100, SSH in, and follow the framework's quick-guide to get it serving in minutes. The full Docker commands for vLLM, TensorRT-LLM, and SGLang are in the H100 benchmark post linked above, and the per-framework docs are at Spheron's LLM quick-guides.
Whichever engine wins your decision framework, the GPU underneath it sets the ceiling on all three of these numbers. Spheron runs bare-metal H100 capacity with no hypervisor tax, billed per minute.
Frequently Asked Questions
It means deciding how to split a fixed GPU's compute and memory across three competing goals: throughput (tokens served per second across all users), latency (how fast one user gets a response), and memory efficiency (how many concurrent sessions fit in VRAM). No engine maximizes all three at once. Batching, KV cache layout, quantization, and compilation are the levers, and each one trades some of one goal for more of another.
There's no single best engine, only a best fit for your traffic. On Spheron's own H100 benchmark with Llama 3.3 70B FP8, TensorRT-LLM led throughput at every concurrency level (8% to 13% ahead of vLLM), SGLang led on TTFT for prefix-heavy traffic (37% lower p50 at 50 concurrent requests), and vLLM led on model breadth, hardware support, and deploy speed. Pick based on your workload shape, not the leaderboard.
PagedAttention (vLLM) manages the KV cache like OS virtual memory: it allocates fixed-size blocks on demand and frees them when a request finishes, so you don't reserve memory for a sequence's maximum possible length upfront. RadixAttention (SGLang) builds on the same page-based idea but organizes cached blocks in a radix tree keyed by token sequence, so requests sharing a prefix, a system prompt, a RAG document, reuse the same cached activations instead of recomputing them.
Yes, and more teams are doing it deliberately rather than standardizing on one engine for everything. A common pattern is vLLM or TensorRT-LLM for high-throughput batch jobs and SGLang for the interactive, prefix-heavy chat or RAG path, with a router in front deciding which backend a request hits based on its shape. See our LLM inference router guide for how that routing layer works in practice.
It still matters, but less than picking the wrong engine for your traffic shape in the first place. Quantization (FP8, INT4, AWQ, GPTQ) and speculative decoding are available on all three engines and each adds a further 1.5-5x depending on the technique and model. They compound with, rather than replace, the engine-level decision: a compiled TensorRT-LLM engine with speculative decoding still beats an unoptimized vLLM deployment, but a well-tuned vLLM deployment with speculative decoding on can beat a default TensorRT-LLM engine without it.
