L40S vs H100 for AI Inference: When the L40S Wins on Cost Per Token (2026 Decision Guide)

Choosing between L40S and H100 for inference is not as simple as comparing price-per-hour. L40S runs $1.80/hr on Spheron against $2.98/hr for H100 SXM5, and H100 delivers 2-3x more tokens per second at batch 1 and 3-5x more at batch 32+. Which one is cheaper per token comes down to one comparison: H100's throughput advantage at your batch size against its price ratio to L40S. In our worked example at the 17 Sep 2026 rates ($0.96/hr vs $2.98/hr, a 3.1x gap), L40S wins at low and moderate batch sizes, the two roughly tie at batch 32 on an 8B model, and H100 pulls ahead at batch 32 on a 14B model. L40S wins clearly on idle and low-traffic endpoints. H100 wins for 70B+ model inference on a single card, NVLink-required training, and long-context workloads.

This post works through the specs, FP8 throughput benchmarks for Llama 3.1 8B and Qwen3 14B, the cost-per-million-token math using live Spheron pricing, and a concrete decision framework for which GPU belongs in your stack. For operators comparing L40S against A100 instead, see the L40S vs A100 cost-per-token guide which runs the same analysis against the Ampere architecture.

TL;DR: When L40S Beats H100

The honest summary: L40S wins on cost per token whenever the price ratio between the two cards is larger than H100's throughput advantage, which holds at low and moderate batch sizes and on any endpoint that sits idle part of the day. At batch 32+ the two are close, and H100 pulls ahead as models grow. H100 wins outright for 70B+ model single-card inference, NVLink-required training, and long-context workloads.

Specs: L40S vs H100 Side by Side

The bandwidth difference is the number that matters most for inference. At 864 GB/s (L40S) vs 3,350 GB/s (H100 SXM5), H100 moves model weights and KV cache 3.9x faster. This is the primary throughput driver at low batch sizes.

Architecture: Ada Lovelace vs Hopper

FP8: Same Feature, Different Ceiling

Both L40S and H100 support FP8 via NVIDIA's Transformer Engine. The engine dynamically selects FP8 or BF16 precision per layer at runtime, using per-tensor scaling to maintain accuracy. For practical inference with vLLM, both GPUs support the same --quantization fp8 flag and produce equivalent output quality.

The gap is throughput capacity. H100 SXM5 delivers 3,958 FP8 TFLOPS with sparsity. L40S delivers 1,466. That is 2.7x more FP8 compute per GPU. At batch sizes where the workload is compute-bound (roughly batch 16+ for 7B-13B models), this advantage directly translates into more tokens per second.

For a deep dive into H100's FP8 Tensor Core architecture, the H100 complete datasheet covers all precision tiers and the Transformer Engine implementation.

Memory Subsystem: GDDR6 vs HBM3

H100 SXM5 uses HBM3 (High Bandwidth Memory, 3rd generation) at 3,350 GB/s. L40S uses GDDR6 at 864 GB/s. The technology difference matters for two reasons.

First, bandwidth. HBM stacks DRAM dies directly on the GPU package, enabling the wide memory interface that delivers 3.35 TB/s. GDDR6 uses conventional PCB-mounted DRAM with a standard memory bus, topping out at 864 GB/s on L40S.

Second, capacity. HBM's die-stacking architecture enables larger capacities in the GPU thermal envelope. H100 SXM5 packs 80GB of HBM3 on the package. L40S carries 48GB of GDDR6. For 70B parameter models, this 32GB gap is the decisive factor: 70B FP8 (70GB) fits on a single H100 but not on a single L40S.

At low batch sizes (batch 1-4), inference is memory-bandwidth-bound. GPU utilization is low; the bottleneck is how fast weights and KV cache can be loaded into SRAM for each forward pass. H100's 3.9x bandwidth advantage directly caps how much L40S can close the throughput gap here.

NVLink: Only H100 Has It

H100 SXM5 supports NVLink 4.0 at 900 GB/s total bidirectional bandwidth per GPU via NVSwitch. L40S is PCIe-only. For multi-GPU inference where tensors need to be exchanged between cards (tensor parallelism), NVLink makes a real difference. PCIe 4.0 provides roughly 64 GB/s bidirectional on a 16-lane slot, about 14x less bandwidth than NVLink.

For tensor-parallel inference on 70B models (where you're splitting the model across 2+ GPUs), H100's NVLink dramatically reduces communication overhead. For training runs with all-reduce gradients, NVLink is close to required for efficient multi-GPU scaling. The L40S's PCIe limitation is not a problem for single-card inference or embedding serving, but it is a meaningful constraint for multi-GPU training.

Benchmark: Cost Per Token at Batch 1, 8, and 32

The throughput figures below are representative estimates derived from community vLLM benchmarks and extrapolated from the L40S FP16 baseline (46 tok/s at batch 1, 336 tok/s at batch 8, Llama 3.1 8B) with FP8 approximately doubling throughput at memory-bound regimes and providing additional gains at compute-bound regimes. Results will vary with vLLM version, prompt length distribution, and system configuration. Use these as directional guidance, not authoritative benchmarks.

Cost formula: cost_per_million = (hourly_rate / tok_per_second) * (1_000_000 / 3600)

The $/M figures below are a worked example frozen at Spheron's on-demand rates recorded on 17 Sep 2026: L40S at $0.96/hr and H100 SXM5 at $2.98/hr, a 3.1x price ratio. Today's rates are $1.80/hr for L40S and $2.98/hr for H100 SXM5; rerun the formula with them.

Llama 3.1 8B FP8

At batch 8, L40S at ~$0.41/M tokens is about 1.26x more cost-efficient than H100 at ~$0.52/M tokens, because H100's 2.5x throughput advantage there is smaller than the 3.1x price ratio. At batch 1 and batch 32, H100's throughput advantage is about 3x, almost exactly the price ratio, so the two cost about the same per token. On an 8B model, the cost-per-token case for L40S rests on moderate batch sizes and idle time, not on raw efficiency at full load.

Qwen3 14B FP8

For 14B models, the bandwidth bottleneck is more pronounced at batch 1 and 8 because larger weights require more memory bandwidth per forward pass. L40S VRAM is sufficient (14B FP8 = ~14GB of weights, leaving 34GB for KV cache), so H100's throughput advantage is wider here than on the 8B model: about 3.25x at batch 1, 2.9x at batch 8 and 3.75x at batch 32. Against a 3.1x price ratio, that makes batch 8 a narrow L40S win, batch 1 a narrow H100 win, and batch 32 a clear H100 win at ~$0.28/M tokens against ~$0.33/M for L40S. The larger the model and the fuller the batch, the more the H100 earns back its higher rate.

The cost-per-token calculation here uses Spheron's on-demand rates recorded on 17 Sep 2026, not today's rates. Because the verdict flips when the price ratio moves past H100's throughput advantage, recompute it with live rates. Check the live GPU pricing page for current rates before making infrastructure decisions. For current H100 market news, on-demand and spot rates across providers, and monthly price updates, see our H100 news and pricing hub.

The cost-per-token figures above are a worked example at Spheron's 17 Sep 2026 rates and are not live. Check current GPU pricing → for live rates.

When L40S Wins

Low-Utilization Endpoints (7B-30B Models)

The strongest case for L40S is when your inference endpoint runs at moderate, bursty, or unpredictable traffic. You pay per GPU-hour, not per token. An H100 sitting idle at $2.98/hr costs more than an L40S sitting idle at $1.80/hr. At full utilization the two cards come out close on cost per token (see the tables above). Lower utilization is where L40S pulls clearly ahead, because an idle hour costs the full hourly rate on either card.

For a concrete example: an internal demo endpoint serving 30 requests/hour at average 200 tokens/response is generating 6,000 output tokens/hour. Both GPUs handle this trivially; neither is anywhere near saturation. Monthly GPU cost at the 17 Sep 2026 on-demand rates: L40S at $0.96 x 720 hours = $691.20. H100 at $2.98 x 720 hours = $2,145.60. L40S saves $1,454.40/month.

For a detailed guide on vLLM deployment on L40S, including configuration flags and container setup, see NVIDIA L40S for AI inference.

Image Generation (SDXL, Flux)

SDXL and Flux.1 diffusion workloads are compute-bound on the UNet/DiT backbone, not memory-bandwidth-bound in the same way as autoregressive LLMs. Both L40S and H100 complete a single SDXL inference pass in broadly similar wall-clock time, with H100 having some advantage from its higher compute density.

The practical difference is small enough that for single-tenant image generation endpoints, L40S generates images at a comparable rate to H100 for a fraction of the hourly cost. The 48GB VRAM is sufficient for SDXL at any standard resolution and for Flux.1 up to 1080p. For parallel batch image generation at high concurrency, H100's compute advantage does scale up, but most image generation deployments are latency-focused rather than batch-throughput-focused.

Embedding Serving

Embedding models (BGE, GTE, sentence-transformers) are small relative to the L40S's 48GB capacity and computationally light. A single L40S can serve millions of embeddings per hour with very low GPU utilization. Running embedding endpoints on H100 is functional but wastes most of the GPU's capabilities.

For multi-modal embedding pipelines or retrieval-augmented generation (RAG) preprocessing, L40S handles the embedding workload at $1.80/hr vs $2.98/hr for an H100. The throughput per dollar for embeddings strongly favors L40S because neither GPU is the bottleneck; the CPU-side tokenization and batching typically are. Even the L40S is more card than a pure embedding endpoint needs; our A10G vs T4 pricing comparison covers two cheaper options that are worth checking before you default to L40S for this workload.

Multi-Tenant SaaS with Per-GPU Isolation

If you're building a SaaS product that gives tenants dedicated GPU access rather than shared inference, cost per GPU-hour is your primary metric. L40S at $1.80/hr lets you offer per-GPU dedicated instances at lower price points than H100.

For 7B-14B models, one L40S per tenant is sufficient for typical serving workloads. MIG partitioning is not available on L40S, so you cannot subdivide it. But for small models, you don't need MIG; the full 48GB card handles a 14B FP8 model with plenty of KV cache headroom.

When H100 Wins

70B+ Model Inference

70B FP8 weights total 70GB. H100 SXM5's 80GB HBM3 fits the model on a single card with 10GB remaining for KV cache at short context lengths. L40S at 48GB requires two GPUs (96GB total) for 70B FP8.

For 70B+ models, a single H100 at $2.98/hr avoids the PCIe tensor-parallel overhead that 2x L40S ($3.60/hr combined) incurs. While 2x L40S is cheaper on paper, the PCIe-limited communication between cards significantly cuts throughput for large model inference, making H100's single-card performance and NVLink scaling the better choice for 70B+ workloads.

Large Batch Serving: Where H100 Catches Up

At batch sizes of 32 and above, the workload transitions from memory-bandwidth-bound to compute-bound. H100's 3,958 FP8 TFLOPS (2.7x the L40S's 1,466) pull ahead in raw throughput, delivering roughly 3x more tokens per second at batch 32 for 7B-13B models.

H100 breaks even on cost per token when its throughput advantage equals its price ratio to L40S. At the 17 Sep 2026 rates (H100 at $2.98/hr, L40S at $0.96/hr) that threshold was 3.1x. A 3x advantage on an 8B model lands right at it, and the 3.75x advantage on Qwen3 14B clears it, which is why H100 wins the batch-32 row in the 14B table. When the price ratio widens, the threshold rises and L40S takes these rows back, so check it against live rates. For teams where throughput ceiling matters more than cost per token (hard latency SLOs, burst handling), H100's raw speed can justify the higher rate either way.

Training and Fine-Tuning

NVLink is the deciding factor for multi-GPU training. H100 SXM5's NVLink at 900 GB/s total bidirectional bandwidth enables efficient all-reduce gradient synchronization across a training cluster. L40S's PCIe-only connectivity caps inter-GPU bandwidth at roughly 64 GB/s, making multi-GPU gradient synchronization roughly 14x slower. For LoRA fine-tuning of 13B-70B models on 2-8 GPU setups, H100 multi-GPU training converges significantly faster than L40S multi-GPU training.

For single-GPU fine-tuning of small models (7B with LoRA or QLoRA), L40S is perfectly capable. But anything requiring multi-GPU gradient synchronization belongs on H100.

Long-Context Inference (32K+ Tokens)

KV cache size scales linearly with sequence length, layer count, and batch size. For a 13B model at 32K context with batch 8, KV cache alone can consume 20-30GB. Combined with model weights (13GB FP8), this pushes against L40S's 48GB ceiling and requires aggressive chunked-prefill or reduced batch size to prevent OOM.

H100's 80GB HBM3 handles large KV caches comfortably, and its 3.35 TB/s bandwidth processes the attention mechanism significantly faster at long contexts. For RAG pipelines with long retrieved contexts or conversational endpoints with extended history, H100 is the safer and faster choice. For long-context inference optimization see the KV cache optimization guide.

Spheron Pricing: L40S vs H100

Live prices from Spheron:

For comparison, hyperscaler H100 rates:

Spheron aggregates compute from 5+ providers into a single marketplace, which is how it offers rates below hyperscaler prices. Both GPUs are available on-demand with per-minute billing and SSH root access.

For batch inference workloads under 30B parameters at moderate utilization, L40S GPU rental at $1.80/hr provides capable FP8 inference at the lowest available price point. Teams running 70B+ inference or multi-GPU training workloads should rent H100 on Spheron where on-demand single-GPU availability makes it straightforward to right-size the allocation. If a consumer Blackwell card is also on your shortlist for smaller models, our RTX 5090 vs H100 vs B200 comparison extends this same cost-per-token framework to that third option.

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 08 Oct 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.

Migration Patterns

Start on L40S, Add H100 When Needed

The practical approach for teams building inference infrastructure: start on L40S for development and lower-traffic serving, then add H100 nodes when throughput demands grow. L40S and H100 instances on Spheron are provisioned independently, so you can run a mixed fleet without migrating the entire deployment.

A common pattern is routing by model size at the load balancer: small and mid-size models (7B-13B) stay on L40S, while 70B+ requests route to H100. This keeps cost-per-hour low for the majority of traffic while handling large model requests on the appropriate hardware.

The L40S vs A100 comparison has a step-by-step migration playbook that also applies when moving from L40S to H100.

Mixed Fleet Strategy

For teams serving multiple model sizes, a mixed fleet lets you right-size per workload:

• L40S nodes: embedding models, SDXL/Flux image generation, 7B-13B LLM serving at moderate traffic
• H100 nodes: 70B+ model inference, high-throughput batch serving, fine-tuning jobs

Route at the load balancer by model ID and estimated batch size. For most teams, this hybrid approach delivers lower total infrastructure cost than running everything on H100 while still having H100 capacity available for workloads that justify it.

If you're weighing self-hosting against a closed API entirely, rather than choosing between L40S and H100, our Meta Muse Spark API pricing vs self-hosted LLMs breakdown runs the same cost crossover math against a managed model instead of your own GPU fleet.

Decision Checklist

Before choosing, work through these questions:

• Model size: is your primary model 70B+ parameters? If yes, H100 on a single card avoids PCIe tensor-parallel overhead that 2x L40S incurs.
• Batch size: what is your P50 batch size? Below 16 means you're memory-bandwidth-bound, where the L40S/H100 gap is proportional to bandwidth ratio (3.9x). Above 32 means compute-bound, where the gap is 2.7x TFLOPS. Compare that gap with the H100-to-L40S price ratio at today's rates: whichever is larger decides cost per token.
• Context length: are you serving prompts longer than 16K tokens at batch 8+? If yes, H100's 80GB handles KV cache headroom L40S lacks.
• GPU utilization: what is your average GPU utilization over a 24-hour window? Below 40-50% means L40S saves heavily on absolute spend; near full utilization the cost-per-token gap narrows and can flip to H100 on larger models.
• Training vs inference: is this a serving workload only? For fine-tuning on multi-GPU setups, H100's NVLink is decisive.
• NVLink requirement: do you need tensor-parallel or pipeline-parallel training across multiple GPUs? L40S has no NVLink.
• VRAM headroom: does your model plus KV cache at target context length and batch size fit in 48GB? Use the formula in howToStep 3 to verify.
• Spot tolerance: can your workload tolerate interruptions? Both H100 SXM5 (spot $2.20/hr, on-demand $2.98/hr) and L40S (spot $1.00/hr, on-demand $1.80/hr) have spot pricing on Spheron. Spot capacity can be reclaimed without notice, so use it only for interruption-tolerant workloads.

If your inference workload runs 7B-30B models at moderate traffic where GPU utilization stays below 50%, L40S on Spheron costs a fraction of H100 SXM5 on-demand per hour. For high-utilization batch serving or 70B+ models, H100's throughput and single-card capacity win out. Check live availability and pricing for both.

Spheron L40S → | H100 GPU pricing → | View all GPU pricing →