Inference Optimization

46 guides in this topic

Once a model is deployed, the real work starts: making it fast and cheap enough to run in production. This is the biggest cluster on the blog because inference optimization has the most moving parts: KV cache management (PagedAttention, LMCache, NVMe offloading), quantization formats (FP8, FP4, NVFP4 versus MXFP4, AWQ), speculative decoding (EAGLE-3, DFlash, multi-token prediction), and structural changes like prefill-decode disaggregation and attention variants built for long context (Ring Attention, log-linear attention, DeepSeek Sparse Attention).

Every technique here comes with a number: how much VRAM it saves, how much it speeds up time-to-first-token, or how it changes cost per million tokens, because "faster" without a benchmark isn't useful when you're deciding what to actually implement.

The pillar post, GPU Memory Requirements for LLMs, is the foundation the rest of this cluster builds on: the VRAM math for weights, KV cache, and quantization across model sizes from 7B to 744B, and how to match a model to the cheapest GPU that actually runs it. Start there if inference is running but you don't yet know where the bottleneck is. If you already know it's the KV cache or the attention kernel, the specific posts in this cluster go deep on exactly that.

Start Here

Inference Optimization

Start Here

GPU Memory Requirements for LLMs: VRAM Calculator & Sizing Guide (2026)

CPU-Free LLM Inference on GPU Cloud: How NVIDIA BlueField-4 DPUs and SmartNICs Cut the Serving-Stack Tax in 2026

Log-Linear Attention on GPU Cloud: The O(N log N) Architecture Challenging Linear Attention for Long-Context Inference (2026 Guide)

GPU Memory Requirements for LLMs: VRAM Calculator & Sizing Guide (2026)

CPU-Free LLM Inference on GPU Cloud: How NVIDIA BlueField-4 DPUs and SmartNICs Cut the Serving-Stack Tax in 2026

Log-Linear Attention on GPU Cloud: The O(N log N) Architecture Challenging Linear Attention for Long-Context Inference (2026 Guide)

All Inference Optimization Guides

Best Open-Source LLMs to Self-Host in 2026: VRAM Tier Guide

Context Engineering for Production AI Agents: KV Cache, Prefix Caching, and Long-Context GPU Economics (2026 Guide)

GPU Cold Start on Serverless LLM Inference: 4 Fixes That Actually Work (2026)

NVFP4 vs MXFP4: 4-Bit Quantization Format Decision Guide for LLM Inference (2026)

SubQ 1M-Preview: What the First Subquadratic LLM Means for GPU Cloud Inference (2026 Guide)

NVIDIA ICMSP Explained: KV Cache NVMe Offload, 5x Inference Gains, and Setup Guide (2026)

DeepSeek Sparse Attention on GPU Cloud: Deploy Long-Context LLMs at 80% Less Compute (2026 Guide)

Multi-Token Prediction on GPU Cloud: Deploy MTP LLMs for 2-3x Faster Inference (2026)

Eagle-3 Speculative Decoding on GPU Cloud: 3-4x Faster LLM Inference (2026)

Test-Time Training on GPU Cloud: Deploy TTT Layers for Adaptive LLM Inference Without Retraining (2026 Guide)

NVIDIA TensorRT Model Optimizer (ModelOpt): FP8, INT4, and FP4 Quantization Guide (2026)

FlashAttention 2 vs FlashAttention 3: H100 and H200 Speedups, FP8 Support, and Migration Guide (2026)

What is FP8 Quantization? AI Inference Performance, Accuracy, and Hardware Support Explained (2026)

NVIDIA Transformer Engine: FP8 Mixed Precision on H100 and H200 (Setup, Code, Benchmarks 2026)

GPU Goodput Engineering: Why AI Clusters Sit at 5% Utilization and How to Fix It (2026)

Sleep-Time Compute on GPU Cloud: Pre-Compute Long-Lived Context for 5x Lower Query Latency

LLM Pruning on GPU Cloud: SparseGPT and Wanda for 50% Model Compression Without Retraining (2026)

LLM Inference SLO Engineering: TTFT, ITL, and P99 Latency Budgets for Production AI (2026)

Ring Attention and Tree Attention on GPU Cloud: Sequence Parallelism for 10M-Token LLM Inference (2026 Guide)

Deploy xLSTM and RWKV-7 on GPU Cloud: Linear-Attention Alternatives for Million-Token Context Inference (2026)

Batch LLM Inference on GPU Cloud: Offline Processing Pipelines for 10x Lower Cost vs Real-Time Serving (2026 Guide)

Deploy FLUX.2 on GPU Cloud: Production Image Generation Setup Guide (2026)

GPU Cost Per Token: Benchmark 7 Major LLMs Across GPU Types in 2026

DFlash on GPU Cloud: 6x Faster LLM Inference with Block Diffusion Speculative Decoding (2026)

MXFP4 Quantization on GPU Cloud: Deploy LLMs at 4-Bit Precision (2026)

AI's Memory Wall Problem: Why More GPUs Don't Fix Inference Latency (2026)

GGUF Dynamic Quantization on GPU Cloud: Deploy LLMs 50% Cheaper with Unsloth Dynamic 2.0

Google TurboQuant: 6x KV Cache Compression for LLM Inference

AWQ Quantization Guide: Deploy LLMs at Half the GPU Cost (2026)

What Is Inference Engineering? The 2026 GPU Cloud Guide

Inference-Time Compute Scaling on GPU Cloud: Allocate More GPU to Think Harder, Not Train Bigger (2026)

LLM Serving Optimization: Continuous Batching, PagedAttention, and Chunked Prefill on H100 (2026)

NVIDIA NIXL and Disaggregated Inference: Move KV Caches Across GPUs at Wire Speed

Prefill-Decode Disaggregation on GPU Cloud: Split LLM Inference for 2x Throughput (2026 Guide)

NVMe KV Cache Offloading for LLM Inference: Serve 10x More Users on the Same GPU (2026)

Reduce Reasoning Model Inference Costs by 8x: GPU Optimization Guide (2026)

Why Your LLM Inference Is Slow (And How to Fix It)

KV Cache Optimization: Serve 10x More Users on the Same GPU (2026)

LoRA Multi-Adapter Serving: Fine-Tune Once, Serve 100 Customers on One GPU

Speculative Decoding Production Guide: 2-5x Faster LLM Inference on GPU Cloud

Deploy Wan 2.1/2.2 for AI Video: GPU Requirements and ComfyUI Setup

FP4 Quantization on Blackwell GPUs: Throughput, Cost, and When It's Worth It

How to Run LLMs Locally with Ollama: GPU-Accelerated Setup Guide

Try It on Real GPUs