FlashAttention Explained: How It Cuts LLM Inference Costs

Standard attention is not slow because a GPU runs out of math to do. It's slow because it writes a full sequence-length-by-sequence-length matrix to HBM and reads it straight back, for every attention layer, on every request. Flash attention explained simply: it's a way to compute the exact same attention output without ever writing that matrix to memory in the first place, and the payoff shows up less in raw speed and more in how much context or how many concurrent requests you can pack onto one GPU before you need another one.

This post covers why attention is memory-bound rather than compute-bound, how FlashAttention's tiling and online softmax avoid the HBM round-trip, what actually changed between FlashAttention-2 and FlashAttention-3, and what the resulting memory savings buy you in capacity terms. For the version-by-version throughput tables and migration flags, we've already written those up in the FlashAttention-2 vs FlashAttention-3 guide; this post is about the mechanism underneath both of them.

FlashAttention Explained: Why Attention Is Memory-Bound, Not Compute-Bound

A GPU's tensor cores can do an enormous number of floating-point operations per second. What they can't do is anything useful while waiting for data to arrive from HBM. Attention, as originally written, spends most of its time waiting.

The Roofline Test: FLOPs per Byte Moved, Not FLOPs Alone

The way to tell whether a kernel is compute-bound or memory-bound is to compare its arithmetic intensity (FLOPs performed per byte of data moved) against the GPU's own ratio of peak compute to peak memory bandwidth. If a kernel does few FLOPs per byte it touches, it sits on the memory-bound side of the roofline no matter how fast the tensor cores are, because the cores spend their time idle, waiting on HBM.

Standard attention lands on that memory-bound side. The Q@K^T matmul, the softmax, and the softmax@V matmul are each individually cheap in FLOPs relative to how much data they read and write, especially once you count that the full N x N score matrix has to round-trip through HBM between those steps. We cover the general roofline framework and how it applies across LLM inference workloads, not just attention, in the AI memory wall guide; the short version is that this is the same reason decode-phase inference in general leaves tensor cores idle, and attention was the first place it got fixed with a dedicated algorithm.

What Standard Attention Actually Writes to HBM

Trace through a standard attention implementation and the memory traffic is the whole story. For sequence length N: compute Q@K^T, producing an N x N score matrix, and write it to HBM. Read it back to apply the softmax, then write the N x N result back to HBM. Read it a third time for the softmax@V matmul.

That's three full round-trips of an N x N matrix through HBM, on top of loading Q, K, and V themselves. At N=2K this is already a meaningful tax; Tri Dao and his co-authors argue the field was missing a principle: "making attention algorithms IO-aware," accounting for reads and writes between levels of GPU memory. Nobody had optimized for the HBM traffic specifically. Everybody had optimized for FLOPs, on a workload where FLOPs were never the bottleneck.

How FlashAttention Tiles Compute to Avoid HBM Round-Trips

FlashAttention's fix doesn't touch the math of attention at all. It changes the order operations happen in, so the N x N matrix never has to exist in HBM.

The Read/Write Path With and Without Tiling

Standard attention: load Q, K, V from HBM once; then write the full score matrix to HBM three separate times as described above. Total HBM traffic scales with N² for the score matrix, on top of the O(N) traffic for Q, K, V.

FlashAttention: split Q, K, and V into blocks small enough to fit in SRAM alongside the GPU's compute units. For each pair of Q and K/V blocks, compute the partial attention scores, apply a running softmax, and accumulate the partial output, entirely within SRAM. Only the final output block gets written back to HBM. The N x N matrix is computed piece by piece and immediately discarded from SRAM once each tile's contribution is folded into the running result. It never gets materialized in HBM at all, at any point.

Online Softmax: Why Tiling Doesn't Break the Math

The obvious objection to computing attention in tiles: softmax needs the sum over the entire row to normalize correctly, and if you're only looking at one tile of K/V at a time, you don't have that full row yet.

The fix is a running, or "online," softmax. Instead of computing the softmax denominator once at the end, FlashAttention keeps a running maximum and a running sum as it processes each tile, and rescales the accumulated output whenever a new tile shifts the running maximum. That keeps each tile numerically stable enough to be processed and discarded without ever holding the full row in memory. The output is mathematically identical to the standard, non-tiled computation, all from an exact algorithm with no approximation involved.

FlashAttention-2 vs FlashAttention-3 Explained: What Actually Changed

FA2 and FA3 both use the tiling and online softmax mechanism above. What changed between them is how well each generation of hardware can execute that mechanism without stalling.

FA2's Parallelism Fix on Ampere/Ada

The original FlashAttention left GPU occupancy on the table: work was partitioned across thread blocks in a way that didn't fully use the GPU's parallel resources, particularly at smaller batch sizes or when running the backward pass. FlashAttention-2 restructured the parallelism (partitioning work differently across warps and reducing non-matmul FLOPs like softmax rescaling) so more of the GPU stayed busy on the same tiling algorithm.

FA3's Warp Specialization and FP8 Path on Hopper

FlashAttention-3 doesn't run on Ampere at all in its Hopper-optimized form. What it adds on top of FA2's tiling: warp specialization, where some warps are dedicated to fetching the next Q/K/V tile asynchronously while other warps compute on the tile already in SRAM, so data movement and math overlap instead of happening in sequence. It also exposes an FP8 path for the Q@K^T and softmax@V matmuls, upcasting back to BF16 for the output. Neither of these is available to FA2, because they depend on Hopper-specific asynchronous data-movement instructions that don't exist on Ampere or Ada silicon.

We keep the throughput tables, the vLLM and SGLang flags, and the FA2-to-FA3 migration checklist in the dedicated FlashAttention-2 vs FlashAttention-3 guide rather than duplicating them here. FA3 is Hopper's ceiling; the next hardware-specific rewrite after that is FlashAttention-4 on Blackwell, which trades FA3's warp specialization for Blackwell's Tensor Memory Accelerator doing the tile prefetch in hardware instead of software.

What This Buys You: Longer Context or More Concurrent Requests on the Same GPU

The reason this matters for a buying decision, not just an engineering one, comes back to that quadratic-versus-linear memory curve.

Quadratic vs Linear Memory: What That Means for Context Length

Standard attention's memory footprint for the score matrix scales with N², so doubling context length quadruples the memory the attention matrix alone needs. FlashAttention's footprint scales with N, so doubling context length only doubles it. The gap between those two curves isn't fixed, it widens every time you push context length further, because a quadratic term outgrows a linear one by a larger multiple at every larger N. Every additional token of context you want to serve costs FlashAttention roughly proportionally more memory; it costs standard attention quadratically more. At 32K-128K context lengths, that difference is the entire reason serving long context is feasible on a single GPU at all, rather than a nice-to-have optimization.

The Capacity Trade: Same VRAM, More Concurrent Sequences

The other side of the same coin is concurrency rather than context length. A fixed pool of HBM on a GPU has to be split between model weights, the KV cache for every in-flight sequence, and any workspace attention needs. Standard attention's workspace for the score matrix competes directly with that pool and grows quadratically with each sequence's context length. FlashAttention's workspace for the same computation is small and roughly constant per tile, regardless of how long the sequence is, so essentially the entire remaining HBM budget after weights and KV cache is available for holding more concurrent sequences rather than being eaten by attention's own scratch space.

This is the part that shows up on an infra bill rather than a latency chart. It's not that FlashAttention makes each request faster in isolation; it's that the memory it frees up is what lets you push batch size up on the same GPU before hitting an out-of-memory wall, and a serving stack's ability to batch more concurrent sequences is what continuous batching and PagedAttention are built to exploit. We go through that batching mechanics in detail in the continuous batching and PagedAttention guide; FlashAttention is what makes the memory available for those techniques to use in the first place.

What This Looks Like in Practice

Take a single H100 SXM5 instance: 80GB HBM3 at 3.35 TB/s of memory bandwidth, on-demand from $3.98/hr on Spheron as of this post's publish date. Model weights and the KV cache for in-flight requests already claim a fixed share of that 80GB before attention even runs. With standard attention, the score-matrix workspace for each concurrent sequence grows with the square of its context length, so as you push either context length or batch size up, that workspace eats into the same 80GB the KV cache needs, and you hit an out-of-memory error well before the GPU's compute is anywhere near saturated. With FlashAttention active (the default in every current serving stack), that workspace stays small regardless of context length, so the constraint you actually hit is the KV cache and weights, which is the constraint you want: more of that 80GB goes to serving more requests or longer context, and less of it goes to a scratch matrix that never needed to exist in HBM at all.

That's the real headline result behind the quadratic-to-linear memory shift: it's not a benchmark number to admire, it's headroom you get to spend on batch size or context length on the exact same card.

Per-minute billing means you can run that comparison, see the memory ceiling for your own model and context length, and shut the instance down within the hour, on either H100 or A100 depending on which architecture your production fleet runs. If your workload is squarely in the "need Hopper's FP8 attention path" territory rather than the baseline tiling behavior every GPU gets, that's the FA3-specific comparison our FA2 vs FA3 guide walks through with full throughput tables.

Pricing fluctuates based on GPU availability. The prices above are based on 01 Sep 2026 and may have changed. Check current GPU pricing → for live rates.

Getting It Running: vLLM and SGLang Default to It, No Flags Needed

The practical good news is that you almost certainly don't need to do anything to get FlashAttention. It's not an opt-in optimization anymore; it's the baseline attention implementation both major serving stacks assume.

vLLM and SGLang both auto-detect the GPU architecture at startup and select the matching FlashAttention version: FA2 on Ampere and Ada (A100, L40S, RTX 4090), FA3 on Hopper (H100, H200), with no attention-backend flag required for the default path. You'll see the selected backend named in the startup logs. The only reason to reach for an explicit override is to force a different backend for debugging or a side-by-side comparison, which vLLM exposes through the --attention-backend flag.

Where this becomes a live decision rather than a settled default is the FP8 attention path FA3 exposes on Hopper, which isn't automatic and does need an explicit flag plus a quality validation pass against your task, or the point where a single GPU's linear-memory ceiling still isn't enough and you need sequence parallelism across multiple GPUs instead. Both of those are covered in depth in the linked guides. For everything short of that, if you're running vLLM or SGLang on a current GPU, you're already getting FlashAttention's memory savings without having set a single flag.

For the lower-level memory bandwidth numbers behind why Hopper and Blackwell get more out of FA3/FA4 than Ampere does, see the HBM3e vs HBM4 vs HBM4e guide, and for how FA3's optional FP8 attention path interacts with FP8 weight quantization, see the FP8 quantization explained guide.

FlashAttention runs by default on every current H100, H200, and A100 instance on Spheron, no configuration required. Rent one by the minute to see the memory headroom on your own model before you decide whether you need a second GPU.

H200 on Spheron → | L40S on Spheron → | View all GPU pricing →