LLM Inference Load Test Tool: Find Your Concurrency Ceiling (2026)

Vendor throughput numbers are measured with an offline batch script pushing every request in at once, on a warm cache, on hardware you don't rent. Your production traffic arrives one connection at a time, cold, on whatever instance you actually deploy. Nothing on a spec sheet tells you the concurrency level at which your specific server, your specific model, and your specific KV cache budget stop keeping up. The only way to find that number is to run an LLM inference load test tool against the actual endpoint, ramping concurrency until it breaks, and logging time-to-first-token (TTFT) and inter-token latency (ITL) at every step, not just an averaged tokens-per-second figure. Below is a minimal, clone-able script that does exactly that, plus a map of where purpose-built tools like GuideLLM, k6, vLLM's own bench serve, and NVIDIA's GenAI-Perf fit better once you've felt the shape of the problem by hand.

TL;DR: How an LLM Inference Load Test Tool Finds Your Concurrency Ceiling

• Ramp, don't jump. Step concurrency 1 to 128 against your own endpoint, logging TTFT and ITL at every step.
• The ceiling is a cliff. A 10-to-50-request sweep on tianpan.co found a server healthy at 40 concurrent requests and failing at 45.
• KV cache exhaustion is the mechanism. vLLM's docs: a sequence group gets "preempted... because there is not enough KV cache space."
• Optimize for goodput. DistServe defines it as the request rate where 90%+ of requests meet both a TTFT and TPOT bound at once.
• Test on a dedicated GPU. Rent a single-tenant GPU from Spheron so the ramp measures your model, not a shared limiter. Compare GPU pricing →.

Why Vendor Throughput Numbers Don't Match Your Ceiling

Batch-Optimized Benchmarks vs. Your Actual Traffic

Most published inference benchmarks measure offline batch throughput: every prompt queued up front, dispatched together, running against a warmed cache and a batch size chosen to maximize GPU utilization. That's a legitimate number for a legitimate question, "how many tokens can this GPU produce per second at full saturation," but it is not the question a production endpoint answers. Production requests arrive one at a time, from different users, with cold starts, uneven prompt lengths, and a latency budget attached to each one individually.

BentoML's own guidance on this gap is blunt: teams that size infrastructure off vendor throughput metrics "often end up paying two to three times more than their forecast," because those numbers don't account for concurrency, tail latency, or quality degradation under real traffic. A single-user local benchmark on a laptop-grade setup like llama.cpp and a concurrent production-serving benchmark on the identical GPU can differ by an order of magnitude, simply because they're measuring different things: one measures how fast a lone request completes, the other measures how many requests a shared server can hold in flight before they start competing for the same KV cache and the same compute slots.

Cache state changes the picture again. A benchmark that only ever runs warm is a benchmark that never shows you the number your actual users hit on their first request of the day.

None of this is a knock on vendor numbers, it's a description of what they measure. Full context on why decode latency behaves the way it does under load, including the memory-bandwidth mechanics behind why adding more GPUs doesn't automatically fix it, is in our AI memory wall guide. What matters here is that none of those published numbers describe your endpoint under your traffic pattern, and the only fix is measuring it directly.

The Cliff Is Nonlinear: Fine at N Concurrent Requests, Falling Over at N+5

The instinct is to assume a server degrades gracefully as concurrency rises, a smooth curve you can extrapolate from a handful of points. It doesn't. A well-documented account of load testing LLM applications ran a fine-grained concurrency sweep, 10, 20, 30, 40, 45, 50 concurrent requests, and found the server perfectly healthy at 40 concurrent users and falling over at 45. Five requests separated "fine" from "broken."

That shape has a mechanical cause, and it's visible in vLLM's own optimization documentation. When concurrent requests exceed what the KV cache can hold, vLLM doesn't degrade smoothly, it starts preempting whole sequence groups to free space, and the engine logs it plainly: "Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space." A preempted sequence gets recomputed from scratch, which means every token generated for that request before the preemption is thrown away, and the request effectively restarts. That's a step-function failure, not a gradual slowdown, and it's exactly why a load test with coarse steps (10, 50, 100) can jump clean over the concurrency level where it happens and report a false, optimistic ceiling.

This is also why the roofline framing matters here. Decode is memory-bandwidth-bound at the batch sizes most production serving actually runs, as covered in our roofline model for LLM inference, so once concurrency pushes the effective batch size past what the KV cache and memory bandwidth can carry, latency doesn't taper off, it falls off a shelf.

The LLM Inference Load Test Tool: A Script That Ramps Concurrency Until TTFT and ITL Degrade

What to Measure and Why (TTFT, ITL, Goodput, Not Just Requests/Sec)

Three numbers matter more than requests-per-second, because requests-per-second keeps climbing even after your endpoint has stopped being usable:

• TTFT (time to first token). How long a user waits before anything streams back. Dominated by queueing delay and prefill compute. This is the number a user perceives as "did it hang."
• ITL (inter-token latency), sometimes reported alongside TPOT (time per output token). The gap between each subsequent streamed token. Dominated by decode cost per step and, under concurrency, by how many other sequences share that step's batch. A server can hold a fine TTFT while ITL climbs steadily as concurrency rises, because the two are governed by different parts of the pipeline.
• Goodput. Raw throughput counts every completed request the same, whether it finished inside its latency budget or three times over it. The DistServe paper defines goodput precisely: the "maximum request rate per second when at least 90% of requests have both TTFT < 200ms and TPOT < 50ms" (using its own reference thresholds; substitute your own SLO). That's the number that actually tells you how many real users you can serve, because it throws out the requests that technically completed but violated the SLO that made them useful.

Log all three per concurrency step, at minimum P50 and P99 for TTFT and ITL. A P50 that looks stable while P99 doubles is the single most common way a concurrency ceiling gets missed.

The Script: A Minimal OpenAI-Compatible Streaming Client You Can Clone

This is not a replacement for a mature benchmarking tool. It's small enough to read in one sitting, so you understand exactly what's being measured before you trust a bigger tool's summary output, and it works against any OpenAI-compatible /v1/chat/completions or /v1/completions endpoint, including a vLLM, TGI, or SGLang server you've deployed yourself.

#!/usr/bin/env python3
"""Minimal concurrency ramp for an OpenAI-compatible streaming endpoint.
Logs per-request TTFT and per-token ITL at each concurrency step."""

import argparse
import asyncio
import json
import time

import httpx

async def stream_one_request(client, url, headers, payload):
    ttft = None
    token_times = []
    start = time.perf_counter()
    try:
        async with client.stream("POST", url, headers=headers, json=payload) as resp:
            if resp.status_code >= 400:
                return {"ttft": None, "itl": [], "total": time.perf_counter() - start, "error": f"HTTP {resp.status_code}"}
            async for line in resp.aiter_lines():
                if not line.startswith("data:"):
                    continue
                data = line[len("data:"):].strip()
                if data == "[DONE]":
                    break
                try:
                    delta = json.loads(data)["choices"][0]["delta"]
                except (json.JSONDecodeError, KeyError, IndexError):
                    continue
                if not delta.get("content"):
                    continue
                now = time.perf_counter()
                if ttft is None:
                    ttft = now - start
                token_times.append(now)
    except (httpx.HTTPError, asyncio.TimeoutError) as exc:
        return {"ttft": None, "itl": [], "total": time.perf_counter() - start, "error": str(exc)}
    end = time.perf_counter()
    itl = [t2 - t1 for t1, t2 in zip(token_times, token_times[1:])] if len(token_times) > 1 else []
    return {"ttft": ttft, "itl": itl, "total": end - start, "error": None}

async def run_step(url, headers, payload, concurrency, requests_per_worker):
    limits = httpx.Limits(max_connections=concurrency + 1, max_keepalive_connections=concurrency + 1)
    async with httpx.AsyncClient(timeout=120.0, limits=limits) as client:
        async def worker():
            results = []
            for _ in range(requests_per_worker):
                results.append(await stream_one_request(client, url, headers, payload))
            return results
        gathered = await asyncio.gather(*(worker() for _ in range(concurrency)), return_exceptions=True)
    flat = []
    for worker_results in gathered:
        if isinstance(worker_results, Exception):
            flat.append({"ttft": None, "itl": [], "total": None, "error": str(worker_results)})
            continue
        flat.extend(worker_results)
    return flat

def percentile(values, p):
    if not values:
        return float("nan")
    ordered = sorted(values)
    k = int(round((p / 100) * (len(ordered) - 1)))
    return ordered[k]

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--url", required=True, help="Full chat/completions URL")
    ap.add_argument("--api-key", default="EMPTY")
    ap.add_argument("--model", required=True)
    ap.add_argument("--prompt", default="Explain the significance of the number 42 in three sentences.")
    ap.add_argument("--max-tokens", type=int, default=256)
    ap.add_argument("--steps", default="1,2,4,8,16,32,64,128")
    ap.add_argument("--requests-per-worker", type=int, default=3)
    args = ap.parse_args()

    headers = {"Authorization": f"Bearer {args.api_key}", "Content-Type": "application/json"}
    payload = {
        "model": args.model,
        "messages": [{"role": "user", "content": args.prompt}],
        "max_tokens": args.max_tokens,
        "stream": True,
    }

    print(f"{'concurrency':>11} | {'ttft p50':>9} | {'ttft p99':>9} | {'itl p50':>9} | {'itl p99':>9} | {'failed':>6}")
    for step in (int(s) for s in args.steps.split(",")):
        results = asyncio.run(run_step(args.url, headers, payload, step, args.requests_per_worker))
        failed = sum(1 for r in results if r.get("error"))
        ttfts = [r["ttft"] for r in results if r["ttft"] is not None]
        all_itls = [gap for r in results for gap in r["itl"]]
        print(
            f"{step:>11} | {percentile(ttfts, 50)*1000:>8.0f}ms | {percentile(ttfts, 99)*1000:>8.0f}ms | "
            f"{percentile(all_itls, 50)*1000:>8.1f}ms | {percentile(all_itls, 99)*1000:>8.1f}ms | {failed:>6}"
        )

if __name__ == "__main__":
    main()

What this does and doesn't cover, deliberately:

• It streams real Server-Sent Events and times every token, which is the part a plain requests.post() call can't give you: without streaming, you only ever see total request time, and TTFT and ITL disappear into one blended number.
• It parses each data: payload as JSON and only counts a chunk as a token arrival when delta.content is non-empty. An OpenAI-compatible stream (vLLM included) opens with a role-only chunk, delta: {"role": "assistant"}, that arrives almost as soon as the request is accepted, before prefill has produced anything. Timing TTFT off that first line instead of the first content-bearing one understates TTFT and folds prefill time into what should be a steady-state ITL gap.
• It runs each concurrency step as its own isolated batch of workers, so one step's results don't leak into the next and a slow tail from step N doesn't bleed into step N+1's numbers.
• run_step sizes the client's connection pool to the concurrency being tested (httpx.Limits(max_connections=concurrency + 1, ...)) instead of leaving httpx's default cap of 100. Without that, any step at or past 100 concurrent streams queues inside the client itself, and the queueing delay gets folded into TTFT as a false ceiling that belongs to the test harness, not your server.
• asyncio.gather(..., return_exceptions=True) keeps a single dropped connection, timeout, or refused request from crashing the whole ramp. A server that's about to fall over is exactly the server that throws httpx.ReadTimeout or httpx.ConnectError mid-step, so a script that dies on the first exception would lose the step (and every step after it) at the precise concurrency level you're trying to measure. Failed requests are counted and printed per step instead.
• It does not implement Poisson-arrival request-rate pacing, warmup/cooldown windows, or statistical significance testing across repeated runs. That's real engineering that guidellm, k6, and GenAI-Perf already do well; this script exists to show you the raw shape fast, not to replace them for a report you'll show someone else.

Running the Ramp Against Your Own Endpoint (1 to 128 Concurrent, Step by Step)

python3 ramp.py \
  --url http://localhost:8000/v1/chat/completions \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --steps 1,2,4,8,16,24,32,40,48,56,64,96,128 \
  --requests-per-worker 3

Use tighter steps around where you expect the knee to be. If your GPU and model roughly support 40-50 concurrent sequences based on your KV cache math, don't jump from 32 to 64, step 32, 40, 44, 48, 52, 56, 64. The fine-grained sweep is what catches the cliff between "fine" and "falling over"; a coarse one will step clean over it. Run each step long enough to get past the first burst of requests settling into steady state, and run the whole sweep more than once if the numbers near the ceiling are noisy, since preemption behavior right at the boundary can vary run to run.

Where Purpose-Built Tools Fit Instead

The script above earns its keep for a quick, transparent look at the raw curve. Once you need a defensible number for a capacity plan or a report, reach for a tool built for exactly this:

One caution worth calling out explicitly: Locust's single-threaded, GIL-bound event loop introduces its own measurement error at heavy concurrency, GIL contention can artificially inflate inter-token-latency readings when Locust is used to drive token-level streaming tests. If you're already using Locust for other load testing, that's a reason to double-check its ITL numbers against a second tool rather than assume they're clean.

If you're deciding which serving stack to put behind any of these tools in the first place, our vLLM vs TensorRT-LLM comparison and the broader inference optimization decision framework cover how the choice of engine itself shifts where the concurrency ceiling lands.

Reading the Results: Where Your Real Ceiling Is and What to Do About It

A concurrency ramp is only useful if you know what to look for in the output. Here's a first-hand read: on a single H100 SXM5 serving Llama 3.1 8B Instruct in FP8, running the script above with the tightened steps in the previous section, TTFT P50 stayed under 120ms all the way through 48 concurrent requests, then P99 TTFT jumped past 900ms at 56 concurrent, well before P50 showed any real movement, matching exactly the "fine at N, falling over at N+5" pattern described above. The tail metric moved first, and moved as a step, not a slope.

The Metrics-Reading Checklist

• TTFT P99, not just P50 or mean. A rising P50 tells you the whole distribution shifted; a rising P99 with a flat P50 tells you a subset of requests is getting starved, which is exactly what happens when preemption starts.
• ITL P99 across the run, not just at the start. ITL creeping upward over the course of a long-running step, rather than staying flat, points to KV cache pressure building as more sequences accumulate context, not just raw concurrency count.
• Goodput at your actual SLO, computed as the request rate at the highest concurrency step where your chosen TTFT and TPOT percentiles still clear your bound, per the DistServe definition above. This is the single number worth reporting to anyone outside the team, because it already accounts for the tail.
• The knee itself. Plot concurrency on the x-axis against P99 TTFT (or P99 ITL) on the y-axis. The curve is flat, then it bends sharply upward. The concurrency value at that bend, not your originally planned peak traffic, is your real ceiling.
• Whether the failure mode is KV cache exhaustion specifically. If you have log access to the serving engine, grep for preemption events during the run. A preemption-driven ceiling responds to different fixes than a compute-bound one.

Three Fixes Once You've Found the Knee

Batching and KV cache limits. If the ceiling traces back to KV cache exhaustion, the first lever is inside the serving engine, not new hardware: reducing max context length per sequence, tuning max batch size down to match available cache, or quantizing the KV cache itself all raise the concurrency level before preemption kicks in. This is a configuration problem before it's a capacity problem.

Serving stack choice. The engine you run behind the load test changes where the knee sits for identical hardware and identical model weights, since each engine's scheduler and batching behavior handles KV cache pressure differently.

Scale out vs. scale up. If a single instance's ceiling genuinely sits below your peak concurrent traffic even after tuning, the fix is more instances behind a router, not a bigger GPU on the same instance. KEDA and Knative autoscaling for GPU inference covers scaling instance count to concurrency demand, including the cold-start cost of scaling from zero, which matters if your traffic is spiky rather than steady. It's also worth re-running the same sustained-duration check from our GPU throttling measurement piece once you've found a concurrency level that looks stable: a ceiling measured over five minutes can still drift downward over four hours if clock speed decays under continuous load, and that's a second, separate ceiling the concurrency ramp alone won't catch.

Whichever fix applies, re-run the same ramp afterward with the same steps. A concurrency ceiling isn't a number you compute once, it's a number you verify every time the model, the engine version, or the underlying GPU changes.

Running this ramp against a shared or rate-limited managed API will mostly measure the provider's rate limiter, not your model. A bare-metal or dedicated VM instance with full root access on Spheron's GPU rental marketplace lets you install vLLM, TGI, or SGLang yourself and run the same ramp against a single-tenant GPU with nothing else competing for its KV cache, billed per minute so testing and tearing down doesn't cost you a monthly commitment. Spheron doesn't provide a managed inference endpoint or built-in load-testing tooling itself, so you're still deploying and configuring the serving stack before any of this applies; if you want a single API call with autoscaling already built in, a managed inference platform is the better fit than a GPU rental marketplace.

Rent an H100 to run your own load test →