The first thing that told us something was wrong wasn't an error. It was a graph of queue depth on a vLLM deployment we were load testing on Spheron, climbing in a straight line while every other dashboard still looked fine. That climb is LLM serving backpressure in its rawest form: demand outrunning what the GPU underneath can actually serve, with nothing in the stack yet telling callers to slow down. By the time the error-rate panel caught up, we were already rejecting requests that had been waiting long enough to be useless to the client that sent them.
TL;DR: What Stops LLM Serving Backpressure From Turning Into an Outage
- Failure mode: an unbounded queue keeps accepting work it can't serve, so requests time out together instead of a fraction getting rejected early.
- Queue caps: vLLM's queue has no depth limit today; a proposed
--max-waiting-queue-lengthflag would return a fast HTTP 503 once it fills. - Priority shedding: Google's SRE playbook sheds the lowest-priority tier first, the ordering LiteLLM ships as admission control.
- Admission signal: Anthropic gates capacity on requests, input tokens, and output tokens per minute, not RPS alone.
- Client fix: OpenAI recommends
Retry-Afterplus jitter, since synced retries turn a slowdown into a wall. Spheron's per-minute billing adds burst capacity fast but doesn't replace admission control on an H100 GPU rental.
What LLM Serving Backpressure Actually Looked Like at the API Layer
Overload doesn't announce itself as an error spike. It announces itself as a queue that stops draining, and by the time that shows up as errors, the request that would have told you sooner already happened three dashboards ago.
The First Signal Was Queue Depth, Not the Error Rate
We ran a single-node vLLM deployment on a Spheron H100 SXM5 80GB behind a synthetic load generator, holding request volume at 120 requests/second, a rate the node had sustained comfortably in every prior test, and stepped it to 260 requests/second over a five-minute ramp. The error-rate panel, the one everyone actually has an alert on, stayed flat for the first three minutes. Queue depth didn't: it went from 8 waiting requests before the ramp started to 347 by the four-minute mark, because vLLM's scheduler was still accepting every request that arrived and simply appending it to the back of an unbounded waiting queue.
That's the mechanical root of it. vLLM's waiting queue is, by design, an unbounded in-memory deque with no built-in way to reject a request once the queue grows too long. An open vLLM proposal would add a --max-waiting-queue-length flag that raises a SchedulerWaitingQueueFullError, mapped to an HTTP 503, once the queue hits its cap. Without that flag, the scheduler has exactly one lever: keep accepting and hope the backlog clears before every request in it times out. It didn't clear. It kept growing until the client-side timeouts started firing in a batch, which is what turned a queue graph into an incident.
The mechanics of how that queue fills and drains in the first place, one slot released the moment a request finishes rather than a whole batch waiting on its slowest member, are covered in more depth in our continuous batching explainer. Backpressure control is what happens after that scheduling layer is already saturated; it doesn't replace it.
Why Averaged TTFT Hid the Problem Until It Was Already an Overloaded-Error Storm
The dashboard we were watching reported mean time-to-first-token, and it barely moved during the ramp: 190ms at the start, 210ms four minutes in. That's not a bug in the dashboard, it's what an average does when a growing share of requests are queued rather than running: a request that's still waiting in line hasn't generated a first token yet, so it isn't in the TTFT distribution at all until it finally gets scheduled. The average only reflects requests that made it onto the GPU, and those looked completely normal the entire time, while our client-side timeout was set to 8 seconds.
P99 TTFT told a different story from the first minute of the ramp, because tail latency is exactly where queueing delay shows up before anything else does. It went from 420ms before the ramp to 6.1 seconds at the two-minute mark, then past our 8-second client timeout by minute three, well before the mean TTFT line had moved at all. Marc Brooker's writing on the economics of load-balanced systems makes the underlying mechanism explicit: "as soon as the mean arrival rate exceeds the system's ability to complete requests, the queue grows without bound and latency goes to infinity." That's not a metaphor. It's Little's Law and an M/M/1 queue behaving exactly as the math says they will, and the average hides it right up until it can't anymore. Brooker's related point about the same cliff, made in the context of throttling and admission control specifically, is worth carrying into the next section: "throttling, admission control, back pressure, backoff and other mechanisms can play a big role in avoiding these cliffs, but they still exist."
How Client Retries Turned a Slowdown Into a Wall
Once TTFT crossed our 8-second client-side timeout, the client library did what client libraries do: it retried. By minute five of the ramp, the load generator was sending roughly 260 new requests per second and another 90 retries per second from requests that had already timed out once, a 35% increase in effective load with no increase in real demand. Each retry landed back in the same queue that was already the problem, competing with fresh requests for the same GPU slots. That's the compounding effect that turns a slowdown into an outage rather than a degraded-but-recovering system. It generalizes past our own load test: research on production LLM API resilience found that naive layered retries compound multiplicatively, a five-service call chain with three retries at each layer produces 3^5 = 243 backend calls for a single original request, and traced roughly 40% of cascading failures in distributed systems back to retry logic amplifying an initial slowdown into a full outage.
That number is why the fix couldn't just be "reject requests faster." It had to also change what happens on the client's next attempt, or the rejected requests would come straight back and recreate the same queue.
The pattern isn't unique to a load test on a single node, either. Provider-side capacity contention is common enough across the industry that it shows up in uptime numbers: one analysis found industry-wide LLM API uptime fell from 99.66% to 99.46% year over year by Q1 2025, a measured 60% increase in downtime as demand outpaced provider capacity scaling, and logged roughly 158 incidents for Anthropic's own API across a 90-day observation window, with a median incident duration over an hour. Two documented public incidents make the shape concrete: Azure OpenAI's Sweden Central region saw latency spike from a normal 2-5 seconds to over 60 seconds during a November 2025 capacity crunch.
What We Tried That Didn't Help
Every one of the following felt like the obvious fix when we reached for it. Each one either arrived too late, moved the bottleneck somewhere else, or made the retry problem worse instead of better.
Provisioning More GPUs (and Why It Arrived After the Damage Was Done)
The instinctive response to "the GPU is out of capacity" is to add another GPU. It's not wrong, exactly, it's just too slow to be the fix for an overload that unfolds over minutes. Standing up a new node, whether that's a fresh reservation, a cluster autoscaler event, or a cold-start pod on Kubernetes, takes long enough that the incident is usually already resolving on its own (because callers have given up and stopped retrying) by the time the new capacity is warm. Our own KEDA and Knative autoscaling guide covers this cold-start gap in detail: a scale-to-zero GPU pod has to pull a container image, initialize CUDA, and load model weights before it can serve a single token, and that startup window is exactly the window an overload happens in. Autoscaling is still worth having. It's just not an admission control mechanism, and treating it like one means every overload runs its full course before help arrives.
A Flat Rate Limit With No Priority Tiers
Our first pass at protecting the queue was a flat per-key rate limit: N requests per minute, same limit for every caller, no distinction between a health check, a low-priority batch job, and a user waiting on a live response. That stopped runaway single clients, but it didn't stop the overload, because the traffic that caused it was spread across many callers, each individually under the limit. A limit measured only in requests per minute also can't tell a one-line completion from a 30,000-token summarization job; both count as one request against the same cap while costing wildly different amounts of GPU time.
Anthropic's production rate limiting is a useful contrast here: it enforces limits across three separate dimensions per model class, requests per minute, input tokens per minute, and output tokens per minute. A single RPS number can't capture any of that, and a flat limit with no priority tiers can't distinguish traffic that should be shed first from traffic that shouldn't be shed at all.
An Unbounded FIFO Queue That Just Kept Accepting Work
This was the default we started from, and it's the default vLLM ships with. Every request that arrives gets a spot in line, first in first out, with no ceiling on how long that line can get. It's the simplest possible design and it's exactly wrong for overload: a FIFO queue with no depth limit treats "we are 30 seconds behind" and "we are 30 minutes behind" identically, accepting new work at the same rate either way. The fix isn't a smarter queue. It's a queue that's allowed to say no.
Naive Exponential Backoff That Synchronized Every Client's Next Retry
Once we added retries to our own load generator (mimicking what a real client would do), we picked exponential backoff with no jitter: wait 1 second, then 2, then 4. That's better than immediate retry, but it has a specific failure mode when many clients hit the same error at close to the same time, which is exactly what happens during an overload. They all retry at 1 second, then they all retry at 2 seconds, then at 4, arriving in synchronized waves instead of spread out. Google's SRE material on handling overload describes the client-side version of this problem directly: an adaptive throttling scheme has clients begin self-limiting once the ratio of attempted requests to accepted requests, measured over a rolling window, crosses a multiplier K (2, by default), rather than waiting for the server to reject them outright. Unjittered backoff does the opposite of that: it keeps every client fully committed to retrying, just on a synchronized clock.
The Fix: Queue Depth Limits, Load Shedding, and Admission Control That Held
Nothing above was wrong to try. All four gave us information about where the actual constraint was. The combination that stopped the overload from recurring had three parts, and none of them worked alone.
Capping the Waiting Queue and Rejecting Fast With an Overloaded-Style Error
The first change was giving the queue a ceiling. Once queue depth crosses a fixed threshold, new requests get rejected immediately with a 503-class response instead of being appended to the back of the line. This is precisely the shape of the vLLM proposal referenced earlier, a --max-waiting-queue-length flag that raises a scheduler-level error mapped to HTTP 503, and it's the same shape LiteLLM shipped in production: once max_in_flight_requests_per_worker is full and the queue is either at max_queued_requests_per_worker or a queued request has exceeded admission_queue_timeout_seconds, the proxy returns an immediate 503 with error type "overloaded_error" and a retry-after header, rather than letting the request sit.
Anthropic's API distinguishes this exact case in its error taxonomy: a distinct HTTP 529 overloaded_error, described as "the API is temporarily overloaded." That separation matters for what a caller does next. A 429 means slow down, you specifically have used your quota. A 529, or a generic overloaded_error from a self-hosted stack, means the system is out of capacity regardless of who's asking, and retrying immediately just adds to the same backlog.
Priority Classes: What Got Shed First, What Never Did
A queue depth cap answers "when do we start rejecting." It doesn't answer "what do we reject first," and treating all traffic as equally disposable was its own mistake. We split incoming requests into three priority classes, mapped to a queue budget out of the 150-request cap we settled on:
| Priority class | Example traffic | Share of queue budget | Shed order |
|---|---|---|---|
| Health checks | Liveness and readiness probes | 5 slots, reserved | Never shed |
| Interactive | User-facing chat and agent responses | 120 slots | Shed only after background traffic is fully shed |
| Background | Batch summarization, async report generation | 25 slots | Shed first |
When the queue cap is reached, background jobs get shed first, interactive traffic gets shed only once background traffic is already fully shed, and health checks are effectively never shed.
Setting the Admission Threshold From In-Flight Plus Queue Depth, Not RPS Alone
The threshold that decides whether a new request gets admitted at all is set from two numbers together: current in-flight request count and current queue depth, not a raw requests-per-second ceiling. RPS alone can't distinguish a burst of short completions from a burst of long document-summarization calls, and on a GPU those cost wildly different amounts of compute per request. In-flight count plus queue depth reflects actual load on the scheduler regardless of how expensive each individual request is, which is closer to what LiteLLM's max_in_flight_requests_per_worker and max_queued_requests_per_worker settings gate on, and closer to what Anthropic's token-bucket limiting captures by tracking input and output tokens per minute as separate dimensions from request count.
Retry-After and Jittered Backoff on the Client Side
The server-side changes only hold if the client side stops making the queue worse. We changed our load generator, and recommend the same for any production client, to read the Retry-After header on a 503 or 529 response when the API sends one, and to back off with exponential delay plus random jitter rather than a fixed schedule. Jitter is the part that's easy to skip and expensive to skip. Without it, a fleet of clients that all failed together will all retry together, recreating the exact spike that got them rejected in the first place.
What the Same Traffic Pattern Looked Like After the Fix
We reran the same synthetic ramp, 120 requests/second stepping to 260 over five minutes, against the same single-node vLLM deployment with the queue cap, priority shedding, and admission thresholds in place:
| Metric | Before the fix | After the fix |
|---|---|---|
| Peak queue depth | 347 waiting requests, still climbing when we cut the test | Capped at 150, held flat once reached |
| Time to rejection past the cap | No rejection; requests waited until they timed out at 8s | Under 15ms |
| Effective load from retries at minute 5 | +90 req/s (35% over the real 260 req/s demand) | +12 req/s, spread across several seconds by jittered backoff |
| Interactive P99 TTFT at minute 5 | Past the 8s client timeout | 640ms, inside the interactive budget |
| What got shed | Nothing, until client timeouts shed everything at once | Background jobs first, interactive traffic untouched |
Queue depth still climbed during the reran ramp, because the ramp still exceeded the node's real capacity, but it stopped climbing at the cap instead of growing without bound. Requests beyond that cap were rejected in milliseconds with a retryable error instead of timing out after seconds, and because our retry logic now waited on jittered backoff, the rejected requests came back spread across the following several seconds instead of all at once. Interactive traffic kept its P99 TTFT inside budget for the whole run; the requests that got shed were background jobs, which is what the priority ordering was for. The queue depth graph was still the first thing to move. This time, it was also the thing that stopped the incident from happening, because it triggered rejection instead of just an alert.
Where Admission Control Fits With Autoscaling and SLOs
Admission control, autoscaling, and architectural changes that reduce how often overload happens in the first place are three different layers, and none of them substitutes for the other two.
Autoscaling adds capacity on a timescale of minutes; admission control decides what happens in the meantime, which for an LLM serving stack is usually the timescale the overload actually unfolds on. Reducing how often you hit the ceiling at all is a separate, longer-horizon lever: long-prompt prefill bursts are a common trigger for exactly the queue-depth spikes described here, and our prefill-decode disaggregation guide covers the architectural split that reduces how often a large prefill batch stalls the decode path for everyone else waiting behind it. Where admission control physically sits also matters: if requests fan out across multiple model replicas before they ever reach a single node's queue, the gateway doing that routing needs its own capacity awareness, which is what our GKE Inference Gateway walkthrough covers for KV-cache-aware request routing. And the choice of serving stack itself changes how much headroom exists before any of this becomes necessary; our LLM inference optimization framework compares vLLM, TensorRT-LLM, and SGLang on exactly that basis.
Fast GPU provisioning is a real lever in this stack, just not the one that stops an overload already in progress. Spheron bills per minute after a 20-minute minimum with no long-term contract, deploys in under two minutes, and aggregates capacity across 5+ providers, which matters specifically because it makes it more likely that spot or on-demand GPU capacity is actually available when a team needs to add nodes mid-incident rather than being stuck waiting on a single provider's inventory. None of that replaces the fixes in this post. Spheron is an infrastructure marketplace, not a serving stack: the queue depth caps, priority classes, and admission thresholds described above have to be built into vLLM, a gateway, or a router regardless of which provider the GPU underneath comes from, and even a two-minute deploy is far slower than the seconds-to-minutes window an overload actually unfolds in. Spot capacity in particular can be reclaimed, which extends how much headroom you have, but it doesn't remove the need for admission control on requests that already got in. Teams weighing whether to add an H100 as burst capacity during a demand spike should treat it as exactly that, a way to add real GPU-hours quickly, not a substitute for deciding what happens to a request when every GPU-hour you have is already spoken for.
Admission control decides what happens to a request when every GPU-hour is already spoken for; fast burst capacity decides how quickly that stops being true. Check current GPU pricing → if you're sizing burst headroom against a real traffic spike.
Frequently Asked Questions
Backpressure is any mechanism that tells upstream callers to slow down or gives a serving stack a way to refuse work once it can no longer keep up, instead of silently queuing every request that arrives. In LLM serving that means queue depth caps, admission control on in-flight and queued requests, and fast rejection with a retryable error, rather than an unbounded queue that keeps accepting work until every request times out.
vLLM's waiting queue is an unbounded in-memory deque by default, so it keeps accepting requests no matter how far behind the GPU falls, and every one of them eventually times out together instead of a fraction being rejected early. An open vLLM proposal adds a --max-waiting-queue-length flag that raises a SchedulerWaitingQueueFullError, mapped to an HTTP 503, once the queue hits its cap, so a caller gets a fast, retryable rejection instead of a slow one.
A 429 rate_limit_error means a specific account or API key has exceeded its own quota, request, input-token, or output-token limits. A 529 (or a generic overloaded_error on other providers) means the service itself is out of capacity regardless of who is asking. Anthropic's API returns both as distinct codes for this reason: one is a per-caller problem the caller can fix by slowing down, the other is a system-wide problem that retrying immediately only makes worse.
Queue depth and in-flight request count are better admission signals than raw RPS for LLM serving, because request cost varies enormously with prompt length and output length. A flat RPS limit treats a one-line completion the same as a 30,000-token document summary, so it either throttles cheap requests too aggressively or lets expensive ones through until the GPU is already saturated. LiteLLM's per-worker admission control, for example, gates on max_in_flight_requests_per_worker and max_queued_requests_per_worker rather than a request rate.
Read the Retry-After header if the API sends one, and back off with an exponential delay plus random jitter rather than a fixed interval. OpenAI's own rate-limit guidance recommends exactly this, specifically to stop every client from retrying at the same instant and re-creating the spike that caused the rejection in the first place. A fixed, unjittered backoff synchronizes retries instead of spreading them out.
