LLM API Rate Limits for AI Agents: Guaranteed Concurrency Guide (2026)

An agent that calls a tool, checks the result, and retries isn't one request against an LLM API's rate limits. It's five, ten, sometimes forty, all fired within the same few seconds, drawing on the same per-minute quota a human typing one chat message at a time never gets close to. That gap is why agent fleets trip LLM API rate limits long before their usage dashboard shows anything near full, and why "rate limit" and "concurrency limit" get treated as the same promise when they're actually two different ones.

TL;DR: Why AI Agents Hit LLM API Rate Limits So Often

AI agents hit LLM API rate limits because tool-calling loops fan a single user action into many parallel, token-heavy requests against the same per-minute quota a chat message never approaches, and an RPM/TPM number describes average throughput over a minute, not how many requests a provider will hold in flight at once.

• Three separate caps: Anthropic's Messages API enforces RPM, ITPM, and OTPM independently per model class, and a 429 fires from whichever one runs out first, even while the other two sit nearly empty.
• Cache changes the budget: Claude's ITPM counts only uncached input tokens, so an 80% cache hit rate against a 2,000,000 ITPM limit effectively processes 10,000,000 total input tokens a minute.
• Spheron: Growth guarantees 32 requests at once on reserved GPUs with a p99 time-to-first-token written into the contract, a flat concurrency number instead of a per-minute quota to reverse-engineer. Compare current GPU pricing.
• Burst risk: in one illustrative scenario from a 23 Apr 2026 writeup on agent rate-limit self-DoS, a single agent action fanning out to roughly 47 parallel tool-call workers hammers one shared token bucket at once while the account's own usage dashboard still shows only 62% of its tier cap used.
• The fix: cache-aware token accounting and a guaranteed-concurrency contract both target the same failure mode, a burst of simultaneous, token-heavy calls that a per-minute average never sees coming until it's already a 429.

Why Agent Loops Trigger 429s That Chat Traffic Never Does

A chat session sends one request, waits for one answer, and sends the next one only after a human reads the first. An agent doesn't wait on a human between steps. It plans, calls a tool, reads the result, decides whether to call another tool or a sub-agent, and repeats, often with several of those branches firing in parallel because nothing in the loop forces them to run one at a time. Spheron's own breakdown of agentic token costs puts the resulting volume at 5 to 30 times the tokens per task of a single chat exchange, and that multiplier alone changes which limiter an account hits first: a fleet that never came close to its RPM ceiling on chat traffic can run straight into its ITPM or OTPM cap once every call is carrying a full tool schema and conversation history instead of one short message.

The shape of the traffic matters as much as the volume. A practitioner writeup on agent rate-limit self-DoS frames the failure bluntly: "Your agent is DoSing itself, and the rate limiter is doing exactly what you told it to." Its illustrative scenario has one user action fanning out to "forty-seven different workers" hitting the same token bucket at once, while the account's own dashboard shows "Token-per-minute usage is at 62% of the tier cap, well inside the green band." An aggregate metric averaged over a minute can look completely healthy while a two-second burst inside that minute is already past what the serving layer can actually hold concurrently. That's the core mismatch this whole post is about: RPM and TPM describe a rate over time, not a ceiling on simultaneous requests.

Retries compound the problem instead of absorbing it. The same writeup estimates that a 2% tool error rate, retried three times through an eight-step agent loop, can compound into a 15-20% end-to-end failure rate, once cumulative latency, context bloat, and planner confusion are stacked on top of the retries themselves. Every one of those retries is also a fresh request against the same rate limit the original call already strained, so a loop that starts tripping 429s tends to make its own situation worse by retrying into the same wall.

And the per-request accounting doesn't help. A chat-sized account allowance built around short messages understates, by roughly 60x on a single call, how fast an agent's larger, context-heavy requests burn through a token-based quota sitting right behind that same RPM number.

RPM/TPM Quotas vs. Guaranteed Concurrency: Two Different Promises

Every major LLM API prices access the same way: a quota measured over a rolling time window. Anthropic enforces requests per minute, input tokens per minute, and output tokens per minute, each with its own cap per model class. Google's Gemini API adds a third axis, enforcing limits across requests per minute, tokens per minute, and requests per day, and those limits are "applied per project, not per API key," plus a separate rolling 10-minute spend-based limit on top, meaning every agent instance sharing a Gemini project draws from one combined pool regardless of which service account fired the call. OpenRouter caps unverified accounts at 50 requests a day, rising to 1,000 a day once an account has spent $10 or more in credits, and limits free-tagged models specifically to 20 requests a minute, then passes paid-model limits straight through to whichever underlying provider is actually serving the request, a pattern Spheron's OpenRouter alternatives comparison covers in more depth.

None of those four mechanisms promise a specific number of requests the provider will hold in flight simultaneously. A 4,000 RPM limit is an average: 4,000 requests spread evenly across sixty seconds, or technically, since Anthropic's limiter uses a token bucket, "continuously replenished up to your maximum limit, rather than being reset at fixed intervals." That smooths bursts somewhat, but it's still a rate, not a concurrency guarantee, and Anthropic says as much in its own documentation: a 60 RPM limit "might be enforced as 1 request per second," so "short bursts of requests can exceed the limit and trigger rate limit errors" even when the per-minute average is nowhere close to the cap.

Guaranteed concurrency is a different kind of promise: a flat, contracted number of requests a provider commits to accepting at the same instant, independent of how that traffic is distributed across a minute. Spheron's managed inference plans state it that way on purpose, Starter at "8 requests at once, guaranteed," Growth at 32, running on reserved GPUs rather than a shared capacity pool. It's a narrower promise than a per-minute quota in one sense, it says nothing about total throughput over an hour, but it's a more legible one for an agent fleet whose actual failure mode is a burst, not a sustained average.

Anthropic's Rate Limits by the Numbers: RPM, ITPM, OTPM, and Cache-Aware Counting

Anthropic publishes its limits by usage tier, and the jump between tiers is large enough to change how an agent fleet should think about its own traffic shape. Here's Claude Sonnet 5.5 across all three published tiers:

Three independent caps, and exceeding any one of them, not just the lowest, returns a 429 that names which limit failed. An organization sitting well under its RPM ceiling can still get throttled on ITPM alone if its agent calls are large and token-heavy, which is exactly the traffic pattern a tool-calling loop produces.

The part of Anthropic's system worth building an agent fleet around is cache-aware ITPM counting. "Claude's default ITPM rate limits count only uncached input tokens," per Anthropic's own documentation, splitting a request's input into three buckets: input_tokens (what's left after the last cache breakpoint) and cache_creation_input_tokens both count toward ITPM, while cache_read_input_tokens do not count toward ITPM at all. Anthropic's own worked example: "With a 2,000,000 ITPM limit and an 80 percent cache hit rate, you could effectively process 10,000,000 total input tokens per minute (2M uncached + 8M cached), because cached tokens don't count toward your rate limit." That's not a minor optimization, it's a 5x multiplier on effective throughput for any workload with heavy context reuse, which describes almost every agent loop that resends its own history on each step.

Two more mechanics matter for an agent fleet specifically. First, sharp traffic increases can trip a separate acceleration limiter: "You might also encounter 429 errors because of acceleration limits on the API if your organization has a sharp increase in usage," and Anthropic's fix is to "increase your traffic gradually and maintain consistent usage patterns," which matters directly for an agent fleet that scales from a handful of sessions to dozens inside a single deploy. Second, OTPM is evaluated against tokens actually generated, not against the max_tokens parameter set on the request, so there's no rate-limit penalty for requesting a larger ceiling than a response will typically need.

How Cached Input Tokens Change the Math: A Worked Agent Request

Cache-aware accounting shows up twice in an agent's economics: once on the rate-limit side, where Anthropic exempts cached tokens from ITPM, and once on the billing side, where Spheron's counted-token weighting prices a cached token at a fraction of a fresh one. Both mechanisms reward the exact pattern an agent loop naturally falls into: resending a large, mostly-unchanged system prompt and tool history on every single step.

Spheron weighs output tokens at 4x, fresh input tokens at 1x, and cached input tokens at 0.25x, dropping to 0.03x specifically on DeepSeek V4.1 Flash. In that post's own worked example, a request carrying 16,000 cached input tokens, 2,000 fresh input tokens, and 400 output tokens comes out to 7,600 counted tokens against an 18,400-token raw total, a 59% reduction purely from how the cached share is weighted. An agent conversation that resends most of its own history on every call is precisely the traffic shape that pushes the cached share, and therefore the savings, higher with every additional step.

That's not just a billing-model abstraction. Spheron's own inference fleet measured it directly: running DeepSeek V4.1 Flash on an 8xB200 node under 32 concurrent agent sessions, with vLLM, FP8 and FP4 precision, a 1M-token context window, and speculative decoding, one copy read 218,578 prompt tokens per second and wrote 2,555 output tokens per second at an 86% cache hit rate, with zero errors, as measured on 15 Oct 2026. Replaying the identical sessions a second time pushed the cache hit rate to 99% and prompt throughput to 352,000 tokens per second. That's a higher cache hit rate than the 80% figure in Anthropic's own ITPM example, and it's the regime real agent traffic lands in once a loop has been running for more than a step or two: the system prompt, tool schemas, and early conversation turns stop changing, and an increasing share of every subsequent call's input is a repeat of something already served.

Once caching has absorbed most of an agent request's input-token cost, both on the rate-limit side and the billing side, what's left to actually constrain is the fresh input, the output, and how many of those calls can run at the same instant. That's the piece a per-minute quota doesn't directly promise and a concurrency guarantee does.

What a Fixed Number of Requests at Once Actually Buys an Agent Fleet

A guaranteed concurrency number is a specific, testable promise: this many requests, running right now, at the same time, regardless of how your traffic happened to distribute across the preceding sixty seconds. Spheron states it as a flat figure per plan, Starter guarantees 8 requests at once with 99.5% uptime, and Growth guarantees 32 requests at once with 99.9% uptime and a p99 time-to-first-token commitment written into the contract, both running on reserved GPUs rather than a pool shared with other tenants' traffic spikes.

Reserved capacity changes what an overload actually looks like, not just how often it happens. In live failure tests on Spheron's own four-node 8xB200 inference fleet on 16 Oct 2026, run through the production gateway, killing a spot node mid-stream let in-flight streams finish on the other copy after about 7 seconds with zero failures; killing a node mid-write on 16 requests still left all 16 ending in one complete, valid tool call; an engine crash restarted on the same node in 3 minutes; and gateway and control-plane restarts dropped 0 of 832 requests during a gateway hand-over. Past the fleet's actual capacity, excess requests get a fast 429 instead of hanging, which matters to an agent loop specifically, because a hung request blocks a retry decision in a way a fast, clean rejection doesn't.

That's a different failure mode from a shared, rate-limited multi-tenant pool, where a provider-side load spike or a neighboring tenant's burst can degrade latency without any single limiter technically being exceeded. A reserved, guaranteed-concurrency slot doesn't eliminate overload, it just makes overload a known, contracted number instead of a moving target behind response headers.

The honest limits matter here too. Spheron's inference plans cover three open models today, GLM-5.3, GLM-5.3-Flash, and DeepSeek V4.1 Flash, so a team whose agent is built specifically on a closed frontier model has no path onto this plan for that model. Starter carries no p99 latency commitment, only the concurrency guarantee and uptime SLA; only Growth and Dedicated add a contracted p99 time-to-first-token. Growth requires a 3-month term and Dedicated a 50B-token-a-month floor, which doesn't fit bursty or exploratory agent traffic the way a per-token API's pay-as-you-go model does. And pricing itself is quoted through a request form rather than published on the page, so there's no self-serve number to diff instantly against a provider's RPM/TPM-gated tier. Spheron's managed inference plans are a fit for steady, production agent traffic that already knows roughly how many simultaneous sessions it runs; they're the wrong shape for a prototype still figuring out its own volume.

Designing an Agent Loop That Doesn't Rate-Limit Itself

Most of the fixes here are about matching how an agent loop fans out requests to what a given limiter actually measures, rather than hoping a bigger tier makes the mismatch go away.

• Cap fan-out with a semaphore, not a hope. Bound how many tool calls or sub-agents a single planning step can fire concurrently, tuned to your account's actual RPM and concurrency headroom, not to an arbitrary round number.
• Budget on tokens, not just requests. A 200-token call and a 12,000-token call cost an identical unit against an RPM limit but a roughly 60x-different unit against ITPM or OTPM, so a retry policy built around request counts alone will miss the limiter that actually fails first.
• Push cache hit rate up deliberately. Keep the system prompt, tool schemas, and early conversation turns stable and in a fixed position so they land ahead of the cache breakpoint on every call. On Claude's API that reclaims ITPM headroom directly; on a counted-token plan it reclaims billed volume by the same mechanism.
• Ramp new traffic gradually. A sudden jump in request volume can trip Anthropic's acceleration limits even while every steady-state RPM and TPM number stays under the published cap, so scale a new agent deployment up over minutes, not instantly.
• Read the limiter-specific headers, not one aggregate number. Anthropic returns separate anthropic-ratelimit-requests-remaining, -input-tokens-remaining, and -output-tokens-remaining headers; checking only an aggregate usage dashboard is what let the 47-worker, 62%-of-cap scenario above look healthy right up until it wasn't. For the admission-control side of the same problem, queue depth and in-flight counts are also better overload signals than raw request rate once you're serving your own endpoint instead of calling someone else's.
• Measure your own ceiling if you're self-hosting. A rented GPU has a real concurrency ceiling too, just one you control and can load test directly instead of inferring from 429 headers, and recent KV cache work has specifically pushed how many concurrent requests a single GPU can hold without adding hardware, as covered in vLLM's NVFP4 KV cache update. Renting a dedicated H100 instance outright removes the shared-tenant rate limit question entirely, at the cost of owning capacity planning yourself.
• Treat rate limits as flow control, not an adversary. As the same practitioner writeup on agent self-DoS puts it, "rate limits in agent systems are not a cudgel against abuse." They're the mechanism that keeps one runaway planning loop from consuming an entire shared account, which is a property worth designing with rather than fighting.

An agent fleet that's outgrown reverse-engineering 429 headers from a shared per-minute quota is the exact traffic shape guaranteed concurrency was built for.

See Spheron's managed inference plans →