Agentic RL infrastructure is really two systems wearing one name. An RL environment, in this context, is just whatever the agent's actions act on and whatever hands back the reward: a shell, a test harness, an API, a simulator. One system is a fleet of CPU sandboxes running the actual agent trajectory against that environment: shell commands, test suites, tool calls, multi-turn back-and-forth that can take minutes or hours before a reward ever shows up. The other system is a GPU trainer that never sees any of that. It only sees the finished rollout and the reward number attached to it. Confuse the two, size them the same way, or put them on the same pool of hardware, and you'll either pay GPU prices for CPU work or build a training loop that can't survive a single preempted node.
TL;DR: Why Agentic RL Splits CPU Sandboxes From GPU Trainers
- Split: Agentic RL rollouts (shell, tests, tool calls) run on CPU sandboxes; only the finished trajectory and reward reach the GPU trainer, per DeepSeek's September 2026 DSec paper, which "decouples rollout lifetime from trainer lifetime."
- Scale: DSec's sandbox layer runs roughly 3 million sandbox instances a day with peak concurrency near 380,000, all outside the preemptible GPU pool.
- Error rate: 3.1% of trajectories in LMSYS's October 2026 agentic RL benchmark ended in an infrastructure error, not a wrong answer.
- Cold start: Cloud sandboxes in that benchmark reached 1.9s median readiness versus 11.7s for local Docker under load.
- Spheron: Spheron's Agent GPU Sandbox bills trainer nodes at the normal on-demand rate, not spot. Compare GPU pricing.
The Two-Tier Architecture: CPU Sandbox Fleet, GPU Trainer Node
Agentic RL infrastructure splits into two tiers that almost never share hardware. The rollout tier runs the agent's actual trajectory, shell access, file edits, test execution, tool calls, often hundreds of them per episode. The trainer tier only runs the backward pass and the optimizer step on batches of finished trajectories and their rewards. A third, optional tier sits in between: the GPU that serves the policy's own tokens back to the agent mid-rollout.
| Layer | What runs there | Typical hardware | Lifespan per unit | On preemption |
|---|---|---|---|---|
| Rollout sandbox fleet | Agent shell, test runners, tool calls, multi-turn trajectory state | CPU-only containers or microVMs | Minutes to hours per rollout | State can often pause and resume |
| Policy-serving GPU | Generates the tokens the agent acts on during the rollout | Pooled or dedicated inference GPU | Runs continuously through the rollout | Mid-generation loss can force a rollout restart |
| GPU trainer node | Backward pass, optimizer step, policy weight update | On-demand GPU (H100, H200, B200) | Hours to days per training run | Full restart unless checkpointed |
DeepSeek's DSec infrastructure is the clearest public description of the rollout tier at production scale. It splits rollout execution into an agent sandbox and a worker container, and both "run outside the preemptible GPU pool." Together they hold the full rollout state, which is what lets a preempted GPU job reconnect and continue "without reconstructing execution through command-log replay" once the trainer is back up. The paper's own framing is direct: the design "decouples rollout lifetime from trainer lifetime."
For the environment and framework layer that produces these rollouts in the first place, Gymnasium, Prime Intellect's Environments Hub and verifier-based reward functions are the building blocks; see our RL environments on GPU cloud guide for how those frameworks size against GPU and CPU resources. On the trainer side, frameworks like TorchForge and Monarch wire rollout workers into a single-controller trainer loop that assumes exactly this two-tier split: rollout workers that can scale and fail independently, and a trainer process that only ever consumes finished batches.
Where the Environment Itself Needs a GPU
Most agentic RL environments are pure CPU work: a shell, a test harness, a browser, an API client. But some reward functions can't be graded without a GPU of their own, separate from both the rollout sandbox and the trainer.
Thunder Compute's CUDA-kernel-reward example is the clean case. The environment "compiles kernels in a CPU Docker sandbox," then "sends verification and profiling jobs to a dedicated GPU sandbox pool of 128 NVIDIA H20 GPUs." Correctness and speed can't be scored without actually running the kernel on real silicon, so that one piece of the environment gets its own GPU pool, isolated from the sandbox fleet that compiles the code and from the trainer that updates the policy afterward. The same pattern shows up anywhere the reward depends on physical simulation, rendering, or any other GPU-bound computation the agent's actions trigger.
Published agent RL setups generally run "128-512 rollouts in parallel, mostly in CPU sandboxes, with the model served on separate GPUs," per Thunder Compute, which is the three-tier split holding at a scale small enough for a single team to run. Reward model training is the other common case where the loop genuinely needs a GPU end to end; our GPU sandbox reward model training walkthrough shows the exact prompt and GPU choice for running that step from a chat session instead of a standing cluster.
What the CPU Sandbox Fleet Actually Costs You
The sandbox fleet isn't free, it's just cheap relative to GPU time, and it has its own scaling and isolation problems that show up before the GPU bill does.
DSec's own numbers show the range: a creation rate exceeding 5,000 sandbox instances per second, single jobs requesting up to 32,000 sandbox instances, and individual nodes hosting up to 800 microVMs or 3,200 containers. The reason that density works at all is utilization: about 90% of DSec's container and microVM sandboxes use no more than 5% of their requested CPU capacity on average, since most of a rollout's wall-clock time is the agent waiting on a tool response or an LLM call, not burning CPU.
Scheduler choice changes the bill more than most teams expect. Daytona's own benchmark of a 50-action SWE-agent-style RL workload at 10,000 trajectories measured aggregate worker-hours of 36 on Daytona against 56 on Kubernetes, 39 on Docker, 85 on EC2, and 449 on ECS/Fargate, for the identical sandbox execution path. That's more than a 12x spread for running the same rollouts on the same workload, which makes the sandbox scheduler itself a cost decision, not an implementation detail. For the isolation mechanics underneath these numbers, bare container versus microVM, GPU passthrough, and how E2B, Daytona and Firecracker compare on a self-hosted bare-metal GPU box, see our AI agent code execution sandbox guide.
Cost Math Per RL Step: Rollouts x Rollout Time x GPU-Hours
The formula is simple even when the infrastructure underneath it isn't: cost per RL step is roughly (rollouts × average rollout wall-clock time × CPU sandbox rate) plus (GPU-hours the trainer actually runs × GPU hourly rate). The first term is usually small. The second term is where the bill actually lives.
Say a batch of 256 rollouts averages 6 minutes of wall-clock time each on CPU sandboxes running in parallel. Even at that concurrency, CPU sandbox time is a small fraction of what the same wall-clock time would cost on a GPU, which is the whole argument for keeping rollouts off GPU hardware in the first place. The trainer side is the opposite: an on-demand H100 node running at $2.98/hr for the 20-30 minutes it takes to backward-pass and update on that batch costs a few dollars regardless of whether the batch came from 256 rollouts or 2,560 sandboxes. Scale the rollout count up by 10x and the sandbox-side cost barely moves the needle; scale the trainer's GPU-hours up by 10x (bigger model, more gradient steps, larger batch) and the bill moves in lockstep. Size the sandbox fleet for concurrency and tail latency, and size the GPU budget around trainer hours, not rollout count.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 16 Nov 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.
For the trainer-side infrastructure itself, actor, critic and reward model stages, multi-node PPO or GRPO, and how 70B-class runs size across H100 versus B200, see our RLHF training infrastructure guide covering verl, OpenRLHF and TRL.
Why Preemption Breaks Long Rollouts (and Why the Trainer Stays On-Demand)
A classic RL env step is seconds long. An agentic rollout can run for minutes to hours of chained tool calls before it produces a reward. That difference is the whole argument for keeping the trainer, and often the policy-serving GPU, on on-demand capacity even when the rollout sandboxes themselves are cheap and disposable.
This isn't a case against spot for every rollout-adjacent GPU, and the sibling guides linked above aren't wrong, they're about a narrower slice of agentic RL than the long tool-using rollouts this post is built around. Our TorchForge and Monarch guide and our RL environments guide, both linked above, are themselves about agentic RL, and both recommend spot for the policy-serving rollout worker: a stateless GPU that runs the policy forward-only against many parallel environments and holds no optimizer state. On preemption it loses one in-flight batch, which TorchForge's own numbers put at 50-200 steps of generation, not nothing. What makes that tolerable in those setups is trajectory length and where the trajectory's state actually lives, and the two guides size that differently. TorchForge's own worked examples size for 8-32 tool calls per episode, short enough that a lost batch is bounded. The RL environments guide's worked examples run a different unit altogether: a verifier scores a batch of 512-2,048 parallel completions per prompt rather than stepping through a chained tool-calling episode, so what's lost on preemption there is a batch of single-shot generations, not a trajectory in progress. Either way, DSec's own design (quoted above) keeps the state that matters, the agent sandbox and its tool-call history, outside the GPU entirely, so a preempted rollout GPU reconnects to a sandbox that remembers where it was instead of restarting a trajectory from nothing. Stretch either of those short units out to the hundreds of chained calls and minutes-to-hours episodes this post is about, and the same lost batch now holds far more accumulated work per preemption. That's a difference of degree on the same architecture, not a different kind of GPU, which is why the policy-serving GPU gets harder to justify on spot the longer a single rollout actually runs.
Infrastructure failure is already a measurable tax before preemption enters the picture. LMSYS's October 2026 benchmark found that 3.1% of trajectories ended in an infrastructure error, "which scores 0 just like a wrong fix," unless those episodes are explicitly logged and filtered out of training. Google's GKE Agent Sandbox team found a similar gap at the cluster level: time-to-first-command for RL rollouts fell from 44-85 seconds on raw Kubernetes to 1.1-8.8 seconds, roughly a 10x improvement, and worst-case tail latency fell from 7.5 minutes to under 10 seconds, about 45x, because a synchronous RL step can't start training until the slowest sandbox in the batch is ready. Mistral AI's Jean-Malo Delignon described the practical effect: it lets them "orchestrate hundreds of thousands of secure environments across clusters and handle spikes of over 30,000 sandboxes on a single cluster" for RL training.
None of that solves preemption on the GPU side, and the two failure modes differ in kind. Thunder Compute notes that spot GPU sandboxes on providers like Daytona and Modal can be "terminated without notice," and that "an evicted rollout loses everything since the episode began," so "every eviction adds a rerun and delays its training step." A trainer job reclaimed mid-step has the same problem unless it's checkpointed deliberately. We learned this running a GPT-2 training node on 8x H100 spot in November 2026: the first node had no volume and no checkpoints, and it was stopped at step 956, 17 minutes in, with the $10.22 it cost buying nothing we could resume. The second node added a 100GB network volume that checkpointed every 500 steps in about 55 seconds each, so a reclaim would have cost at most 500 steps instead of the whole run. Most agentic RL setups skip that discipline entirely by keeping the trainer on-demand instead.
The policy-serving GPU has its own version of this, worse when the eviction lands during startup. We separately measured an 8x B300 spot node taking 8 minutes 47 seconds to download 510GB of model weights and 40 minutes 19 seconds to finish starting, in November 2026, mostly CUDA graph capture and FlashInfer autotuning rather than reading weights off disk, roughly $40.60 before it answers a single request. Get reclaimed anywhere in that window and the next attempt pays the same cold-start tax again, on top of whatever rollout progress was lost.
Contrast that with what's possible when the workload is stateless. Every 8-GPU spot type we used for inference serving was reclaimed at least once on 14 October 2026, and on 16 October one B200 region stopped five of our spot VMs in a day, frequent enough that we run every model as two copies on separate nodes with automatic recovery. Killing a spot node mid-stream in a live test on 16 October 2026 let the in-flight response finish on the other copy in about 7 seconds with zero failures. That redundancy works because a serving request has no memory of its own beyond the current response. A rollout mid-trajectory doesn't have a warm standby copy of its own command history sitting idle elsewhere, so the same spot risk that's a minor blip for stateless serving is often a total loss for the rollout.
Running the Trainer Node From Chat With an Agent GPU Sandbox
Given that the trainer (and usually the policy-serving GPU) needs to stay on reliable, on-demand capacity, the practical question is how a small team operates that node without standing up a cluster for it. Spheron's Agent GPU Sandbox does that from inside an existing agent chat session: an MCP-connected agent in Claude, Cursor, Codex CLI or any remote MCP client provisions its own GPU VM and runs a command in the background for up to 24 hours while streaming output back to the chat.
The approval model fits a long, expensive trainer run better than it fits a quick throwaway job. Every deploy, stop and terminate goes through a preview showing the price and status, gated by the connection's own spend limit, so an agent can't quietly run up a bill on a multi-hour training step without a human having already set the ceiling. The sandbox program only connects outward, opens no inbound port, and logs each command's text and exit code, with the full output saved on the machine itself so a person can tail the real training log instead of trusting only the agent's summary. Billing runs at the instance's normal on-demand rate, the same whether or not the sandbox is involved, which matches the argument above: this isn't a spot node that might vanish mid-epoch.
It's also not a documented RL feature. Neither the docs page nor the feature page mentions reinforcement learning, GRPO, or any RL framework by name; this post's RL use extends a general-purpose, approval-gated pattern for operating a GPU VM from chat, not a shipped RL feature. Two limits matter in practice. A single background command tops out at 24 hours, so a multi-day RL run has to be split into repeated sessions with checkpointing in between. And the sandbox operates one GPU VM for one agent; it isn't a fleet-orchestration layer for the hundreds or thousands of parallel CPU rollout sandboxes a production agentic RL run needs, that layer gets built separately.
A long agentic rollout can run for an hour before it produces a single reward, which is exactly the kind of job that shouldn't be sitting on a node that can disappear mid-run. Spheron's Agent GPU Sandbox lets an agent provision and run a trainer node on on-demand GPU capacity, from H100 through B300, directly from chat, with every deploy and stop requiring your approval first.
Frequently Asked Questions
Agentic RL trains a model on full multi-turn trajectories, dozens or hundreds of tool calls, shell commands and environment responses chained into one episode, instead of the single state-action-reward step that classic RL (and classic RLHF) scores. The reward usually only arrives at the end of the trajectory, which is why the trajectory itself, not a single step, is the unit the infrastructure has to protect.
The rollout's own work, running a shell command, executing a test suite, calling an API, waiting on a tool response, is control flow and I/O, not matrix math. None of it benefits from a GPU. DeepSeek's DSec infrastructure runs its agent sandboxes and worker containers entirely outside the preemptible GPU pool for exactly this reason: the GPU only earns its cost during the LLM forward pass and the trainer's backward pass.
It depends on what's running on the GPU. Short, stateless classic-RL rollout workers tolerate spot reasonably well because losing one costs seconds. A GPU trainer node or a policy-serving GPU mid-generation on a long agentic rollout is a worse fit: losing it can mean losing minutes to hours of accumulated tool calls with no cheap way to resume, which is why most agentic RL setups keep that layer on-demand.







