A CUDA test suite agent sandbox solves a narrow, specific problem: your test fails on a laptop with no GPU, and whatever debugs it needs a real one, with root, before it can even reproduce the crash. This walkthrough covers a concrete example through end to end, from a cold clone to a fixed kernel: what sudo and a 24-hour background job actually unlock for kernel work, what compute-sanitizer and Nsight Compute output look like relayed back into chat, and what the whole loop costs.
TL;DR: How a CUDA Test Suite Agent Sandbox Debugs a Failing Kernel
- Mechanism: a CUDA test suite agent sandbox from Spheron runs on a GPU VM (H100, H200, B200, B300 and more) with sudo, so the agent reproduces, diagnoses and fixes the kernel instead of reporting pass or fail.
- Tools: compute-sanitizer's checkers catch correctness bugs; CUDA-GDB and Nsight Compute handle crashes and performance regressions.
- Controls: every deploy, Stop and terminate needs approval first, only the deployer can use it, and a connection can carry a spend limit per hour.
- Runtime: a background job runs up to 24 hours, past the 10-minute default a Racecheck pass or Nsight Compute sweep would otherwise hit.
- Cost: the sandbox carries no extra fee; Spheron H100 on-demand runs $2.65/hr as of 03 Nov 2026.
The Prompt: Pointing a CUDA Test Suite Agent Sandbox at a Failing Test
The workflow starts the same way the LoRA fine-tuning walkthrough does: add the Agent GPU Sandbox as an MCP server once, then send the instruction in chat. Spheron's own feature page gives the baseline prompt verbatim: "Clone my repo and run the CUDA tests that fail on my laptop." That line is doing real work. "Fails on my laptop" is a tell that the failure is GPU-shaped, something CPU-only agent sandboxes like E2B, Daytona or Firecracker, covered in Spheron's agent code execution sandbox guide, structurally cannot reproduce, because none of them attach a GPU.
Once the agent clones the repo and runs the suite, a second, more specific prompt picks up where the first leaves off: "This kernel crashes on Hopper. Find out why and fix it." That's the prompt this walkthrough follows. The agent provisions an H100-class VM (Spheron's sandbox supports H100, H200, B200, B300 and more, single- or multi-GPU), clones the repo with git clone run directly on the machine, and installs whatever build toolchain the test suite needs. A large corpus or dataset beyond the code itself has to come down the same way: file transfer between chat and the sandbox VM is capped at 5MB per file, so anything bigger is pulled with curl, git, or huggingface-cli run on the VM, not pasted through chat.
From there the loop is the one a human would run by hand: reproduce the failure, diagnose it with the right tool, patch the kernel source, rebuild, and rerun the test. The difference is who's typing the commands, and whether that person has a GPU attached to do it with.
What sudo Access and Background Jobs Change for Kernel Work
A CUDA kernel crash rarely explains itself from a stack trace. Finding the actual cause needs access that a sandboxed, unprivileged shell doesn't have, and a time budget longer than a typical tool-call timeout. Inside Spheron's Agent GPU Sandbox, agents run shell commands as the login user with sudo, which changes several things specifically for kernel debugging:
- Driver and toolkit control. Installing a specific CUDA Toolkit version, or a matching driver to reproduce a report tied to one, is a root-level operation. Without it the agent is stuck debugging whatever's already on the box.
- ptrace and core dumps for CUDA-GDB. Attaching a debugger to a running kernel, or loading a GPU core dump from the moment of a crash, needs the process permissions sudo provides.
- Hardware performance counters for profiling. Most serverless GPU platforms block the counter access `ncu` and `nsys` need, failing with
ERR_NVGPUCTRPERM. Bare-metal or VM-level root, the same access a debugging session needs for ptrace, is what avoids that error. - A 24-hour background job ceiling instead of a 10-minute default. That's the difference between a slow compute-sanitizer
racecheckpass or a multi-configuration Nsight Compute sweep getting killed mid-run and being left to finish. - Approval gates around all of it. Every deploy, Stop, Start and terminate shows a price preview and needs explicit approval, only the person who deployed the instance can use its sandbox, the VM opens no inbound port, and a connection can carry a spend limit per hour. Every command is logged by text and exit code, and your own SSH key keeps working throughout, so you can attach to the same VM mid-debug and see exactly what ran.
That last bullet answers whether handing an agent root on a GPU box is safe: the sandbox doesn't remove the risk, it bounds how far a bad command can go and makes it visible after the fact.
Reading compute-sanitizer and Nsight Compute Output Back in Chat
Once the agent has sudo and room to run a long command, the diagnosis leans on tools that exist independent of any agent. NVIDIA's Compute Sanitizer is a functional-correctness checking suite shipped in the CUDA Toolkit with four tools: Memcheck for out-of-bounds and misaligned memory access and leaks, Racecheck for shared-memory race conditions, Initcheck for reads of uninitialized device global memory, and Synccheck for invalid __syncthreads() or __syncwarp() usage. It's the tool NVIDIA now points developers to directly: cuda-memcheck was deprecated starting with CUDA 11.6 and removed outright in CUDA 12.0, with compute-sanitizer as its replacement. The agent runs compute-sanitizer --tool memcheck ./test_binary the same way a person would; the first 1MB of output streams back into chat, and the full output saves to /var/log/spheron-agent/ on the VM for anything longer.
If compute-sanitizer finds nothing but the test still fails or hangs, the next step is CUDA-GDB, NVIDIA's extension to GDB for debugging CUDA applications. This combination catches real production bugs, not just test-suite ones: a 30% tokens/sec regression from a CUDA driver update showed why kernel-level debugging tools matter.
When the failure is a timing or occupancy problem rather than a crash, the agent reaches for Nsight Compute instead, targeting the kernel with ncu --replay-mode kernel --set full --kernel-name <name>. This is where the sandbox's shell-only nature shows its limit: the .ncu-rep trace still has to come down to a local machine and open in the Nsight Compute desktop GUI to read the roofline chart, exactly like profiling any other remote GPU box. A failing test that turns out to be a performance rather than correctness bug is worth checking against what the occupancy numbers in that chart actually mean before chasing the wrong fix.
Running the Loop Yourself, Step by Step
The walkthrough above compresses what would otherwise be a day of manual debugging into a handful of chat messages, but the mechanics behind it are the same five steps on every repo, not something specific to this one Hopper crash. Set the sandbox up once and the next failing kernel runs through the identical process.
Connect the MCP server once, per machine or per Claude Code session:
claude mcp add --transport http spheron https://connect.spheron.ai/mcpFrom there, the process looks like this on any kernel bug, not just the one this post follows:
- Send the first prompt, naming the repo and the symptom: "Clone my repo and run the CUDA tests that fail on my laptop."
- Once the agent confirms which test is failing and on which GPU, hand it the second prompt: "This kernel crashes on [GPU]. Find out why and fix it."
- Approve the deploy preview when it shows the GPU tier and the price. This is the only gate standing between the prompt and a running VM.
- Let the agent run compute-sanitizer first. If that pass comes back clean, approve the longer CUDA-GDB or Nsight Compute step as a background job instead of a blocking one, so a slow Racecheck pass or a full Nsight sweep has room to finish.
- Read the patch the agent proposes against the original source before it rebuilds and reruns the test. The agent will tell you what it changed; check it did what it said.
- Approve Stop or terminate once the agent reports a passing test and the final diff.
Every approval gate, file-transfer limit and spend cap mentioned in this post is documented in full in Spheron's Agent GPU Sandbox docs, which is worth reading once before the first real run.
Using the Sandbox Properly: What to Hand Off and What to Check
Running this from chat instead of a terminal only saves time if the agent does the typing while you still own the judgment calls. A few habits separate a loop that actually frees up your afternoon from one that just moves the same manual work somewhere less visible.
Let compute-sanitizer run all four of its tools in sequence before escalating to CUDA-GDB. Memcheck, Racecheck, Initcheck and Synccheck each catch a different class of bug, and running all four instead of guessing which one applies is cheap: a few GPU-minutes apiece against the hour a manual debugger session can burn chasing the wrong hypothesis. Tell the agent explicitly to try all four before it reaches for anything heavier.
Push the slow steps into a background job and stop watching the chat window. A Racecheck pass on a large kernel, or an Nsight Compute --set full sweep, can run for the better part of an hour, and there's nothing useful about staring at the output while it streams. Set a spend limit per hour on the connection first, then let the 24-hour background ceiling do its job and come back once the agent reports a result instead of a timeout.
Read the full log before trusting a one-line summary. Only the first 1MB of output streams into chat; the complete run lands at /var/log/spheron-agent/ on the VM. An agent that found three issues and only mentions the loudest one in its summary is a real failure mode worth checking for, not a hypothetical.
Keep your own SSH session open to the same VM while the agent works. It costs nothing, and it means you can see exactly what ran, mid-debug, without interrupting the loop.
Cost and Time: From Failing Test to Fixed Kernel
The two cheap steps in this loop are compute-sanitizer and CUDA-GDB: both run the test binary once or twice with no replay overhead, so they bill a few minutes of GPU time at whatever hourly rate the instance is running. Nsight Compute is the expensive step, because ncu --set full replays each kernel to collect every counter set. Spheron's own profiling benchmarks put a --set full pass on a 70B-scale model forward at 20-30 GPU-minutes, costing $0.55-$0.83 on H100 SXM5 on-demand at the 11 May 2026 rate; --set default for a single replay pass cuts that to under $0.15, at the cost of incomplete roofline data. A kernel-level test suite is almost always smaller than a full model forward, so a sweep across a handful of failing kernels should land inside that range, not above it.
Stacked together, a realistic loop runs: a few minutes to clone and build, a compute-sanitizer pass for well under a dollar, and, only if that comes back clean, an Nsight Compute pass in the $0.15-$0.83 range depending on scope, plus whatever the fix-rebuild-retest cycle adds on top. That excludes the few minutes the VM itself takes to go from deploy approval to ready, immaterial next to a multi-step debugging session and wasteful only if the whole job was a one-line compile check.
Pricing fluctuates based on GPU availability. Spheron's H100 rate above is live as of 03 Nov 2026; the Nsight Compute cost figures reflect benchmarks recorded on their own stated date and may have changed. Check current GPU pricing → for live rates.
A test that fails on a laptop because there's no GPU to run it on isn't a test an agent can fix from inside a chat window alone. It needs the same hardware and the same tools a CUDA engineer would reach for, with controls that make sudo on a GPU box something you can hand over without losing track of what ran.
Debugging a CUDA kernel from chat only works if the agent can get its hands on a real GPU, with root, long enough to actually reproduce the crash.
Frequently Asked Questions
It can do more than report pass or fail, as long as it has a real GPU under it. Spheron's Agent GPU Sandbox documents the prompt 'This kernel crashes on Hopper. Find out why and fix it.' as a named example, which only works because the agent gets shell access with sudo on a GPU VM: it can run compute-sanitizer, attach cuda-gdb, pull an Nsight Compute trace, edit the source, and rerun the test, not just read the failure message a CI job already produced.
The same ones a CUDA engineer would reach for on a terminal. NVIDIA's compute-sanitizer catches memory, race, uninitialized-read and sync bugs with its Memcheck, Racecheck, Initcheck and Synccheck tools. CUDA-GDB steps through active kernels, blocks, threads and warps, and can load a GPU core dump from a crash. Nsight Compute profiles a specific kernel's counters when the bug is a performance regression rather than a correctness crash.
The sandbox is built to bound the risk rather than remove it. Every deploy, Stop, Start and terminate shows a price preview and needs your explicit approval, only the person who deployed the instance can use its sandbox, the VM opens no inbound port, and every command the agent runs is logged with its text and exit code. A connection can also carry a spend limit per hour, and your own SSH key keeps working the whole time so you can look at exactly what the agent is doing mid-debug.
It depends on which tool finds the bug. A single compute-sanitizer or cuda-gdb pass over a small test suite runs in minutes on the GPU's normal hourly rate. A full Nsight Compute sweep is the expensive step: a '--set full' pass on a 70B-scale forward took 20-30 GPU-minutes and cost $0.55-$0.83 on H100 SXM5 on-demand at Spheron's 11 May 2026 rate, against under $0.15 for a lighter '--set default' pass. Spheron's own H100 on-demand rate runs $2.65/hr as of 03 Nov 2026, and the sandbox itself carries no separate fee.
