Engineering

Nsight Systems vs Nsight Compute: Profiling LLM Inference

Back to BlogWritten by Published Nov 17, 2026
Nsight Systems vs Nsight ComputeNsight Systems Profilingnsys TraceGPU ProfilingLLM InferenceInference OptimizationGPU Kernel Development
Nsight Systems vs Nsight Compute: Profiling LLM Inference

Throughput drops, and the dashboard just says slow. That's a symptom, not a diagnosis, and chasing it by reaching for a bigger GPU or a different batch size before you know which resource is actually the bottleneck burns money with no guarantee of a fix. The real question underneath that symptom is Nsight Systems vs Nsight Compute: one reads the whole application timeline, the other opens a single kernel's hardware counters, and reaching for the wrong one first wastes the hour you meant to save. This post isn't a tool-by-tool reference; it walks one real nsys command end to end, shows what the timeline actually looks like when it's lying to you, and names a bottleneck we found and fixed on our own fleet, then turns that finding into a rental decision.

TL;DR: Nsight Systems vs Nsight Compute for LLM Inference

Nsight Systems reads the whole inference timeline to find which phase is slow; Nsight Compute opens one kernel's hardware counters to find out why. Run nsys first, every time.

  • Order: Nsight Systems profiles the full run with no source changes, so it goes first; Nsight Compute then opens only the one kernel nsys flags, following NVIDIA's own documented workflow of right-clicking "Profile Kernel" inside a Nsight Systems report.
  • Capture: a scoped nsys trace uses -y to skip boot and warmup and -d to bound the capture window, so you get a 30-second decode trace instead of a multi-minute recording.
  • Access: hardware counter collection needs root-level access to the GPU; Spheron's H100 SXM5 on-demand instances provide that as dedicated VMs, running $3.96/hr on-demand as of 19 Nov 2026.
  • Real bottleneck: on one 8xB200 node, a cold start that should have taken minutes took 43.6 minutes, and the symptom (GPU rows idle, host busy) pointed straight at CPU thread contention, not the GPU, as measured on 16 Oct 2026.

Why "Low Throughput" Is Not a Diagnosis

"Slow" is a measurement, not a cause. Before opening any profiler, rule out the two things that make a perfectly healthy serving stack look broken on paper.

The first is measuring during warmup. Several of the kernels in a modern serving stack compile or tune themselves on first use, and a benchmark run against that first pass reports the compile time, not the steady-state speed. On one 8xB200 node on 15 Oct 2026, GLM-5.3-Flash's first sweep after boot ran at 34 tokens/sec per user at 32 concurrent users with 40-second first-token latency, while its linear-attention kernels were still compiling; warm, the same model and node held 129 tok/s with 0.84-second first tokens. DeepSeek V4 Flash showed the same pattern on the same date: 36 tok/s at 32 users with 8-second first tokens on the first sweep while its FP4 kernels tuned, 113 to 116 tok/s warm. A cache volume that keeps compiled kernels across boots is the fix; without it, every restart re-pays that compile tax and looks like a regression to a dashboard that can't tell the difference.

The second is treating a vague symptom as a root cause. Low throughput could mean the GPU is doing too little math per byte moved (memory-bound), too much math relative to what the hardware can issue (compute-bound), or nothing at all while the CPU does something else (host-side launch overhead). Those three have different fixes, and none of them is "add more GPUs." If you're starting from the symptom level rather than a specific stalled kernel, why your LLM inference is slow is the place to start; this post picks up from there, where you already suspect something specific and need to find it in a trace.

Capturing a Trace: the Exact nsys Command We Used

Nsight Systems profiles an application with no application changes required to enable profiling, which is the reason it goes first: you don't need to add instrumentation to find out where time is going. Point it at the command that starts your inference server and it records everything by default.

The naive version of that command records the whole run, boot included, which buries the thirty seconds you actually care about under minutes of model load and warmup. Scope it instead:

bash
nsys profile \
  -t cuda,nvtx,osrt \
  -y 120 \
  -d 30 \
  -o decode-trace \
  --force-overwrite true \
  -- python -m vllm.entrypoints.openai.api_server \
       --model <your-model> \
       --tensor-parallel-size 8

Each flag is doing a specific job. -t cuda,nvtx,osrt sets which APIs nsys traces; `nsys profile`'s `-t`/`--trace` flag takes a comma-separated list with no spaces and defaults to cuda, opengl, nvtx, and osrt if you leave it off. osrt is what shows host-side activity, thread creation and scheduling, so keep it in if you suspect the CPU and not just the GPU. -y 120 is the delay flag and -d 30 is the duration flag; together they tell nsys to wait 120 seconds (past boot and warmup) before it starts recording, then record 30 seconds of steady-state traffic. If you'd rather tie the window to a code region instead of a wall-clock guess, nsys profile also supports -c/--capture-range with none, cudaProfilerApi, hotkey, or nvtx as the trigger, plus the same -d/-y pair to scope it, which is the better option once you've instrumented your serving loop with NVTX ranges.

One tradeoff worth knowing before you run this: Nsight Systems' hardware trace mode is faster than its older software-instrumented trace, but tracing every CUDA API still adds measurable runtime overhead, so a traced run that takes 10 to 15 percent longer than usual is not proof your serving stack got slower.

Pull the file off the GPU host and summarize it before opening the GUI:

bash
nsys stats --report cuda_gpu_kern_sum --report cuda_gpu_trace decode-trace.nsys-rep

cuda_gpu_kern_sum gives a ranked list of kernels by total GPU time, which is usually the first thing worth reading. cuda_gpu_trace gives the per-event trace if you need individual launches. Running `nsys stats --help-reports` lists every report the installed version ships, and it's worth scanning once; there are reports for NVTX ranges, OS runtime calls, and NCCL collectives that don't get opened nearly as often as they should.

Reading the Timeline: Compute Gaps, Memory Stalls, and Host-Side Launch Overhead

Open the .nsys-rep in the Nsight Systems GUI and you're looking at rows stacked by thread and device, time running left to right. Three patterns account for most of what goes wrong in inference serving.

Compute gaps. The CUDA HW row shows kernels as solid blocks. A gap between blocks, where the GPU is doing nothing, is dead time by definition; the question is why. NCCL collectives in tensor-parallel decode show up as their own named blocks rather than gaps, so if you see a true empty gap with no NCCL activity nearby, check what the CPU row is doing in that same window before assuming it's a GPU problem.

Memory stalls. A kernel that's running, not gapped, but still slow is the harder case, because the timeline alone won't tell you why. A kernel at low arithmetic intensity, decode-phase attention is the classic example, spends most of its time waiting on HBM rather than computing, and that shows up in the timeline only as "this kernel takes longer than it looks like it should," not as an idle gap. Our roofline model guide covers the arithmetic intensity math that tells you whether a given kernel should be memory-bound before you go looking for a bandwidth problem in the trace.

Host-side launch overhead. This is the pattern that catches the most people off guard, because it looks exactly like a compute gap on the GPU row, but the CPU-side osrt or Python stack row is busy the entire time. The GPU isn't stalled on anything GPU-related; it's waiting for the host to finish issuing the next batch of work. Eager-mode PyTorch decode loops, where every step round-trips through Python before the next kernel launch, are the most common cause, and CUDA graph capture is the standard fix, since it replays a whole batch of kernel launches as one unit instead of issuing them one at a time. Our torch.compile and CUDA graphs guide walks through capturing that graph for an inference loop.

Reading these three apart is the entire value of a system-level trace. None of them needs a single hardware counter, which is the segue into when you actually do need one.

Nsight Systems vs Nsight Compute: Which One Answers Your Question

The honest answer to "Nsight Systems vs Nsight Compute" is that they answer different categories of question, and NVIDIA's own Nsight Compute documentation is explicit about the order: open a Nsight Systems report first, find the kernel on the critical path that's underperforming, then right-click it and choose "Profile Kernel" to launch a preconfigured Nsight Compute session on just that kernel. You don't choose one tool over the other; you use Nsight Systems to decide whether Nsight Compute is even worth opening.

Map that back onto the three patterns from the timeline section above, and the choice gets concrete rather than abstract:

What the timeline showed youOpen nextWhy
A compute gap with no NCCL activity nearbyNsight Systems, stillConfirms it's a real gap, not a host stall, before you touch a counter
A kernel that's running but slower than its size suggestsNsight Compute, on that one kernelNeeds replayed hardware counters nsys never collects
GPU row idle, host-side osrt row busy the whole timeNsight Systems' own timeline is the full answerNothing is running on the GPU for Nsight Compute to profile
"The server feels slow," no kernel named yetNsight SystemsOpening Nsight Compute here means guessing which kernel to profile

The mistake worth naming explicitly is opening Nsight Compute first on a kernel you picked by eye. NVIDIA's documentation notes that even inside Nsight Compute, its own API Statistics view "cannot be used as a replacement for Nsight Systems" when the question is about CPU-side performance, a direct acknowledgment that the two tools aren't interchangeable even when they're both open on the same machine. Once you've narrowed things to a specific kernel, the full roofline read, the complete tool lineup (torch.profiler, Triton Proton), and how to get hardware counters on a remote host without a display all live in our Nsight Compute and PyTorch Profiler guide, which picks up exactly where this section stops.

A Real Bottleneck From Our Fleet (and What Fixed It)

Not every bottleneck needs a kernel-level trace to find. The clearest example from our own fleet didn't involve Nsight Compute at all, because the GPU wasn't the thing that was slow.

Standing up DeepSeek V4.1 Flash on a fresh two-copy 8xB200 node took 43.6 minutes from launch to serving, measured on 16 Oct 2026. That number alone doesn't say where the time went, and a timeline-level view is exactly the tool for the question "where did 43 minutes go," because GPU rows sitting mostly idle while something else runs is the host-side launch overhead pattern described above, just stretched from microseconds to minutes.

Two causes accounted for almost all of it. 15.8 minutes came from vLLM handing every GPU worker 40 CPU threads on a node that didn't have 40 threads to spare per worker; giving each worker its actual share of the node's CPUs cut model load to under a minute. Another 13 minutes came from vLLM re-running kernel tuning on every single boot, because its cache folder name kept changing between runs, so the tuned configuration from the last boot was never reused; reusing that saved tuning removed the 13 minutes outright. Starting both model copies at once instead of sequentially took care of most of the rest. The node now goes from VM boot to both copies serving in 9.8 minutes.

None of that is a Nsight Compute finding. It's the kind of thing that shows up as a CUDA HW row doing almost nothing while a CPU row stays busy for a stretch measured in minutes rather than microseconds, which is the same pattern this post's timeline section describes at the per-kernel scale. The lesson generalizes past this one node: before reaching for a kernel-level counter tool, check whether the GPU is actually running anything during the slow window. If it isn't, Nsight Compute has nothing to tell you, and the fix is almost always on the host side.

What the Bottleneck Tells You About Which GPU to Rent Next

The category of bottleneck you find determines what buying a different GPU would actually get you, and it's worth being specific about that before spending more on hardware.

A host-side launch overhead problem, like the CPU thread contention above, is not fixed by a faster GPU. The GPU was never the constraint; more vCPUs per worker, fewer tenants per host, or persisting a compiled-kernel cache across restarts fixes it for a fraction of the cost of a hardware upgrade.

A memory-bound kernel is a real reason to look at bandwidth, not just at peak FLOPS. H100 vs H200 covers the case directly: the H200's higher-bandwidth HBM3e outruns the H100 specifically on the memory-bound side of inference, which is where decode-phase attention lives at the batch sizes most production serving actually runs. If your roofline read puts a kernel well to the left of the ridge point, that's the upgrade that pays off, not more compute.

A compute-bound kernel is the one case where a bigger GPU genuinely helps, and even then the tradeoff isn't just raw speed per user. Splitting one 8xB300 node into two 4-GPU copies gave a lower per-user speed than running one 8-GPU copy on 8xB200: for a single user running GLM-5.3, that's 254 tok/s against 324 tok/s, but the B300 split served twice the concurrent agent sessions, 64 against 32. Whether that trade is worth it depends on whether you're optimizing for one fast session or many concurrent ones, and that's a question the trace can't answer for you; it can only tell you which resource you're actually short of.

Spheron's current on-demand and spot catalog spans B300, H100, B200, H200, MI300X, GH200, A100, RTX PRO 6000, RTX 5090, RTX 6000 Ada, L40S, and RTX 4090, so once a trace tells you whether you're short on bandwidth, compute, or host CPU, you can rent the specific GPU that fixes it instead of the next one up by default. $3.96/hr on-demand and $2.20/hr spot H100 SXM5, as of 19 Nov 2026, is cheap enough to re-run the same trace on a smaller GPU first, to confirm a bottleneck is architecture-specific before committing to anything bigger. The instance comes with the root-level access nsys and ncu both need to collect hardware counters at all, which most shared serverless GPU hosts block outright. That root-access point follows from how Spheron describes the instances themselves, "a fully provisioned VM or bare metal instance with your selected GPU" including "a dedicated IP address with full root access," not from a published profiling guarantee; the pricing page itself never uses the word profiling anywhere. If you're already profiling on a GPU box you or your employer owns, none of this is a reason to move it; the point only matters once you're choosing a rental.

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 19 Nov 2026; check current GPU pricing → for live rates across the catalog.


Finding the bottleneck is the hard part. Renting the GPU that actually fixes it is the easy part once a trace has told you which resource you're short of.

H100 on Spheron → | View all GPU pricing → | Get started on Spheron →

FAQ / 04

Frequently Asked Questions

Because Nsight Compute profiles one kernel at a time, and the only way to know which kernel deserves that depth is to have already seen it on a Nsight Systems timeline next to everything else running during inference. NVIDIA's own Nsight Compute documentation sets out the same order: open a Nsight Systems report, find the kernel on the critical path that's underperforming, then right-click it and choose 'Profile Kernel' to hand it to Nsight Compute. Opening Nsight Compute first means guessing which kernel to profile.

Use nsys profile's -y/--delay flag to skip past model load and warmup, and -d/--duration to bound the capture to a fixed window, for example -y 120 -d 30 to skip the first two minutes and capture 30 seconds of steady-state decode traffic. Nsight Systems' User Guide documents a -c/--capture-range flag too (none, cudaProfilerApi, hotkey, or nvtx), which gives finer control if you want the capture tied to a specific code region instead of wall-clock time. -t/--trace picks which APIs to record; the default is cuda, opengl, nvtx, and osrt.

The CUDA HW row goes idle for a stretch while a CPU-side row, usually osrt, stays busy. That pattern means the GPU is waiting on the host, not the other way around; common causes are Python dispatch overhead in an eager decode loop, CPU thread contention during process startup, or disk I/O while loading model weights. Nsight Compute can't tell you this: NVIDIA's own Nsight Compute documentation says its API Statistics view 'cannot be used as a replacement for Nsight Systems' for CPU-side questions.

Yes, as long as the instance gives you root-level access to the GPU, since both tools need that to collect hardware performance counters. A sandboxed or heavily multi-tenant serverless host is more likely to block counter collection outright.

// Try It Yourself

Try It on Real GPUs

The GPUs behind these guides are the ones you can rent here: H100s, H200s, B200s, and more, billed per minute after a 20-minute minimum runtime, with no contracts. Pick one and you are live in under two minutes.

View Pricing
→
Or talk to our team
Spheron GPU instance going live and linking data centers across the globe