DGX Spark's 128GB of unified memory looks like plenty of room for a RAG stack, until a 70B-class LLM moves in and the budget for everything else collapses to single digits. The real DGX Spark use cases for retrieval-augmented generation come down to a budgeting problem: one shared memory pool, split three ways between an embedding model, a GPU-resident vector index, and the LLM doing generation, where the biggest of the three eats almost all of it. This piece treats the 128GB as a line-item budget, shows exactly where a 70B LLM leaves you, and lays out the measured throughput once everything is loaded.
TL;DR: What Are the Best DGX Spark Use Cases for Local RAG?
- Memory ceiling: DGX Spark's advertised 128GB reports roughly 119.68GB usable to the OS, and a 70B LLM at FP8 alone takes about 70GB of it.
- What's left: that leaves roughly 49.68GB for the embedding model, the FAISS vector index, and KV cache combined once a 70B model is loaded.
- Decode speed: LMSYS's October 2025 benchmark measured Llama 3.1 70B FP8 on DGX Spark via SGLang at 803 tokens/sec prefill but only 2.7 tokens/sec decode.
- CPU offload: Arm's GB10 RAG build runs query embedding on the Grace CPU in 70-90ms, an architecture option neither an RTX 5090 nor a Mac Studio can do.
- Handoff: Spheron rents H100, H200, and B200 by the minute for RAG stacks that outgrow one DGX Spark's pool. Check live GPU pricing.
DGX Spark Use Cases for RAG: The Stack, and Where Each Piece Runs
A local RAG stack is three programs, not one. An embedding model turns the incoming query, and every document in your corpus ahead of time, into vectors. A vector index holds those vectors and answers "which of these is closest to this query" fast enough to feel instant. An LLM takes the retrieved chunks and the question and writes the answer. On a discrete GPU, all three have to fit in one card's VRAM, or you're shipping tensors over PCIe or the network between them. DGX Spark's pitch is that its Grace CPU and Blackwell GPU share one coherent LPDDR5x pool, so none of that movement is strictly necessary.
That shared pool is also why DGX Spark's own use cases skew toward exactly this kind of multi-component workload instead of a single giant model. NVIDIA's own multi-agent chatbot reference build for DGX Spark bundles a RAG agent alongside coding and image-understanding agents on one box, and its default configuration uses roughly 120GB of the 128GB pool, with NVIDIA's own instructions telling users to stop other workloads first. That's the clearest signal from NVIDIA itself that a real multi-agent RAG setup, not a toy demo, is meant to use nearly the whole machine.
Does DGX Spark's Grace CPU Help With RAG Embeddings?
Yes, and it's a trick unique to this box's architecture. Arm built a practical RAG pipeline on a GB10 system that runs query embedding, E5-base-v2 at int8 precision, on the Grace CPU cores in 70-90 milliseconds, while generation (Llama 3.1 8B at Q8_0 via llama.cpp) and the FAISS vector search both run on the Blackwell GPU side. The point isn't that the CPU is faster; it's that embedding a short query is cheap enough to hand to Grace's Arm cores entirely, which frees the GPU's scarce 273 GB/s of memory bandwidth for the part of the pipeline that's actually bandwidth-starved: token-by-token decoding. An RTX 5090 or a Mac Studio has no equivalent move, because neither pairs a CPU and GPU on one coherent, addressable memory pool the way DGX Spark's Grace-Blackwell superchip does.
The same Arm build gives the clearest first-hand read of how little a small RAG stack actually needs: DRAM usage climbed from about 3.5GiB idle to about 14GiB during query execution. That's the floor. The ceiling, as the next section shows, depends entirely on whether you load an 8B generation model or a 70B one.
Budgeting Memory: Embedding Model, Vector Index, and a 70B LLM Together
A full RAG stack on DGX Spark fits three components into roughly 119.68GB of usable memory: a 70B LLM at FP8 takes about 70GB on its own, an embedding model takes a few gigabytes more, and whatever's left funds the FAISS vector index plus KV cache. At 70B, that remainder is about 49.68GB, enough for an index somewhere in the low millions of vectors before the math stops working.
FAISS's VRAM formula is deterministic: vectors × dimensions × 4 bytes, plus roughly 20% for GPU runtime overhead. Running that formula against DGX Spark's pool, with a 70B FP8 LLM and a small embedding model already loaded, looks like this:
| Component | Memory needed | Running total | Headroom left in 119.68GB pool |
|---|---|---|---|
| 70B LLM, FP8 (e.g. Llama 3.1 70B) | ~70 GB | ~70 GB | ~49.68 GB |
| Embedding model (Stella 1.5B, BF16) | ~3 GB | ~73 GB | ~46.68 GB |
| FAISS-GPU index, 1M vectors (768-dim), ~20% overhead | ~3.6 GB | ~76.6 GB | ~43 GB left for KV cache |
| FAISS-GPU index, 5M vectors (1536-dim), ~20% overhead | ~37 GB | ~110 GB | ~9.7 GB left for KV cache |
| FAISS-GPU index, 10M vectors (1536-dim), ~20% overhead | ~73 GB | ~146 GB | Doesn't fit, over budget by ~26 GB |
The 10-million-vector row is the one that matters. It isn't a soft slowdown, it's a hard allocation failure: a 70B-class LLM and a vector index that size simply cannot both live in one DGX Spark's pool at once. The 5-million-vector row is the realistic ceiling, and it leaves a thin enough KV cache budget that long contexts or more than a handful of retrieved chunks per query will eat into it fast. Swap the 70B model for an 8B one, and the entire equation changes: at roughly 70GB less memory pressure from the LLM alone, the same box could comfortably hold a 10-million-vector index with room to spare, which is closer to the kind of RAG deployment NVIDIA's own stack guidance and Arm's CPU-offload build (described above) are actually built around.
For the same formula applied to fine-tuning instead of RAG, and the source of the 119.68GB usable-pool figure this table builds on, see our DGX Spark fine-tuning guide.
Measured Throughput: Retrieval Latency and Tokens Per Second
Retrieval isn't the bottleneck in a DGX Spark RAG stack. Generation is, and the gap between the two is large enough to change what the box is actually good for.
On the retrieval side, a developer who compiled FAISS with native GPU support directly on a DGX Spark, Grace CPU and GB10 Blackwell GPU together, ran a 5-million-vector search that hit 96% GPU utilization. That's a fully-loaded, working search path, not a toy benchmark. Combined with Arm's 70-90ms query-embedding figure on the Grace CPU, the entire retrieval half of a RAG turn, embed the query, search the index, is comfortably under a tenth of a second.
Generation is a different story once the model gets large. LMSYS measured Llama 3.1 70B in FP8 on DGX Spark via SGLang at 803 tokens/sec prefill and just 2.7 tokens/sec decode. Decode is the token-by-token phase that determines how fast an answer actually streams, and 2.7 tokens/sec is well below conversational speed; a one-paragraph answer would take the better part of a minute to finish printing. Drop to an 8B model instead, and the same LMSYS SGLang benchmark shows decode throughput scaling from 20.5 tokens/sec at batch size 1 up to 368 tokens/sec at batch size 32, inside the same 128GB pool, which is the batching headroom DGX Spark's memory capacity is actually built to exploit. For the full breakdown of that scaling curve against a discrete GPU, see our DGX Spark vs RTX 5090 tokens-per-second comparison.
Between those two extremes sits the most production-shaped number available. Signal65's DGX Spark case study measured RAG inference serving Nemotron-3-Nano-30B-A3B at up to 4 concurrent users with a 7.9-second time to first token and roughly 39 tokens/sec throughput, and the same study says one DGX Spark can serve up to 8 users on that 30B model. That's a usable, if not fast, multi-user RAG endpoint, built on a model roughly 2.3x smaller than the 70B case this piece budgets for. The same report runs RAG serving by day and LoRA fine-tuning by night on one unit specifically because both workloads draw on the same shared memory pool and are never run at the same time, which is the clearest evidence that DGX Spark is a single-tenant box, not a multi-tenant server, regardless of which model size you pick.
Where It Breaks: Index Size, Context Length, and Concurrent Users
Three things break the budget above in practice, and none of them show up until you push past the comfortable middle of the table.
Index size breaks first, and cleanly. The 10-million-vector FAISS index in the budgeting table above doesn't fit alongside a 70B LLM; it's over the remaining memory by roughly 26GB, not a tight squeeze. Past that point, the fix isn't tuning, it's architecture: shrink the generation model, shrink the index with product or binary quantization, or move to a disk-backed index like Milvus with DiskANN that doesn't need the whole thing resident in GPU memory at once. Our self-hosted vector database guide covers that path, along with the sharding Milvus and Qdrant need once a corpus outgrows what one GPU's memory can hold outright.
Context length breaks second, and quietly. KV cache scales with sequence length and with the number of retrieved chunks stuffed into the prompt, and the 5-million-vector row in the budgeting table already leaves under 10GB for it. A RAG pipeline that retrieves five chunks per query at a few thousand tokens each will burn through that headroom well before the vector index itself becomes the limiting factor, and the failure mode is a slowdown or an out-of-memory error rather than a clean error message pointing at the cause.
Concurrent users break third, and hardest, at 70B. Signal65's 4-to-8 concurrent-user ceiling applies to a 30B-class model at 39 tokens/sec aggregate, not to 70B at 2.7 tokens/sec single-stream. Adding a second concurrent user to a 70B deployment doesn't halve an already-fast number; it roughly halves an already-unusable one. DGX Spark's genuinely usable 70B RAG use case is single-developer or small-team, one query at a time. Multi-user serving on this box means stepping down to a 30B-class model, not pushing 70B harder.
Moving the Same Stack to Cloud GPUs When It Outgrows the Box
The handoff point is whichever of the three limits above you hit first: an index past roughly 5-10 million vectors at 1536 dimensions, a context length that eats the thin KV cache margin left after a large model and a large index, or more than one or two concurrent users on anything 70B-class. In every case, the fix is the same architecture, embedding model, vector index, and LLM colocated on one node, just on hardware with more memory and more bandwidth than DGX Spark's 273 GB/s.
A single H100 SXM5 brings 3.35 TB/s of memory bandwidth to the same decode step that limps at 2.7 tokens/sec on DGX Spark's 70B case, and multi-GPU H100 or H200 nodes add the NVLink tensor-parallel bandwidth that neither DGX Spark nor a single discrete card has at all. Our agentic RAG infrastructure guide runs the same embedding-plus-vector-search-plus-LLM colocation pattern and the same FAISS VRAM formula used in the budgeting table above, but against H100 and L40S cloud GPUs instead of one unified-memory box. Spheron rents H100 capacity with per-minute billing and no commitment, which fits a RAG workload that's outgrown DGX Spark for exactly as many hours as it actually needs, rather than requiring a second desk-side purchase.
On-demand H100 runs $3.96/hr and spot $2.20/hr as of 12 Nov 2026; H200, with its larger 141GB HBM3e pool for bigger indexes alongside the same model, runs $5.51/hr on-demand and $3.32/hr spot; B200 runs $10.17/hr on-demand and $5.22/hr spot. Compare those two tiers directly rather than assuming one is always cheaper; the real differentiator for an always-on RAG endpoint is reliability, not price. In our own inference fleet testing, every 8-GPU spot type we ran was reclaimed at least once on 14 Oct 2026, and on 16 Oct 2026 one B200 region stopped five of our spot VMs in a single day. If a RAG endpoint does run on spot, keep two copies of the model on separate nodes, spread across regions or providers, with automatic recovery set up, the same pattern a spot-first inference fleet needs regardless of workload. Spheron doesn't sell or rent the DGX Spark or any GB10 unit itself, so the local-hardware purchase decision stays a separate question from the hourly cloud rate; our DGX Spark vs cloud rental break-even hours breakdown runs that math directly against live H100, H200, and B200 rates.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 12 Nov 2026; other figures in this piece reflect their sources' most recently published data and may have changed since. Check current GPU pricing → for live rates.
A RAG stack that formula-fits on DGX Spark's 119.68GB pool is still worth measuring on your own corpus before you trust it with a 70B model. Once the index, the context length, or the concurrent users outgrow one box, Spheron's H100 nodes give the stack the bandwidth DGX Spark can't, billed by the minute.
Frequently Asked Questions
A full retrieval-augmented generation stack on one box: an embedding model for queries and documents, a GPU-resident vector index (FAISS, Qdrant, Milvus, or Weaviate), and an LLM for generation, all sharing DGX Spark's unified memory pool instead of talking over a network. NVIDIA's own multi-agent chatbot reference build bundles a RAG agent with coding and image-understanding agents on a single DGX Spark, using roughly 120GB of the 128GB pool by default.
It depends almost entirely on the LLM. DGX Spark reports roughly 119.68GB usable to the OS, not the advertised 128GB. A small stack, an int8 embedding model plus a FAISS index plus an 8B LLM, needs well under 20GB: Arm's GB10 RAG build measured DRAM climbing from about 3.5GiB idle to about 14GiB during query execution. Load a 70B LLM instead and the budget changes completely: at FP8 that model alone needs roughly 70GB, leaving about 49.68GB for the embedding model, the vector index, and KV cache combined.
Memory-wise, yes, with room for a multi-million-vector index alongside it. Speed-wise, the honest answer is no for anything resembling a live chat session. LMSYS measured Llama 3.1 70B in FP8 on DGX Spark via SGLang at 803 tokens/sec prefill but only 2.7 tokens/sec decode, well below conversational speed. Retrieval itself isn't the bottleneck; a FAISS-GPU search on DGX Spark has been measured at 96% GPU utilization on a 5-million-vector index. The LLM's decode speed is.
Using FAISS's deterministic VRAM formula (vectors times dimensions times 4 bytes, plus roughly 20% runtime overhead), a 70B FP8 LLM plus a small embedding model leaves about 46.68GB. A 5-million-vector index at 1536 dimensions needs roughly 31GB raw, about 37GB with overhead, which fits with a thin margin left for KV cache. A 10-million-vector index at the same dimension needs roughly 61GB raw, about 73GB with overhead, which doesn't fit in the remaining budget at all.
Yes, and it's an option neither an RTX 5090 nor a Mac Studio has, because neither pairs a CPU and GPU on one coherent memory pool the way DGX Spark's Grace-Blackwell superchip does. Arm demonstrated running query embedding (E5-base-v2, int8) on the Grace CPU cores in 70-90 milliseconds, while generation and FAISS vector search ran on the Blackwell GPU side, leaving the GPU's limited memory bandwidth free for the step that actually needs it.
