AI agent infrastructure is not one sizing problem, it's five. The workflow orchestrator, the LLM backend, the embedding model, the vector database, and the memory store each have different hardware needs, and treating the whole thing as "how big a GPU does my model need" leads teams to rent H100s for components that would run fine on a $0.20/hr CPU node.
That mistake is common because most GPU sizing guides start and end with the LLM. This one starts with the full stack: what actually touches a GPU, what a realistic VRAM budget looks like across the components that do, and a priced reference architecture you can rent today. If you want the deeper math on latency budgets and per-pattern GPU selection for the LLM layer specifically, read how to build GPU infrastructure for AI agents first; this post assumes that foundation and focuses on composing the rest of the stack around it.
About a third of organizations building AI agents are already self-hosting their own models rather than calling a hosted API, according to LangChain's 2026 State of Agent Engineering report, which surveyed more than 1,300 practitioners. The report attributes that choice to high-volume cost optimization, data residency requirements, and regulatory constraints in sensitive industries, not infrastructure cost alone. Worth noting: the same report found cost is cited less often as a production blocker than in prior years, since falling model prices have shifted attention away from raw spend. Self-hosting today is more often a control decision than a pure cost play.
What Actually Needs a GPU in AI Agent Infrastructure (and What Runs Fine on CPU)
A typical production agent stack has five layers, and only two of them need a GPU.
| Layer | Example tools | Needs a GPU? |
|---|---|---|
| Workflow orchestrator | n8n, LangGraph, CrewAI, Temporal | No, CPU only |
| LLM inference backend | vLLM, Ollama, TGI serving your model | Yes, GPU-resident |
| Embedding model | BGE-M3, Qwen3-Embedding, TEI-served models | Yes, but far smaller than the LLM |
| Vector database (index + search) | Qdrant, Weaviate, Milvus | No, CPU only (FAISS is the exception) |
| Memory store | Mem0, Zep, Redis-backed session state | No, CPU only |
Orchestration is a CPU cost, full stop. n8n's own deployment docs say the platform "isn't CPU intensive" and that even small cloud instances are enough for most use cases, with memory, not CPU or GPU, as the actual constraining resource. Their documented minimum for a self-hosted instance is 2 CPU cores, 2GB RAM, and 20GB SSD storage, with no GPU requirement at any tier. LangGraph, CrewAI, and Temporal workflows follow the same pattern: they call your inference endpoint over HTTP and do control-flow logic locally. None of that needs GPU compute. For a deeper look at right-sizing the CPU side once tool calls and orchestration steps pile up, see the CPU-to-GPU ratio guide for agentic inference.
Vector search runs on CPU for the databases most teams actually use. Qdrant's own documentation describes the engine as relying "primarily on CPU acceleration for scalability and efficiency," with GPU support offered as an optional accelerator for index building, not a requirement for serving queries. Weaviate and Milvus follow the same CPU-first design for their index and search paths. FAISS is the outlier: it's the one popular open-source vector library built for native GPU-accelerated search. If your stack uses Qdrant, Weaviate, or Milvus, the GPU in that part of your architecture is only there for the embedding step upstream, not the database itself. Our self-hosted vector database guide covers CPU sizing for all three at production scale.
Two layers do need GPU-resident state: the LLM backend, because the model weights and KV cache have to live in VRAM for the agent loop to respond in real time, and the embedding model, because encoding text into vectors is a forward pass through a neural network. Both benefit from GPU throughput, but their VRAM footprints are wildly different in scale, which is the next thing to get right.
Sizing GPU Memory for Local-Model Agent Backends
The short answer: budget the LLM's weights and KV cache first, since that's 70-90% of your total VRAM, then add the embedding model on top if you're colocating it, plus 10-15% runtime overhead. A 7B-13B agent backend with a small embedding model typically fits in 30-40GB total, which rules out consumer 24GB cards but comfortably fits a single 48GB or 80GB rental.
The VRAM Formula: Weights + KV Cache + Embeddings + Overhead
The GPU infrastructure for AI agents guide linked above sizes the LLM backend on its own: weights plus KV cache plus overhead. That's correct for a single-model deployment, but it undercounts a real agent stack, because the embedding model sitting next to your LLM has its own VRAM line item. Add it in explicitly instead of folding it into a generic overhead buffer:
Total VRAM = LLM weights + (KV cache per session × concurrent sessions) + embedding model + overheadLLM weights in FP16 (add 15-30% for serving overhead separately):
- 7B model: ~14GB
- 13B model: ~26GB
- 70B model: ~140GB
KV cache scales with context length and concurrency. A modern GQA-based 7B model at 4K context uses roughly 0.5GB of KV cache per session in FP16; at 32K context that grows to around 4GB per session. This is the number that blows up VRAM budgets on long-context, multi-turn agent conversations, so size it for your real conversation length, not a single-turn estimate.
Embedding model weights are the smallest line item, and it's worth pricing them separately rather than lumping them into "the model." Qwen3-Embedding-8B needs roughly 8-10GB of VRAM at Q8 quantization, small next to a 13B LLM but not negligible if you're trying to fit everything on one 24GB card. BGE-M3, at 0.6B parameters, fits comfortably on any GPU with 8GB or more of VRAM. Our self-hosted embeddings and rerankers guide covers TEI deployment and quantization tradeoffs for both.
Worked Example: A 3-Component Agent Stack on One GPU
Take a customer support agent: a 13B LLM backend, BGE-M3 for embeddings, and a moderate concurrency target of 20 sessions at 8K context.
| Component | VRAM |
|---|---|
| 13B LLM weights (FP16) | ~26GB |
| KV cache, 20 sessions @ 8K context (~1.25GB/session) | ~25GB |
| BGE-M3 embedding model | ~2GB |
| Runtime overhead (vLLM, CUDA context, ~10%) | ~5GB |
| Total | ~58GB |
That total rules out a 48GB card and points to an 80GB GPU with headroom, since you don't want to run production at 95%+ VRAM utilization. On Spheron, an A100 80GB SXM4 currently runs from $1.31/hr on-demand, enough to colocate the LLM and embedding model on one GPU and leave the vector database and orchestrator on CPU nodes entirely. Drop concurrency or context length and a single H100 SXM5 or even a 48GB card gets you there for less; the formula above is what you re-run for your own numbers, not a fixed answer.
Pricing fluctuates based on GPU availability. The prices above are based on 19 Aug 2026 and may have changed. Check current GPU pricing → for live rates.
Reference Architecture: Self-Hosted AI Agent Infrastructure on Rented GPU Cloud
Here's how the five layers map onto actual rented infrastructure, split by what needs a GPU and what doesn't.
Orchestration Layer (CPU) - Workflow Engine or Agent Framework
Run n8n, LangGraph, or a custom agent runtime on a CPU-only instance. Spheron's pricing data currently shows a CPU node from about $0.20/hr on-demand, which comfortably covers n8n's documented minimum of 2 CPU cores and 2GB RAM with room to scale for production traffic. This layer talks to your inference endpoint over HTTP; it has no business consuming GPU budget. If your agents call GPU-backed MCP tool servers, the GPU-accelerated MCP server deployment guide covers keeping that boundary clean between the CPU-side orchestrator and any GPU-backed tools it calls.
Inference Layer (GPU) - vLLM/Ollama Serving the LLM
This is where your VRAM budget from the formula above lands. Provision a dedicated GPU instance running vLLM with continuous batching for production traffic, or Ollama for lighter dev/test workloads. Set --max-num-seqs based on your target concurrent sessions; it's the primary lever for VRAM utilization. Keep this instance on-demand, not spot, if a live user is waiting on the response: spot interruption mid-conversation is a broken session, not a retry.
Memory and Retrieval Layer (Mixed) - Vector DB + Embedding Model
Split this layer in two. The embedding model needs GPU time for each new document or query it encodes, and if VRAM allows, colocate it on the same GPU as your LLM to avoid a second GPU rental entirely, per the worked example above. The vector database itself, Qdrant, Weaviate, or Milvus, runs its index and search on a separate CPU instance sized for your dataset's RAM footprint, not GPU count. Persistent agent memory (Mem0, Zep) sits on the same CPU tier as the vector database; our agent memory infrastructure guide covers sizing that store independently from retrieval.
Cost of Self-Hosting Agents vs Calling Hosted LLM APIs
Self-hosting wins on unit economics once call volume is high enough to keep the GPU busy; it loses when the GPU sits mostly idle. There's no universal crossover point, it depends entirely on your actual traffic, so run the comparison on your own numbers rather than a rule of thumb from a vendor blog.
The variables that matter: calls per day, average tokens per call, and how much of the day the GPU is actually doing inference versus waiting for the next request. A rented GPU costs the same per hour whether it's serving 5 requests or 500. Hosted APIs invert that: cost scales linearly with usage but never approaches zero for idle capacity, because there's no idle capacity to pay for.
There's a second variable that the cost math alone misses. MIT's 2025 "State of AI in Business" report found that 95% of enterprise generative AI pilots failed to show measurable P&L return, despite $30-40 billion in enterprise investment. That's a strong argument for teams wanting direct visibility into what their AI spend is actually buying, rather than treating inference as an opaque per-token line item on a vendor invoice. Self-hosting doesn't fix a bad agent design, but it does make the cost structure legible in a way a metered API bill often isn't.
Before sizing hardware, run your own numbers through our AI agent cost calculator, which turns calls-per-day, tokens-per-call, concurrency, and model size into an estimated GPU bill. For a head-to-head on cost, latency, and data control specifically, see GPT-6 API vs self-hosted LLMs.
Scaling Considerations as Agent Concurrency Grows
The reference architecture above holds for a single-GPU, low-to-moderate concurrency deployment. Three things change as concurrency grows.
KV cache becomes the binding constraint before compute does. A production benchmark of 100 concurrent agent sessions on bare-metal GPUs found KV cache exhaustion, not CPU or raw GPU saturation, was the limiting factor at scale, per Spheron's 100 concurrent AI agents case study. Plan your VRAM budget with that ceiling in mind, not just peak throughput.
Multi-agent pipelines add coordination overhead that single-agent sizing doesn't capture. Once you're running an orchestrator plus multiple sub-agents, each with its own tool calls, you're managing a different scaling problem than a single LLM endpoint; the multi-agent AI infrastructure guide covers topology and cluster design at that stage.
Fleet-level autoscaling needs its own cost model. Past a handful of GPUs, manually provisioning instances per traffic spike stops working. Scale AI agent fleets on GPU cloud covers autoscaling patterns and cost modeling for larger deployments. If you're choosing between providers at this stage, our best GPU cloud for AI agents comparison breaks down cost and latency across the field.
LangChain's report puts a number on the adoption curve behind all of this: 57.3% of surveyed organizations now have agents in production, rising to 67% among enterprises with 10,000+ employees versus 50% for organizations under 100 people. The report's authors put it plainly: "Few teams are betting on a single provider." Over three-quarters of respondents route across multiple models in production or development based on cost, latency, and task complexity, which is the same multi-provider logic that applies to infrastructure. Spheron aggregates GPU capacity from 5+ providers on the backend, so scaling a fleet doesn't mean betting your availability on a single data center's capacity.
Getting Started: A Self-Hosted Agent Stack Checklist
- Map your stack to the GPU/CPU split from the first section. Confirm which of your five layers are GPU-bound before pricing anything.
- Run the VRAM formula for your LLM and embedding model: weights + (KV cache × concurrent sessions) + embedding model + 10-15% overhead.
- Provision the GPU layer separately from the CPU layer. Rent an H100 for inference and embeddings, and a CPU node for orchestration, vector search, and memory.
- Start on-demand for anything user-facing. Move background or batch tasks to spot only once you've confirmed they can checkpoint and resume.
- Profile before you scale. Watch VRAM utilization and KV cache growth under real traffic before committing to a bigger instance or a second GPU.
Check the Spheron docs for provisioning steps once you've settled on an instance size.
A self-hosted agent stack is only as efficient as its weakest sizing decision, and the weakest one is usually putting a GPU under a component that never needed one. Spheron gives you GPU rental by the component that actually needs it, from A100s for embedding and inference to CPU-only nodes for everything else.
Frequently Asked Questions
No. The orchestrator (LangGraph, CrewAI, n8n), the vector database's search path (Qdrant, Weaviate, Milvus), and the memory layer (Mem0, Zep) all run on CPU. Only the LLM backend and the embedding model need a GPU. n8n's own docs say it isn't CPU-intensive and even small instances handle most workloads, with memory as the real constraint, not compute.
For a 7B-13B LLM plus a small embedding model at moderate concurrency, budget 30-40GB of VRAM total: roughly 14-26GB for the LLM's weights, 4-10GB for the embedding model, several GB for KV cache, and 10-15% overhead. That fits on a single 48GB or 80GB GPU. Scale the LLM's share up first since it dominates the total.
It depends on volume and utilization. Self-hosting wins at sustained high call volume because you pay a flat hourly GPU rate instead of per-token fees. It loses if the GPU sits idle most of the day. Run the math on your actual calls-per-day and average tokens-per-call before committing hardware, and don't discount the API route purely on infrastructure cost: LangChain's 2026 survey found infrastructure cost is now cited less often as a production blocker than in prior years, with falling model prices doing the work.
Only the embedding step needs the GPU. Qdrant, Weaviate, and Milvus run their index and search on CPU; Qdrant additionally offers optional GPU-accelerated index building, but that's not the same as needing a GPU to serve queries. Colocate the embedding model on the LLM's GPU if you have VRAM headroom, and let the vector database itself run on a CPU node or a CPU-only container next to it.
