Deploy CrewAI on GPU Cloud: Production Multi-Agent Workflows with Self-Hosted LLM Inference (2026 Guide)

CrewAI prototypes run fine against the OpenAI API. Production crews with 5-20 agents hit latency walls fast, and the per-token cost compounds with every agent in the pipeline. A 5-agent sequential crew makes 5 serial LLM calls per run; at ~$0.05 per execution on GPT-4o, 10,000 daily runs becomes a ~$500 daily API bill before you've added any tooling. Self-hosting the inference backend on a GPU changes the math.

CrewAI in the Multi-Agent Landscape

Three frameworks dominate production multi-agent deployments: CrewAI, LangGraph, and AutoGen. They solve different problems.

CrewAI is the fastest path to a working multi-agent pipeline. You define roles, goals, and backstories in plain text; the framework handles routing, task delegation, and result aggregation. The tradeoff is less control over execution state compared to LangGraph. If your use case needs conditional branching based on mid-run results, checkpoint-based resumability on spot instances, or explicit human interruption points, the LangGraph vs LangChain comparison walks through when the graph model pays off.

For straightforward workflows like research-synthesis, document processing, or multi-step analysis pipelines with defined agent specializations, CrewAI wins on simplicity.

Why the LLM Backend Becomes the Bottleneck in Production

The fanout math is simple and brutal.

A 5-agent sequential crew makes 5 LLM calls per run. Each call takes 2-6 seconds depending on output length and model size. Total latency per run: 10-30 seconds minimum. At 100 concurrent users, that is 500 simultaneous LLM calls queuing against a shared API endpoint with rate limits.

In hierarchical process mode, the manager LLM runs concurrently with worker agents, so multiple requests hit the backend at the same time. A crew with a manager plus 4 workers can generate 5 concurrent LLM calls in a single crew execution cycle.

KV-cache pressure compounds the problem. When multiple agents share a single GPU-resident model endpoint, each concurrent request occupies VRAM for its context window. A 4096-token context at Qwen3-32B FP8 needs roughly 1.5GB of KV-cache per slot. With 10 concurrent agents, that is 15GB consumed by cache alone, on top of the ~35GB needed for the model weights — roughly 50GB total, which exceeds the L40S (48GB). Running 10 concurrent Qwen3-32B agents requires an A100 80GB or H100 80GB; a 24GB consumer card does not even fit the model weights. vLLM's continuous batching and PagedAttention manage how this cache memory is allocated across concurrent requests, enabling 2-4x more concurrent agents than naive static batching.

The API endpoint alternative hides these mechanics behind a rate limit error. You hit 429s and add exponential backoff. Token throughput stays capped. Self-hosting puts you in control of the concurrency ceiling.

Production Architecture

The production stack has four layers. Each one can run on separate machines or co-locate depending on budget.

For most teams, Layers 1 and 3-4 run on a small CPU instance ($20-50/month). Layer 2 (the GPU) is the only costly piece, and it is also the only piece that matters for throughput.

GPU Sizing Guide

VRAM math for a crew depends on three numbers: model weight size, KV-cache per concurrent agent, and buffer headroom.

For FP8 quantization (vLLM default for supported models), model sizes are:

• 8B model: ~9GB
• 14B model: ~15GB
• 32B model: ~35GB
• 70B model: ~75GB

KV-cache per concurrent request at 4096-token context:

• 8B model: ~0.5GB per slot
• 32B model: ~1.5GB per slot
• 70B model: ~3.5GB per slot

L40S instances on Spheron fit the largest category of real CrewAI deployments: 3-10 agent crews with 7-32B models in sequential mode. The 48GB at FP8 can run Qwen3-32B with headroom for 2-3 concurrent agent requests before KV-cache pressure degrades throughput.

For crews that need 32B models in hierarchical mode or anything larger, A100 on Spheron with its 80GB provides the buffer. Plan for no more than 85% VRAM utilization in production to leave room for cache growth during peak concurrency.

Step-by-Step: Deploying vLLM on Spheron for CrewAI

1. Provision the GPU instance

Log into app.spheron.ai. For most CrewAI deployments with models up to 32B, pick the L40S (48GB). For larger models or high-concurrency hierarchical crews, pick A100 80GB. Choose on-demand for a persistent production service; use spot only for batch or offline crew runs.

Once the instance is up, SSH in and verify the GPU:

nvidia-smi

2. Install Docker with the NVIDIA container toolkit

apt-get update
apt-get install -y docker.io nvidia-container-toolkit
systemctl restart docker

Verify that Docker can see the GPU:

docker run --gpus all --rm nvidia/cuda:12.1-base-ubuntu22.04 nvidia-smi

3. Deploy vLLM

For a 32B model on an L40S:

docker run --gpus all --ipc=host \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-32B \
  --dtype fp8 \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 64 \
  --max-model-len 8192

Key flags:

• --max-num-seqs 64: maximum concurrent requests (concurrent agents). Tune down if you see OOM errors.
• --gpu-memory-utilization 0.85: leaves 15% of VRAM free for spikes.
• --max-model-len 8192: cap context length to control KV-cache size.

For a 70B model on two H100s:

docker run --gpus all --ipc=host \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-72B \
  --dtype fp8 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 32

Verify the endpoint is up:

curl http://localhost:8000/v1/models

4. Configure CrewAI to use the vLLM endpoint

Install CrewAI (pin to 0.80 or later for the stable Process enum and LLM class):

pip install "crewai>=0.80" crewai-tools

Option A: environment variables (applies globally to all agents)

import os
os.environ["OPENAI_API_BASE"] = "http://YOUR_SPHERON_IP:8000/v1"
os.environ["OPENAI_API_KEY"] = "sk-placeholder"
os.environ["OPENAI_MODEL_NAME"] = "Qwen/Qwen3-32B"

Option B: per-agent LLM config (allows different models per agent)

from crewai import LLM

local_llm = LLM(
    model="openai/Qwen/Qwen3-32B",  # note the openai/ prefix for LiteLLM routing
    base_url="http://YOUR_SPHERON_IP:8000/v1",
    api_key="sk-placeholder",
)

Note the difference: the environment variable form uses the bare model name (Qwen/Qwen3-32B); the LLM constructor form requires the openai/ prefix (openai/Qwen/Qwen3-32B). This is a LiteLLM routing requirement, not a CrewAI quirk.

5. A working 3-agent crew example

import os
from crewai import Agent, Task, Crew, Process, LLM

# Point CrewAI at the self-hosted vLLM endpoint
local_llm = LLM(
    model="openai/Qwen/Qwen3-32B",
    base_url="http://YOUR_SPHERON_IP:8000/v1",
    api_key="sk-placeholder",
    max_tokens=2048,
    temperature=0.7,
)

# Define agents with specific roles
researcher = Agent(
    role="Research Analyst",
    goal="Find accurate, current information on the given topic",
    backstory="You specialize in finding and summarizing technical information.",
    llm=local_llm,
    verbose=True,
    max_iter=3,
)

writer = Agent(
    role="Technical Writer",
    goal="Produce clear, accurate technical content based on research findings",
    backstory="You turn complex technical information into readable content.",
    llm=local_llm,
    verbose=True,
    max_iter=3,
)

reviewer = Agent(
    role="Quality Reviewer",
    goal="Ensure accuracy, clarity, and completeness of the final output",
    backstory="You catch errors, gaps, and unclear explanations before publishing.",
    llm=local_llm,
    verbose=True,
    max_iter=2,
)

# Define tasks
research_task = Task(
    description="Research the topic: {topic}. Focus on recent developments and key facts.",
    expected_output="A structured summary with key findings, sources, and gaps.",
    agent=researcher,
    max_execution_time=120,  # seconds
)

writing_task = Task(
    description="Write a 500-word technical summary based on the research findings.",
    expected_output="A 500-word technical summary ready for publication.",
    agent=writer,
    context=[research_task],
    max_execution_time=120,
)

review_task = Task(
    description="Review the summary for accuracy, clarity, and completeness.",
    expected_output="The reviewed summary with any corrections noted.",
    agent=reviewer,
    context=[writing_task],
    max_execution_time=60,
)

# Assemble the crew
crew = Crew(
    agents=[researcher, writer, reviewer],
    tasks=[research_task, writing_task, review_task],
    process=Process.sequential,
    verbose=True,
)

result = crew.kickoff(inputs={"topic": "vLLM continuous batching"})
print(result)

Run this against a single agent first (Crew(agents=[researcher], tasks=[research_task])) to confirm the endpoint connection before adding the full pipeline.

Memory and Persistence: Hooking CrewAI to Mem0 and Zep

CrewAI's built-in memory=True flag uses ChromaDB in a local directory for short-term, entity, and long-term memory within a single session. It is fast to set up and works well for single-session workflows.

crew = Crew(
    agents=[researcher, writer, reviewer],
    tasks=[research_task, writing_task, review_task],
    process=Process.sequential,
    memory=True,  # enables built-in ChromaDB-backed memory
)

For cross-session persistence, the built-in memory resets on process restart. To retain knowledge across crew runs, you need an external backend. The agent memory infrastructure guide covers Mem0, Zep, and Letta deployment on Spheron GPUs, including how to run the embedding server and vector store alongside the inference backend.

The pattern for CrewAI specifically: deploy a Mem0 server pointing at your vLLM endpoint for LLM-based memory extraction, and your TEI embedding server for vector storage. Then create a custom CrewAI tool that calls the Mem0 API:

from crewai_tools import tool
import requests

@tool("memory_search")
def memory_search(query: str) -> str:
    """Search long-term memory for relevant past information."""
    response = requests.post(
        "http://your-mem0-server:8000/v1/memories/search",
        json={"query": query, "user_id": "crew_default", "limit": 5},
    )
    response.raise_for_status()
    memories = response.json().get("results", [])
    return "\n".join(m.get("memory", "") for m in memories)

Attach the tool to the agents that need historical context (typically the researcher and reviewer). The writer rarely needs past memories.

Observability for Multi-Agent Execution

Multi-agent traces are harder to debug than single-agent ones because failures can be buried in the middle of a pipeline. Langfuse is the cleanest option for CrewAI because LiteLLM has native Langfuse instrumentation: set three environment variables and all LLM calls are traced automatically.

export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_HOST=https://cloud.langfuse.com  # or your self-hosted instance

With those variables set, every crew execution generates a trace with one span per agent LLM call. For a 5-agent crew, you see 5 spans (or more if agents retry):

crew_kickoff [total: 47.2s]
  ├─ researcher [12.3s, 1,847 tokens, $0.00 at self-hosted]
  ├─ writer     [18.9s, 2,203 tokens, $0.00 at self-hosted]
  └─ reviewer   [16.0s, 1,102 tokens, $0.00 at self-hosted]

The per-span token counts tell you which agent is generating the most output. If the writer agent is at 4x the tokens of the researcher, your writing task's expected_output instructions are too loose. Tighten them or reduce max_tokens on the writer's LLM config.

Arize Phoenix is a good alternative if you prefer a fully self-hosted trace backend. It supports OpenTelemetry and has a LiteLLM exporter.

Reliability Patterns

Timeout handling

Set max_execution_time on each Task to prevent runaway agents from blocking the pipeline:

research_task = Task(
    description="Research the topic: {topic}",
    expected_output="A structured summary.",
    agent=researcher,
    max_execution_time=120,  # 2 minutes per task
)

If the task times out, CrewAI raises a TaskExecutionError. Catch it at the crew level and decide whether to retry or return a partial result.

Agent retries

Set max_iter on each Agent to control how many times an agent can attempt a task before the framework gives up:

researcher = Agent(
    role="Research Analyst",
    goal="Find accurate information",
    backstory="...",
    llm=local_llm,
    max_iter=3,  # will retry up to 3 times on output format failures
)

Higher max_iter costs more tokens. For agents with tool calls that are likely to succeed on first try, set max_iter=2. For agents with complex output format requirements, max_iter=3-4 is reasonable.

Partial failure recovery with hierarchical process

In hierarchical mode, the manager LLM reviews each worker agent's output before passing it downstream. If a worker fails to meet the task criteria, the manager can re-delegate to the same agent with additional context.

crew = Crew(
    agents=[researcher, writer, reviewer],
    tasks=[research_task, writing_task, review_task],
    process=Process.hierarchical,
    manager_llm=local_llm,
    verbose=True,
)

The manager uses additional LLM calls per task (one to delegate, one to validate output), so budget 1.5-2x the LLM calls of sequential mode. The benefit is better output quality and automatic handling of agent failures without explicit error handling code.

Cost Economics: Self-Hosted vs API

Using live Spheron pricing and GPT-4o's published rates (input: $2.50/1M tokens, output: $10.00/1M tokens as of early 2026), here is the cost breakdown for a 5-agent crew that processes 2,000 input tokens and generates 500 output tokens per agent per run:

The GPU server runs 24 hours regardless of crew run count, so the self-hosted cost is flat per day. The API cost scales linearly with usage. At 500 runs/day, the L40S on-demand at $17.28/day beats GPT-4o at ~$25/day — the crossover point where self-hosting becomes cheaper. H100 on Spheron at $92.16/day breaks even around 2,000 runs/day where API costs hit ~$100. For 32B-class models at high throughput, the L40S is the stronger value pick; H100 80GB earns its cost for models above 40B or crews that need headroom for larger KV-cache.

H100 SXM5 spot pricing (around $1.69/hr vs $3.84/hr on-demand) cuts the crossover point substantially for H100 workloads. Note that L40S spot can run above on-demand during peak demand periods, so check live rates before assuming spot is cheaper. A single vLLM server can also handle crews from multiple services simultaneously, spreading the fixed GPU cost across more usage.

Pricing fluctuates based on GPU availability. The prices above are based on 26 May 2026 and may have changed. Check current GPU pricing → for live rates.

For teams running larger agent fleets where multiple crews compete for the same GPU capacity, see the scaling AI agent fleets guide for GPU tiering strategies across orchestrator and worker layers.

Reference Architecture: CrewAI + vLLM + Mem0 on Spheron

This is the complete working configuration for a production 3-agent crew with persistent memory:

import os
import requests
from crewai import Agent, Task, Crew, Process, LLM
from crewai_tools import tool

# vLLM endpoint on Spheron GPU
LOCAL_LLM = LLM(
    model="openai/Qwen/Qwen3-32B",
    base_url="http://YOUR_SPHERON_IP:8000/v1",
    api_key="sk-placeholder",
    max_tokens=2048,
    temperature=0.7,
)

MEM0_URL = "http://YOUR_MEM0_SERVER:8000"

@tool("memory_search")
def memory_search(query: str) -> str:
    """Search long-term memory for context relevant to the query."""
    response = requests.post(
        f"{MEM0_URL}/v1/memories/search",
        json={"query": query, "user_id": "crew_shared", "limit": 5},
    )
    response.raise_for_status()
    results = response.json().get("results", [])
    if not results:
        return "No relevant memories found."
    return "\n".join(f"- {m.get('memory', '')}" for m in results)

@tool("memory_add")
def memory_add(content: str) -> str:
    """Store a new memory for future crew runs."""
    response = requests.post(
        f"{MEM0_URL}/v1/memories/",
        json={"messages": [{"role": "user", "content": content}], "user_id": "crew_shared"},
    )
    response.raise_for_status()
    return "Memory stored."

researcher = Agent(
    role="Research Analyst",
    goal="Find accurate, current technical information",
    backstory="You focus on facts, primary sources, and recent developments.",
    llm=LOCAL_LLM,
    tools=[memory_search, memory_add],
    max_iter=3,
)

writer = Agent(
    role="Technical Writer",
    goal="Turn research into clear, accurate technical content",
    backstory="You write for engineers who need precise, concise explanations.",
    llm=LOCAL_LLM,
    max_iter=3,
)

reviewer = Agent(
    role="Quality Reviewer",
    goal="Catch errors, gaps, and unclear sections before output is finalized",
    backstory="You verify technical accuracy and flag anything that needs more evidence.",
    llm=LOCAL_LLM,
    tools=[memory_search],
    max_iter=2,
)

research_task = Task(
    description="Research: {topic}. Search memory first for prior findings.",
    expected_output="Structured summary: key facts, sources, gaps. Max 400 words.",
    agent=researcher,
    max_execution_time=120,
)

writing_task = Task(
    description="Write a 500-word technical explanation of the research findings.",
    expected_output="500-word technical summary. No marketing language.",
    agent=writer,
    context=[research_task],
    max_execution_time=120,
)

review_task = Task(
    description="Review the summary. Flag any factual errors or unclear sections.",
    expected_output="Final summary with review notes. Mark corrections inline.",
    agent=reviewer,
    context=[writing_task],
    max_execution_time=60,
)

crew = Crew(
    agents=[researcher, writer, reviewer],
    tasks=[research_task, writing_task, review_task],
    process=Process.sequential,
    verbose=True,
)

result = crew.kickoff(inputs={"topic": "vLLM PagedAttention and KV-cache management"})
print(result)

The same crew pattern adapts well to a self-hosted AI coding assistant workflow - swap the researcher for a code-reading agent and the writer for a code-generation agent.

CrewAI's production ceiling is set by the LLM backend it calls. Self-hosting vLLM on a bare-metal GPU gives consistent throughput across all concurrent agents without per-token API fees stacking up with every crew run.

Rent L40S on Spheron → | Rent H100 on Spheron → | View all GPU pricing →