Mistral Medium 3.5 GPU Requirements: 4-GPU Setup (2026)

Mistral's model card lists Medium 3.5 as GA with a release date of April 28, 2026: a dense 128B-parameter transformer, 256K context window, under a Modified MIT license (Mistral model card). Mistral then introduced the model publicly on May 22 in an announcement that frames it as a public preview alongside Vibe Remote Agents, its new asynchronous cloud coding sessions (Mistral AI). Either way, the hardware framing is specific: Medium 3.5 "runs self-hosted on as few as four GPUs." That's the smallest cluster in Mistral's current flagship-to-mid lineup that still needs real multi-GPU tensor parallelism, one step up from Mistral Small 4, which gets by on two.

This guide covers Mistral Medium 3.5 GPU requirements in full: VRAM math for a dense 128B model, a head-to-head on 4x H100 vs 4x H200, and a complete vLLM deployment walkthrough with live Spheron pricing.

Mistral Medium 3.5 at a Glance

Medium 3.5 is Mistral's first "flagship merged model," combining instruction-following, reasoning, and coding into one checkpoint rather than shipping them as separate model lines (Mistral AI).

On benchmarks, Mistral reports 77.6% on SWE-Bench Verified, ahead of Devstral 2 and models like Qwen3.5, and 91.4 on τ³-Telecom, an agentic tool-use benchmark (Mistral AI). Medium 3.5 replaces Devstral 2 as the default model in Mistral's Vibe CLI and powers the new Vibe Remote Agents and a Le Chat "Work mode" for cross-tool workflows. If you're comparing it against the model it displaces, our Devstral deployment guide covers the smaller 24B coding model's own GPU footprint.

The dense architecture is the detail worth sitting with before you size a cluster. Every other model in Mistral's current self-hostable lineup, Large 3 and Small 4, is a Mixture of Experts model, where only a fraction of total parameters activate per token. Medium 3.5 breaks that pattern: all 128B parameters compute on every forward pass. Our MoE inference optimization guide covers what changes when you're planning around active parameters instead of total ones, which is the opposite of what this post walks through.

Mistral Medium 3.5 GPU Requirements: VRAM Math for a Dense 128B Model

The short version: at FP8, Medium 3.5's weights run about 128GB, and with runtime overhead the practical floor sits near 147GB before KV cache. That's why Mistral's four-GPU minimum lands on H100 or H200 class hardware rather than smaller cards. Dense models don't get an active-parameter discount the way Large 3 or Small 4 do; the full 128B has to be resident in VRAM regardless of how the router (or lack of one, in this case) works.

The formula is straightforward:

total_vram = (total_params x dtype_bytes) + (total_params x dtype_bytes x 0.15) + kv_cache_budget

A few notes on that table:

BF16 at ~294GB technically fits inside a 4x H100 node's 320GB, but it leaves only about 26GB for KV cache, which isn't enough room for meaningful context length or concurrency. Third-party self-hosting guides converge on the same numbers for BF16 and FP8 footprints, which is a useful sanity check against the formula above (aimadetools.com). Practically, BF16 belongs on 4x H200 (564GB), where the extra headroom supports the full 256K window comfortably.

FP8 is the config most teams should target. It's the native precision for Hopper and Blackwell tensor cores, and it's the quantization Mistral's own recipe assumes. On 4x H100 (320GB), FP8 weights plus overhead leave roughly 173GB for KV cache. On 4x H200 (564GB), that headroom grows to around 417GB, which is the difference between "supports 256K context at modest concurrency" and "supports 256K context with real batch depth."

Q4_K_M drops the floor low enough that a single H200 can technically hold the model, though at that point you're trading quality for GPU count and should benchmark the degradation against your own eval set before committing to production traffic on it. An independent guide puts INT4/GGUF weights at roughly the same ~64GB figure, which lines up with the formula above (LushBinary). For a general reference on how this math scales across model sizes, see the GPU memory requirements guide for LLMs.

The Minimum Viable Cluster: 4x H100 vs 4x H200

Mistral's four-GPU floor isn't a single fixed config: H100 and H200 both clear the FP8 VRAM requirement, but they land in different places on cost, context headroom, and throughput. Here's the comparison, with live per-GPU Spheron pricing as of 04 Aug 2026.

On-demand, 4x H200 costs about 25% more per hour than 4x H100, and spot narrows that gap to roughly 14%. What you get for the difference is KV cache headroom: 417GB versus 173GB is the gap between running the full 256K context window at meaningful concurrency and running it at just a handful of simultaneous sessions.

One independent production-config guide puts a real number on that trade-off: 4x H100 80GB at FP8 "delivers full 256K context, ~100 tokens/second throughput, and OpenAI-compatible API via vLLM" as its recommended sweet spot (Clore.ai). That's a reasonable default if your traffic is a handful of concurrent agentic sessions rather than a high-QPS chat API. If you're serving more concurrent users at long context, or you want NVLink bandwidth to reduce tensor-parallel communication overhead across 4 GPUs, 4x H200 is the safer floor. H100 GPU rental on Spheron and H200 capacity on Spheron both support 4-GPU bundles for either path.

Either config beats the alternative most teams default to out of habit: renting 8 GPUs "to be safe." Medium 3.5's own dense-model math doesn't reward that. Four H100s or four H200s already clear the FP8 floor with real KV cache room; an 8-GPU node buys you headroom this model doesn't need at double the hourly cost.

Pricing fluctuates based on GPU availability. The prices above are based on 04 Aug 2026 and may have changed. Check current GPU pricing → for live rates.

Step-by-Step: Deploy Mistral Medium 3.5 with vLLM

Prerequisites

• 4x H100 SXM5 80GB or 4x H200 SXM5 141GB GPU instance
• CUDA 12.1 or later, Python 3.10 or later
• ~150GB persistent storage for FP8 weights (or ~300GB if pulling BF16 first)
• Hugging Face account with access to the model repository

Provision a Spheron instance

Log in at app.spheron.ai, go to GPU Cloud, and select H100 SXM5 or H200 SXM5 with a GPU count of 4. Use a PyTorch 2.5 / CUDA 12.4 base image. Spot pricing works fine for development and evaluation; use on-demand for anything serving production traffic.

# Verify GPU setup after SSH access
nvidia-smi -L
nvidia-smi topo -m  # confirm NVLink topology

Install vLLM

pip install -U vllm

python -c "import vllm; print(vllm.__version__)"

Always install the latest release. Mistral's tokenizer and config formats need current vLLM support to load correctly.

Download the weights

huggingface-cli download mistralai/Mistral-Medium-3.5-Instruct-2604 \
  --local-dir /data/models/mistral-medium-3-5

du -sh /data/models/mistral-medium-3-5/

Verify the exact FP8 checkpoint size from the model card before you provision storage; download time depends on your instance's bandwidth.

Launch: 4x H100 FP8

vllm serve mistralai/Mistral-Medium-3.5-Instruct-2604 \
  --quantization fp8 \
  --tensor-parallel-size 4 \
  --tokenizer_mode mistral \
  --config_format mistral \
  --load_format mistral \
  --tool-call-parser mistral \
  --gpu-memory-utilization 0.90 \
  --max-model-len 65536 \
  --kv-cache-dtype fp8 \
  --enable-chunked-prefill \
  --port 8000

Flag notes:

• --tensor-parallel-size 4: splits every layer across the 4 GPUs. Requires NVLink or a fast PCIe topology for low-latency serving.
• --tokenizer_mode mistral --config_format mistral --load_format mistral: required for Mistral's native checkpoint format; omit these and the server won't load the model correctly.
• --tool-call-parser mistral: needed for structured tool-calling output, relevant if you're using Medium 3.5 for agentic workloads.
• --max-model-len 65536: a conservative starting context length on 4x H100's smaller KV cache budget. Raise it toward the full 262144 if your workload needs it and you're watching KV cache usage.
• --kv-cache-dtype fp8: halves KV cache memory, which matters more on H100's tighter headroom than on H200.

Launch: 4x H200 FP8

vllm serve mistralai/Mistral-Medium-3.5-Instruct-2604 \
  --quantization fp8 \
  --tensor-parallel-size 4 \
  --tokenizer_mode mistral \
  --config_format mistral \
  --load_format mistral \
  --tool-call-parser mistral \
  --gpu-memory-utilization 0.90 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --enable-chunked-prefill \
  --port 8000

With 417GB of KV cache headroom, 4x H200 can run the full 256K context window (--max-model-len 262144) at meaningfully higher concurrency than the H100 config. Drop --max-num-seqs if you see KV cache pressure at that context length under real load.

Test the API

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-Medium-3.5-Instruct-2604",
    "messages": [{"role": "user", "content": "Summarize the trade-offs between tensor and expert parallelism in two sentences."}],
    "max_tokens": 256
  }'

Or with the OpenAI Python client:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-required")

response = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-Instruct-2604",
    messages=[{"role": "user", "content": "Write a function that retries a flaky API call with exponential backoff."}],
    max_tokens=512,
)
print(response.choices[0].message.content)

Serving Medium 3.5 With vLLM or SGLang: Config and Throughput

vLLM is the more battle-tested path for Medium 3.5 today: native support for Mistral's tokenizer format, chunked prefill, and FP8 KV cache. SGLang is a legitimate alternative if you're already running it elsewhere in your stack, with RadixAttention giving it an edge on workloads with heavy prompt-prefix reuse, like repeated agentic tool-calling loops that share a long system prompt across turns. Our vLLM vs SGLang benchmarks cover the throughput and feature trade-offs between the two frameworks in more depth than fits here.

To serve with SGLang instead:

python -m sglang.launch_server \
  --model-path mistralai/Mistral-Medium-3.5-Instruct-2604 \
  --tp 4 \
  --tool-call-parser mistral \
  --reasoning-parser mistral \
  --context-length 65536 \
  --port 8000

--tp 4 sets tensor-parallel degree to match your GPU count, and the --tool-call-parser and --reasoning-parser flags mirror vLLM's Mistral-specific parsing for tool calls and reasoning output.

On throughput, one independently published production config reports roughly 80-140 tokens/second on 4x H100 at FP8, with sustained performance closer to 100 tok/s under typical load (Clore.ai). Treat that as a planning estimate, not a guarantee; your own throughput depends heavily on sequence length mix, batch composition, and how much of your traffic hits the deeper reasoning mode described below. Given the 256K context window, KV cache tuning matters more here than on smaller models; the KV cache optimization guide covers PagedAttention and FP8 KV cache tuning in more depth.

Reasoning Toggle: Instant vs Deep Mode for Agentic Workloads

Medium 3.5 supports a reasoning_effort parameter on chat completions, and unlike Small 4's three-tier low/medium/high dial, Medium 3.5 exposes a binary toggle: "none" for a fast, instant-style reply with the thinking chunk omitted, and "high" for a full reasoning trace before the final answer, at the cost of more output tokens. Mistral's own guidance is direct about when to reach for the deeper mode: for agentic and coding use cases, reasoning_effort="high" is the recommended setting (Mistral docs).

# Fast path: no thinking chunk, lower latency
response = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-Instruct-2604",
    messages=[{"role": "user", "content": "What does this HTTP 429 response mean?"}],
    extra_body={"reasoning_effort": "none"},
)

# Deep path: full reasoning trace, recommended for agentic and coding tasks
response = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-Instruct-2604",
    messages=[{"role": "user", "content": "Debug this failing test and propose a fix with reasoning."}],
    extra_body={"reasoning_effort": "high"},
)

For a chat assistant answering short factual questions, "none" keeps latency and token spend down. For Vibe Remote Agents-style long-horizon tool use, where the model plans multiple steps, calls tools, and checks its own output, "high" is what Mistral tuned the model against. If you're weighing Medium 3.5 against Mistral's dedicated reasoning line instead of its configurable toggle, the Magistral deployment guide covers Mistral's always-on chain-of-thought model, which trades the per-request switch for reasoning that's on by default.

Mistral Medium 3.5 vs Mistral Large 3: Which to Deploy on GPU Cloud

Both are current-generation Mistral models built for coding and agentic work, but they sit at very different points on the hardware curve.

Medium 3.5 is the lighter self-hosting commitment by a wide margin. Its dense 128B footprint fits comfortably on a 4-GPU node that costs well under half of Large 3's minimum FP8 config. Large 3's 675B total parameter budget, even with only 41B active per token, still has to sit fully resident in VRAM, which is why it needs 4x B200 or 8x H200 at minimum. See the full Mistral Large 3 deployment guide for that model's VRAM math and expert-parallelism setup.

Choose Medium 3.5 when a 4-GPU cluster is the ceiling of what you want to commit to, when Modified MIT licensing fits your deployment terms, or when your workload is squarely agentic coding and tool use, the exact profile Mistral built the model around. Choose Large 3 when you have the budget for an 8-GPU-class cluster and want the larger total parameter count's headroom on tasks outside the coding-agent lane, or when Apache 2.0's fewer restrictions matter more than Modified MIT's for your use case.

Mistral Medium 3.5 vs Mistral Small 4: Choosing Your Self-Host Tier

The comparison against Small 4 cuts the other direction: Medium 3.5 needs more GPUs, not fewer, and the architecture difference is what drives it.

Small 4's MoE routing keeps its active-parameter compute closer to a 6-8B model even though 119B total parameters have to stay resident in memory, which is why it can run on as few as 2x H200. Medium 3.5's dense architecture means there's no active-parameter shortcut: every one of its 128B parameters computes on every token, and that's reflected directly in the four-GPU floor. Full setup detail for the smaller tier is in the Mistral Small 4 deployment guide.

The practical decision usually comes down to workload shape rather than raw benchmark scores. If you need one model that competently covers instruction-following, vision, and code across a broad range of tasks at the lowest GPU cost, Small 4 on 2x H200 is hard to beat. If your workload is specifically long-horizon agentic coding, where SWE-Bench and tool-use performance matter more than breadth, Medium 3.5's extra GPU is buying real capability Mistral built and tuned the model for.

Spheron GPU Pricing for Mistral Medium 3.5

Prices below are derived from the Spheron GPU pricing API as of 04 Aug 2026. On-demand figures use the lowest available per-GPU rate across non-spot offers; spot figures use the lowest per-GPU spot rate.

Spot instances cut the hourly rate by roughly 27-33% at these configs. They're a reasonable fit for development, evaluation, and batch agentic runs that can tolerate preemption. On-demand is the safer choice for anything serving live traffic, particularly Vibe Remote Agents-style sessions that run for extended periods without a natural checkpoint.

If you're weighing self-hosting against Mistral's own API instead of committing to a cluster, our Mistral API pricing vs self-hosted breakdown walks through the crossover math at $1.50/$7.50 per million tokens, including where Medium 3.5 fits now that it has an open-weight self-hosting path.

Pricing fluctuates based on GPU availability. The prices above are based on 04 Aug 2026 and may have changed. Check current GPU pricing → for live rates.

Medium 3.5 is the smallest cluster commitment in Mistral's current flagship-to-mid lineup, and both 4x H100 and 4x H200 are available on-demand or as spot on Spheron.

Deploy Mistral Medium 3.5 on Spheron →