Run Qwen3 Locally: GPU Requirements, Setup & Costs (2026)

Qwen3's smaller sizes run comfortably on a single consumer GPU, which is why "run Qwen3 locally" is one of the most searched setup questions since Alibaba shipped the family in April 2025. This guide sizes every open-weight Qwen3 variant by VRAM, walks through the Ollama commands to get one running in minutes, and covers the point where a local box stops being enough and vLLM on a rented GPU takes over. If you're comparing model choice more broadly first, our VRAM tier guide for self-hosting open-source LLMs places Qwen3 alongside Phi-4, Ministral, and GLM at each GPU tier.

Qwen3 ships as six dense models (0.6B, 1.7B, 4B, 8B, 14B, 32B) and two mixture-of-experts models (30B-A3B, 235B-A22B), all under an Apache 2.0 license, all supporting up to 128K context, and all with a hybrid "thinking" mode you toggle per request. Nathan Lambert, an AI researcher at Interconnects, called the release "the best of both worlds for open models, peak performance and size scales," and noted it's "a major validation of the DeepSeek R1 recipe and distillation in general." That size range is exactly what makes the lineup worth sizing carefully before you pick one.

Where You Can Run Qwen3 Locally: Model Lineup by GPU

Every Qwen3 size from 0.6B to 32B is dense, meaning VRAM scales linearly with parameter count and precision. The two MoE models break that pattern: size for total parameters, not the smaller active-parameter figure you'll see quoted in benchmarks. Below 8B, almost any modern GPU works. At 32B you need a 24GB+ card and tight quantization. Above that, you're renting.

Dense Models (0.6B-32B): VRAM at FP16, FP8, and INT4/AWQ

These are weight-only sizes. Add 15-20% for activations and framework overhead, plus KV cache that grows with context length and batch size. That's why a 32B model at ~16GB INT4 weights still needs a 24GB card, not a 16GB one, once you leave room to actually generate anything.

The Qwen team publishes official FP8 checkpoints on Hugging Face, including Qwen3-32B-FP8, using fine-grained FP8 quantization with a 128-element block size rather than naive per-tensor scaling, which is why FP8 Qwen3 holds up close to the bf16 original on most tasks instead of showing the accuracy cliff cruder int8 schemes can produce.

MoE Models (30B-A3B, 235B-A22B): Why Total Params, Not Active Params, Decide VRAM

This is the mistake that trips up most people sizing hardware for Qwen3's MoE variants. Qwen3-30B-A3B activates only 3 billion parameters per forward pass, reportedly matching or beating QwQ-32B while doing roughly a tenth of the per-token compute. That efficiency is real for speed. It does nothing for VRAM.

Every expert in a mixture-of-experts model has to sit in memory because the router can send any token to any expert, and which experts get used shifts token by token. So Qwen3-30B-A3B needs VRAM for all 30B parameters, and Qwen3-235B-A22B needs VRAM for all 235B, even though each forward pass only touches 3B and 22B of them respectively.

Qwen3-30B-A3B at FP8 fits a single H100 or H200 with room for meaningful KV cache. Qwen3-235B-A22B does not fit on any single GPU at any quantization level available today, consumer or data center. Even at AWQ INT4's ~117GB, you need a multi-GPU setup with enough combined VRAM, which is where renting comes in later in this guide.

Run Qwen3 Locally with Ollama (the Fastest Path)

The fastest way to run Qwen3 locally is Ollama: install it, pull a size that fits your GPU, and you have a working chat session in under five minutes with zero manual quantization or CUDA configuration. For the general Ollama mechanics (installation, GPU detection, the REST API), see our how to run LLMs locally with Ollama guide. This section covers what's specific to Qwen3.

ollama pull and ollama run Commands by Size

Ollama's official Qwen3 tags download the model pre-quantized to GGUF Q4_K_M by default. Here's what each size actually downloads and roughly what GPU it needs:

Run any of them with ollama run qwen3:8b for an interactive session. Ollama exposes the same OpenAI-compatible REST API on localhost:11434 for programmatic use, and you can toggle thinking mode mid-conversation with /think and /no_think.

Worth calling out: those GGUF download sizes run 20-30% larger than a naive 4-bit calculation would suggest (Qwen3-8B computes to ~4GB at pure INT4, but the actual GGUF download is 5.2GB). That's Q4_K_M's mixed-precision K-quant scheme keeping certain tensors, like embeddings and attention output layers, at higher precision to protect quality. It's a real cost, and worth knowing before you assume a size fits your card by parameter-count math alone.

One more default worth knowing: Ollama runs with a 2048-token context window unless you raise it. If your prompts or documents run longer than that, set num_ctx explicitly (via /set parameter num_ctx 8192 in an interactive session or the num_ctx option in an API call), and budget extra VRAM for the larger KV cache that comes with it.

Where Ollama's Default Quantization Breaks Down for 32B+ and MoE Variants

Ollama's Q4_K_M default is the right tradeoff for anything up to 14B on a mainstream GPU. It gets shakier past that point for three reasons.

First, headroom. Qwen3-32B's ~20GB GGUF download leaves only 3-4GB of a 24GB card for KV cache, which caps you to a few thousand tokens of context before you hit an out-of-memory error, well short of what a coding or long-document task usually needs.

Second, quality at 4-bit on a 32B model matters more than it does at 8B. The relative accuracy hit from 4-bit quantization tends to compound with fewer, larger experts or deeper networks, and Qwen3-32B is the largest dense model in the family, so it has the least margin to spare.

Third, and this is the hard wall: Qwen3-30B-A3B at ~19GB is workable on a 24GB card only with a short context budget, and Qwen3-235B-A22B's 142GB simply isn't an option on consumer hardware, full stop, regardless of quantization scheme. That's the point where local Ollama stops being the answer and you either accept a smaller Qwen3 size or move the workload to a rented GPU running vLLM.

When Local Isn't Enough: Deploying Qwen3 with vLLM on a Rented GPU

Three things push people off a local box: a model size their GPU can't fit even at 4-bit, a context length their KV cache headroom can't cover, or more than one or two concurrent users. Ollama is built for a single interactive session; vLLM's continuous batching and PagedAttention are built to serve many requests in parallel on data-center hardware. If any of those three apply, this is the setup that replaces ollama run with a production-grade OpenAI-compatible endpoint.

Installing vLLM and Picking a Quantization Flag

Provision a GPU sized to your target model. Spheron's live inventory as of 13 Aug 2026 includes H100 SXM5, H100 PCIe, H100 NVL, H200 SXM5, A100 80GB (SXM4 and PCIe), B200, and B300; check current GPU pricing before you commit, since availability shifts by the hour. SSH in, confirm your driver with nvidia-smi, then install vLLM:

pip install vllm --upgrade
python -c "import vllm; print(vllm.__version__)"

vLLM supports Qwen3 text models natively from version 0.8.4, and thinking mode's enable_thinking flag requires 0.9.0 or later. Pick your --quantization flag based on the GPU:

• fp8 on H100/H200/B200 (native FP8 Tensor Cores): best quality-to-VRAM ratio, real throughput gain over BF16.
• awq on any GPU including A100 (no native FP8 support): smallest footprint, small accuracy tradeoff, routed through vLLM's awq_marlin kernel on Ampere and newer.
• Leave the flag off entirely for BF16 if VRAM isn't the constraint and you want the reference-quality baseline.

Launch Commands for 8B (Single GPU), 32B (Single H100), and 235B-A22B (Multi-GPU)

The pattern is the same at every size: match the GPU to the model, then swap the --quantization flag and model repo. Here's the 32B config most teams land on, since it's the single-GPU production sweet spot:

vllm serve Qwen/Qwen3-32B \
    --quantization fp8 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 32768 \
    --port 8000

The 8B config swaps in a smaller GPU and, on hardware without native FP8 support, an AWQ checkpoint (Qwen/Qwen3-8B-AWQ with --quantization awq). The 235B-A22B config swaps in --tensor-parallel-size 8 across 8x H100 and the full Qwen/Qwen3-235B-A22B weights. For the complete command set across all seven sizes, A100-specific quantization notes, and exact runtime VRAM math including framework overhead, see our deploy Qwen3 on GPU cloud guide, which is the deeper reference once you've picked a size here. Note that Alibaba's newer Qwen3.7 line is API-only with no downloadable weights; if you're searching broadly for "Qwen3" hardware requirements and land on Qwen3.7, it's a different, closed model from the open-weight family covered here.

Test the deployment the same way you'd test any vLLM server: a curl request to the OpenAI-compatible http://localhost:8000/v1/chat/completions endpoint with your chosen model ID in the payload.

Quantization Options to Shrink the VRAM Footprint

You've already seen three quantization formats across this guide: GGUF Q4_K_M (Ollama's default), FP8, and AWQ INT4. Here's the actual quality and throughput tradeoff, not just the VRAM math.

FP8 vs AWQ INT4 vs GGUF Q4_K_M: Quality and Throughput Tradeoffs

Qwen's own benchmarks, run on Qwen3-8B on H20 GPUs with SGLang at a 30,720-token input (batch size 1), show a real throughput difference between formats, not just a memory one:

FP8 wins on both axes here: half the VRAM of BF16 and roughly 1.5x the throughput, because H20 and H100-class GPUs have native FP8 Tensor Cores that BF16 can't use. AWQ-INT4 still beats BF16 on throughput (about 1.3x) despite the extra dequantization work, because the smaller memory footprint reduces the bandwidth bottleneck that dominates decode-phase inference.

The gap gets more dramatic on hardware without a fast batched-inference engine. Qwen's benchmarks for Qwen3-14B running through plain Hugging Face Transformers at the same 30,720-token input show BF16 at 155.38 tok/s versus AWQ-INT4 at 363.25 tok/s, a 2.3x speedup from quantization alone, on the exact same GPU. If you're not using vLLM or SGLang, quantization is doing double duty: it's not just fitting the model, it's compensating for a slower serving stack.

For local use, GGUF Q4_K_M (what Ollama pulls by default) is the practical choice since it needs no manual setup, runs on llama.cpp's CPU/GPU-hybrid engine, and preserves roughly 95% of full-precision quality on most conversational and writing tasks. For rented-GPU production serving, FP8 is the better default on H100/H200/B200; fall back to AWQ INT4 only when you need to fit a larger Qwen3 size onto a smaller GPU than FP8 allows.

Cost-Per-Token: Qwen3 vs Comparable Llama and Gemma Sizes

Once you're paying by the hour for a GPU instead of running on hardware you already own, cost-per-token becomes the number that matters. Here's how Qwen3-32B stacks up against the two most common alternatives at a similar size class.

Qwen3-32B vs Gemma 3 27B vs Llama 3.1 70B on Quality and $/M Tokens

A100 lacks native FP8 Tensor Cores, so INT8 or AWQ INT4 is the right call there; FP8 only makes sense on H100, H200, or B200 where the hardware can execute it natively.

Prices are Spheron on-demand per-GPU rates as of 13 Aug 2026, split from live DEDICATED offers, not blended with spot. Quality, by each model's own published benchmarks:

These come from three separate benchmark suites (Qwen's own technical report, Google's Gemma 3 model card, and Meta's Llama 3.1 model card), so treat them as directional rather than apples-to-apples. What's clear is the shape of the tradeoff: Qwen3-32B leads on reasoning-heavy benchmarks (GPQA-Diamond, MMLU-Redux) at half the parameter count of Llama 3.1 70B, and Gemma 3 27B is competitive on math and code at an even smaller footprint that fits a cheaper A100.

Here's a worked cost example using Spheron's own published Qwen3-32B benchmarks: single-stream FP8 throughput on H100 runs 65-80 tok/s. At the 72.5 tok/s midpoint, that's roughly 261,000 tokens/hour, putting single-stream cost around $10.15 per million tokens on-demand ($2.65/hr divided by 0.261M tokens). Production serving with vLLM's continuous batching multiplies effective throughput 4-10x for concurrent requests, so realistic batched cost lands closer to $1.50-2.50 per million tokens on the same hardware. As a rule of thumb, decode cost scales roughly with active parameter count at a fixed precision and batch size, so expect Llama 3.1 70B to run meaningfully more expensive per token than Qwen3-32B at equivalent batch sizes, since it needs more than double the active compute per forward pass. Gemma 3 27B, with fewer active parameters than Qwen3-32B, typically comes out cheaper per token on comparable hardware. Benchmark your own workload with vllm bench serve before committing, since real throughput depends heavily on your prompt length, output length, and concurrency pattern.

For a deeper cost-per-token breakdown across DeepSeek V3.2, Llama 4, and Qwen 3, including spot pricing scenarios, see our DeepSeek vs Llama 4 vs Qwen 3 cost comparison.

Pricing fluctuates based on GPU availability. The prices above are based on 13 Aug 2026 and may have changed. Check current GPU pricing → for live rates.

Production Serving Tips: Batching, Concurrency, and Context Length

A few settings matter more than the rest once you're running Qwen3 on a rented GPU with real traffic:

• --max-num-seqs controls how many requests vLLM batches concurrently. Higher values improve aggregate throughput but increase per-request latency under load; tune it against your actual concurrency, not a default.
• --gpu-memory-utilization caps total GPU memory vLLM will claim, typically 0.9. Raising it buys more KV cache headroom but leaves less margin before an out-of-memory error under a traffic spike.
• --max-model-len directly controls KV cache pre-allocation. Set it to what your workload actually needs, not the model's full 128K ceiling; a lower cap frees real VRAM for concurrent requests.
• Thinking mode token cost. Enabling enable_thinking produces a hidden reasoning block before the final answer, which isn't billed separately in most local setups but consumes real GPU compute and adds latency. For high-concurrency production endpoints, consider defaulting to non-thinking mode and opting in per-request only where accuracy on math or logic tasks justifies it.
• Expert parallelism for MoE. If you scale Qwen3-30B-A3B or 235B-A22B across multiple GPUs, add --enable-expert-parallel in vLLM for better throughput than plain tensor parallelism on sparse models.

Qwen3 runs great locally through Ollama up to about 14B, and Spheron picks up exactly where your local GPU runs out of room, from a single H100 for the 32B model up to multi-GPU H100 clusters for the full 235B-A22B MoE.

H100 SXM5 on Spheron → | A100 GPU rental | Check current GPU pricing →

Get started on Spheron →