Deploy Grok 2.5 on GPU Cloud: VRAM and Setup (2026)

xAI open-sourced Grok 2.5's weights on Hugging Face on August 24, 2025. Elon Musk announced it directly: "The @xAI Grok 2.5 model, which was our best model last year, is now open source" (TechCrunch). Almost a year later, it's still one of the more downloaded 270B-parameter open-weight models on the platform, and there's a real gap between "you can download it" and "you know exactly what it takes to run it." This is the second part: the license terms, the exact hardware floor, and the documented SGLang command that gets it serving.

If you've read a generic "is Grok open source" explainer already, skip ahead. This is a deployment guide: what the xAI Community License actually restricts, why the checkpoint hard-requires 8 GPUs at more than 40GB each, and the specific commands to get it running on rented hardware.

Grok 2.5's License and What It Actually Lets You Do

Grok 2.5's weights ship under the xAI Community License Agreement, not Apache 2.0 or MIT. That distinction matters more than most "open-weight" coverage lets on, because the license carves out a restriction that directly affects anyone building on top of another AI model with Grok 2.5's help.

What the xAI Community License Permits and Restricts

The license text permits commercial use, redistribution, and fine-tuning, subject to xAI's Acceptable Use Policy. If you distribute the weights or any derivative of them, you're required to include the full license agreement, a specific copyright notice ("This product includes materials licensed under the xAI Community License. All rights reserved. Copyright © xAI [year]."), and display "Powered by xAI" somewhere in the interface or materials. That's a real obligation, not a suggestion, and it's easy to miss if you're used to permissive licenses that don't require attribution in the UI.

Commercial Use vs the "No Training Other Models" Clause

Here's the clause that trips people up: you cannot use the materials, any derivative of them, or their outputs, including generated data, "to train, create, or improve any foundational, large language, or general-purpose AI models," except when fine-tuning xAI's own materials under the terms of the agreement itself. In plain terms, you can build a product on Grok 2.5, fine-tune it, and charge customers for access to it. What you can't do is use Grok 2.5's outputs as training data to build or improve a different foundation model, yours or anyone else's. If your use case is inference, fine-tuning, or an application layer on top of Grok 2.5, the license is workable. If your use case is generating synthetic data to train a competing base model, it isn't, and that's a meaningfully different restriction from a straightforwardly permissive license like Apache 2.0.

Hardware Requirements: Why You Need 8x GPUs at 40GB+ Each

This is the part most coverage skips entirely, and it's the part that determines whether you can actually run this thing. The model card is specific: TP=8, each GPU needing more than 40GB of memory, no smaller supported split (Hugging Face model card).

Model Architecture: 270B Total, 115B Activated (MoE)

Grok 2.5 totals approximately 270 billion parameters (269,515,497,472 exact), with roughly 115 billion (115,019,056,768) activated per forward pass (community architecture analysis on the model card discussion). It's a mixture-of-experts model: 8 experts per MoE layer, 2 active per token, so each forward pass touches about 43% of total parameters. The architecture runs 64 transformer layers at a model dimension of 8,192.

That MoE split is why VRAM sizing and compute cost decouple from each other here. Compute-wise, a forward pass costs closer to what you'd pay for a ~115B dense model. But every one of the 270B parameters still has to sit resident in GPU memory, because you don't know in advance which 2 of 8 experts a given token will route to. That's the same trade-off you'd hit sizing any sparse MoE deployment; the MoE inference optimization guide covers the general memory-vs-compute math if you want the broader picture beyond this specific model.

Why TP=8 Is Not Optional With This Checkpoint

Tensor parallelism splits a model's weight matrices across GPUs so each card holds a fraction of every layer. xAI published Grok 2.5's checkpoint pre-sharded for TP=8, and there is no officially supported way to run it on TP=4 or TP=2 without re-sharding the weights yourself, which is an unsupported, manual process most teams shouldn't take on for a production deployment.

The practical floor: 8 GPUs, each with more than 40GB of VRAM. The checkpoint on Hugging Face totals approximately 539GB across 39 safetensors files, stored at native bf16 precision, roughly 2 bytes per parameter across ~270B parameters, which is where that 539GB figure comes from. Loaded raw at that precision, the weights alone would need about 67GB per GPU, well past what a 40GB card can hold. The --quantization fp8 flag in the launch command is what makes an 8x40GB+ floor achievable at all: FP8 stores each parameter in roughly half the space of bf16, bringing the resident weight footprint down to somewhere around 34GB per GPU before KV cache and activation memory. That's still tight on a card with exactly 40GB, which is why the model card's ">40GB per GPU" reads as a hard minimum rather than a target, and why teams with real batch-size or long-context requirements look to 80GB-class cards for headroom. For a broader reference on how VRAM math scales across model sizes and quantization tiers, see the best open-source LLMs to self-host VRAM tier guide.

GPU Options That Clear the 40GB-Per-Card Bar

We pulled live rates for an 8-GPU cluster on Spheron as of 08 Jul 2026, filtered to on-demand (dedicated) rates, since spot rates get reclaimed without notice and this is a checkpoint you don't want to babysit mid-restart:

The A100 80GB cluster is the cheapest way to clear the requirement and still get NVLink between all 8 cards, which matters at TP=8: every forward pass exchanges activations across the full cluster, and PCIe-only interconnect (as on L40S) will bottleneck throughput even though the raw price looks close. Rent A100 80GB instances on Spheron if VRAM headroom above the 40GB floor matters less than cost. If you want more room for longer contexts or larger batches, an H100 SXM5 cluster gives you the same NVLink topology with 80GB per card and higher Tensor Core throughput, and an H200 SXM5 cluster pushes to 141GB per card if KV cache headroom at long context is the deciding factor.

Pricing fluctuates based on GPU availability. The prices above are based on 08 Jul 2026 and may have changed. Check current GPU pricing → for live rates.

Serving Grok 2.5 With SGLang on Spheron

xAI documents SGLang, not vLLM, as the supported serving path for this checkpoint. If you're deciding between the two for other models in your stack, the vLLM vs SGLang benchmarks cover the general tradeoffs, but for Grok 2.5 specifically, SGLang is the one with a documented, tested launch command.

Prerequisites: SGLang >= v0.5.1 and Downloading the 539GB Checkpoint

Install SGLang v0.5.1 or later before touching the model. Then pull the weights:

pip install --upgrade pip
pip install "sglang[all]>=0.5.1"

# ~539GB across 39 files. Budget storage accordingly and expect the
# download to take a while; retry on transient errors rather than
# restarting from scratch.
hf download xai-org/grok-2 --local-dir /local/grok-2

Provision the storage before the download starts, not after. On Spheron, you can spin up an 8-GPU instance and attach volume storage through the dashboard or the API if you want to script the provisioning step; either way, size local disk for at least 600-650GB to leave room for the checkpoint plus working space.

Launch Command: FP8 Quantization and Triton Attention Backend

This is the exact command from xAI's model card, unmodified:

python3 -m sglang.launch_server \
  --model /local/grok-2 \
  --tokenizer-path /local/grok-2/tokenizer.tok.json \
  --tp 8 \
  --quantization fp8 \
  --attention-backend triton

A few things worth knowing before you run it. The --tokenizer-path flag points at tokenizer.tok.json, not a standard Hugging Face tokenizer file, so if you're scripting this into an existing pipeline, don't assume the default tokenizer resolution will find it. The --quantization fp8 flag is what keeps the per-GPU footprint down near the ~34GB figure from the hardware section; running it at full bf16 precision would push per-GPU weight memory back up toward 67GB and blow past the 40GB-per-card floor on most of the GPUs in the table above. For a deeper look at what FP8 quantization is actually doing to precision and throughput, see the FP8 quantization explainer. The --attention-backend triton flag is required for this checkpoint's attention implementation; SGLang's default backend selection won't pick the right one automatically for Grok 2.5.

Grok 2.5 also uses a specific separator token, <|separator|>, as part of its post-training chat template. If you're building a custom inference wrapper rather than hitting SGLang's OpenAI-compatible endpoint directly, make sure your prompt formatting uses the correct chat template rather than a generic one, or you'll get degraded output that looks like a model bug when it's actually a formatting mismatch. For the broader SGLang production setup, including RadixAttention and multi-GPU tuning beyond this one launch command, see the SGLang production deployment guide.

Verifying the Deployment and Basic Inference Test

Once the server is up, confirm the model loaded and check that request handling looks right:

curl http://localhost:30000/v1/models

You should see /local/grok-2 (or however you've named the served model) in the response. Follow with a basic completion request against SGLang's OpenAI-compatible endpoint to confirm end-to-end inference works before you point production traffic at it:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/local/grok-2",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "max_tokens": 64
  }'

If that returns coherent output, TP=8 sharding and the FP8 quantization path both loaded correctly across all 8 GPUs. A garbled or empty response is more often a tokenizer path or chat template mismatch than a hardware problem, so check those first before assuming a GPU is misconfigured.

Grok 2.5 vs Other Open-Weight MoE Models on GPU Cost

Grok 2.5's 8x40GB+ floor sits at the lighter end of open-weight MoE hardware requirements, not the heavier end. Trillion-parameter checkpoints like Kimi K2.6 need 8x H200 or larger just to clear their VRAM floor at usable precision; Grok 2.5's 270B total size means the A100 80GB cluster in the table above is a genuinely viable option, not just a theoretical one.

We pulled those figures straight from each model's own deployment guide rather than estimating, so the comparison holds up against the same FP8 assumptions used in the hardware section above. For a full breakdown of VRAM and live cost across a wider set of open-weight models, the GPU requirements cheat sheet is the single-table reference. If Grok 2.5's 270B size is more than your workload needs, DeepSeek V4-Flash clears a similar GPU floor at a smaller activated-parameter cost; if you want more capability and can afford the extra hardware, GLM-5.1 is the current top open-weight model on Code Arena. For a broader look at cost-per-token across the field, the open-weight frontier model showdown covers GPT-OSS, GLM, and DeepSeek head-to-head.

FAQ

How much VRAM does Grok 2.5 need to self-host? Eight GPUs at more than 40GB each, running TP=8. That's not a rough estimate, it's the exact configuration xAI documents, and the released checkpoint doesn't support anything smaller.

Can I run it on fewer than 8 GPUs? Not with the published checkpoint. Re-sharding for TP=4 is technically possible but unsupported and not something most teams should take on just to save GPUs.

Is Grok 2.5 free to use commercially? Yes, under the xAI Community License, as long as you follow the Acceptable Use Policy and don't use the weights, derivatives, or outputs to train a different foundation model.

What SGLang version do I need? v0.5.1 or later, using the exact launch command in the deployment section above.

How does the MoE architecture change what "270B parameters" actually means for cost? You need memory for all 270B parameters, but compute cost per token tracks closer to the ~115B activated per forward pass, since only 2 of 8 experts fire per token.

An 8-GPU cluster is the real floor for Grok 2.5, and NVLink interconnect is what keeps TP=8 from bottlenecking on activation exchange. Check live GPU cluster pricing → across the full lineup before you provision.

Get started on Spheron →