MoE Inference

Mixture-of-experts models need a different deployment playbook than dense ones: you have to hold every expert in memory but only compute on a handful per token, which changes how you think about VRAM, parallelism, and cost. This cluster covers that playbook end to end, from deploying specific MoE releases (DeepSeek V4 and V4-Flash, Kimi K2.6 and K3, GLM-5.1 and 5.2, Qwen3.6, Mistral Large 3, Grok 2.5, Hunyuan 3, and more) to the underlying mechanics: expert parallelism, wide expert parallelism with online load balancing, and dynamic expert quantization that keeps hot experts in FP8 and cold ones in INT4.

We also cover fine-tuning MoE models specifically, since standard LoRA setups don't handle router stability or auxiliary loss the same way dense fine-tuning does, and speculative decoding for MoE, which needs draft heads that understand expert routing.

The pillar guide, MoE Model Inference on GPU Cloud, lays out the core tradeoff: how expert parallelism and memory layout interact, and how to deploy efficiently with vLLM, SGLang, and spot pricing. Start there before jumping into a specific model's deployment guide, since the concepts carry across almost every MoE release in this cluster.