Inkling Model GPU Requirements: Self-Host the 975B MoE (2026)

Inkling model GPU requirements are the first question any team asks after reading the spec sheet. Thinking Machines Lab released Inkling on July 15, 2026: a 975B-parameter Mixture-of-Experts model with 41B active parameters per token, and its own model card puts the number that matters most for self-hosting front and center (source). The BF16 checkpoint needs a minimum of 2TB of aggregated VRAM. The NVFP4 quantized version drops that to 600GB. Neither number fits on a single GPU node you'd casually spin up, so this is a hardware-sizing problem before it's a serving problem, and it's exactly the kind of multi-GPU math this blog exists to work through.

Inkling is Mira Murati's Thinking Machines Lab's first open-weight release, and it debuted at 41 on the Artificial Analysis Intelligence Index, the leading open-weights score from a US lab, ahead of Nemotron 3 Ultra (38), Gemma 4 31B (29), and gpt-oss-120b (24) (source). It's also Apache 2.0 licensed, which means no usage restrictions once you've cleared the VRAM bar. If you're already working from our best open-source LLMs to self-host in 2026 guide, Inkling sits well above every tier in that guide and forces a different set of hardware decisions, which is what the rest of this post walks through. For the underlying VRAM math this post builds on, see GPU memory requirements for LLMs.

Inkling's Architecture: 975B Total, 41B Active, and the Thinking-Effort Dial

Inkling is a 66-layer decoder-only transformer with a hidden size of 6,144 and a sparse MoE feed-forward backbone (source). The 975B-to-41B ratio is the entire economic argument for MoE at this scale: you pay VRAM for 975B parameters sitting resident, but compute for a forward pass costs roughly what a 41B dense model would cost. That's a 24x gap between what you store and what you run, and it's why Inkling needs a GPU cluster that looks wildly oversized next to its actual per-token compute.

Reasoning depth is controllable through a reasoning_effort parameter with seven named levels: none, minimal, low, medium, high, xhigh, and max (source). That's a wider dial than most reasoning models ship with. Moonshot's Kimi K3, for comparison, only exposes three values (low, high, max) and can't turn reasoning off at all; Inkling's none setting is a genuine off switch. More on tuning this dial later in the post.

MoE Routing: 256 Experts, 6 Routed Plus 2 Shared Per Token

Each MoE layer holds 256 routed experts plus 2 shared experts that are always active. The router selects 6 of the 256 routed experts per token, so every forward pass touches 8 experts total (6 routed + 2 shared) out of 258 available (source). The VRAM consequence is blunt: all 258 experts across all layers have to be resident in memory, because the router can send any token to any expert on any forward pass. You don't get to skip loading the experts that happen to fire less often.

This puts Inkling in the same weight class as a run of large open MoE releases from the past few months. Nemotron 3 Ultra is 550B total with 55B active. Mistral Large 3 is 675B total with 41B active, the same active-parameter count as Inkling despite a smaller total. GLM-5.2 is 744B total with roughly 40B active. Inkling's 975B total is the largest of the four, and its active-parameter count landing right alongside Mistral Large 3's is a useful anchor if you've already sized hardware for that model.

Hybrid Local-Global Attention and Native Multimodal Input

Of Inkling's 66 layers, 55 use sliding-window attention with a 512-token window, and 11 use global attention with a 1,024-token bias extent, in a repeating 5-to-1 local-to-global pattern (source). Local layers run grouped-query attention at 64 query heads to 16 key-value heads (a 4:1 ratio); global layers tighten that to 64 query heads over 8 key-value heads (8:1). Rather than RoPE, Inkling uses a learned, input-dependent relative-position bias derived from the query and key states, with RMS normalization applied before attention.

The practical upside of mostly-local attention is KV cache efficiency at long context. Only 11 of 66 layers carry the full quadratic cost of global attention; the other 55 cap their attention window at 512 tokens regardless of sequence length. Combined with a context window of up to 1,048,576 tokens in the open-weight release (256K on the hosted Tinker API) (source), that architecture choice is what makes a 1M-token context window survivable on GPU memory rather than an instant KV cache blowout.

Inkling was also pretrained on 45 trillion tokens spanning text, images, audio, and video, and can reason natively across all four modalities even though current outputs are limited to text (source). That native multimodal training shows up directly in the benchmark numbers: Inkling scores 73.5% on MMMU Pro, a multimodal reasoning benchmark, alongside 97.1% on AIME 2026, 77.6% on SWEBench Verified, and 43.9% on SimpleQA (source).

Inkling Model GPU Requirements: BF16 vs NVFP4 GPU Counts

The short version: BF16 needs a 2TB VRAM floor and NVFP4 needs 600GB. Both numbers come straight from Thinking Machines' own model card, and both are set by total parameter count, not the 41B active count. Two precisions, two very different clusters.

BF16 Checkpoint: 2TB VRAM, 8x B300 or 16x H200

Raw weight math at BF16 (2 bytes/param) is 975B x 2 = 1,950GB, or roughly 1.95TB. The published 2TB minimum leaves only thin headroom above that raw figure, which is why the recommended configs run well past the bare floor: 8x NVIDIA B300 SXM6 (288GB each, 2,304GB total) or 16x NVIDIA H200 SXM5 (141GB each, 2,256GB total) (source). Neither config fits in a single 8-GPU node at full precision. The B300 path clears it in one node; the H200 path needs two 8-GPU nodes joined by InfiniBand, which matches the official vLLM recipe's guidance that BF16 on Hopper requires a multi-node setup with a disaggregated expert-parallel (DEP) strategy across 2 nodes (source).

NVFP4 Quantized: 600GB VRAM, 4x B300 (W4A4) or 8x H200 (W4A16)

Pure 4-bit weight math (0.5 bytes/param) works out to 975B x 0.5 = 487.5GB, so the published 600GB minimum bakes in roughly 110GB of overhead beyond raw weights. The two supported paths are W4A4 on 4x B300, and W4A16 on 8x H200 (source). The naming tells you the hardware story directly: W4A4 means 4-bit weights and 4-bit activations, which needs Blackwell's native FP4 tensor cores (SM100+) to compute without a dequantization step. W4A16 means 4-bit weights but 16-bit activations, which is what Hopper runs because H200 has no native FP4 tensor cores; the GPU keeps activations at higher precision and only compresses the stored weights.

Notice that both NVFP4 configs land far above the 600GB floor: 4x B300 gives you 1,152GB, and 8x H200 gives you 1,128GB. That's not wasted capacity so much as it's the tensor-parallel degree the model actually needs for the FFN compute to spread cleanly across GPUs. For MoE models specifically, GPU count is set by parallelism requirements as much as by raw VRAM division. The NVFP4 vs MXFP4 quantization guide covers the format tradeoffs and hardware support matrix in more depth if you're deciding between quantization schemes for a different model.

VRAM Sizing Table by Config

For a broader comparison against the H200/B200/GB200 family, including where each GPU generation's memory bandwidth and NVLink domain matters for tensor parallelism at this scale, see H200 vs B200 vs GB200.

Where Inkling Fits in the Best Open-Source LLMs to Self-Host in 2026

Our best open-source LLMs to self-host in 2026 guide covers tiers from 12GB up through 80GB+ single-GPU setups, and its top tier tops out around single-node H100/H200/B200 deployments serving 70B-class dense models or mid-size MoEs. Inkling doesn't fit anywhere in that guide. Even at NVFP4, its 600GB floor is more than 4x the 141GB a single H200 offers, and at BF16 it's more than 24x a single H200's capacity.

That puts Inkling in a distinct bracket alongside the other trillion-adjacent open MoEs that have shipped this year: DeepSeek V4-Flash (284B total, 13B active) is the smallest of the group and the easiest to self-host, GLM-5.2 (744B total, ~40B active) and Mistral Large 3 (675B total, 41B active) sit in the middle, and Inkling's 975B total is the largest. If your team is already running one of these and evaluating whether Inkling is worth the upgrade, the honest answer from Thinking Machines Lab itself is worth reading in full:

"Inkling is not the strongest overall model available today, open or closed. Instead, a combination of qualities makes it a good open-weights base for customization: multimodal capabilities, efficient thinking, and availability on Tinker for fine-tuning." - Thinking Machines Lab, Inkling release notes

That's a narrower pitch than most model launches make, and it's a useful filter: Inkling is the pick when multimodal input, controllable reasoning depth, and Tinker fine-tuning matter to your use case specifically, not a blanket "biggest number wins" upgrade.

Serving Inkling With vLLM or SGLang on Rented H100/H200 Clusters

Inkling ships with day-zero inference support across vLLM, SGLang, Hugging Face Transformers, llama.cpp, TokenSpeed, and Unsloth (source). For teams comparing the two production-grade serving engines here, vLLM vs SGLang 2026 covers the RadixAttention vs PagedAttention tradeoff in general; the notes below are specific to Inkling's checkpoint and flags.

One hardware note before the commands: Thinking Machines' official recipe scopes support to "NVIDIA Hopper (H200/H20) or Blackwell" (source), which pointedly excludes H100. The math explains why. An 8x H100 SXM5 node has 640GB of total VRAM, which technically clears the 600GB NVFP4 floor but leaves almost nothing for KV cache, request batching, or any headroom at all. H200's extra 61GB per GPU (141GB vs 80GB) is what turns a razor-thin fit into a workable production node, which is why the H100 SXM5 generation isn't the recipe's recommended path for this specific model even though it handles plenty of other 70B-500B class MoEs comfortably.

vLLM NVFP4 Setup (TP8 on H200, TP4 on GB200/B300)

The official vLLM recipe's default config targets 4x GB200 at --tensor-parallel-size 4 for the NVFP4 checkpoint. On Hopper hardware, NVFP4 runs at --tensor-parallel-size 8 on H200 instead. Required flags and environment variables, per the recipe:

export VLLM_USE_V2_MODEL_RUNNER=1
export FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1

vllm serve thinkingmachines/inkling-nvfp4 \
  --tokenizer-mode inkling \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-auto-tool-choice \
  --tensor-parallel-size 8 \
  --kernel-config.enable_flashinfer_autotune=False \
  --trust-remote-code

Set --tensor-parallel-size 4 instead of 8 if you're deploying on a Blackwell-class node like B300 or GB200 NVL72, matching the recipe's default. Verify the exact checkpoint repository name and flag set against the live recipe before deploying, since vLLM and SGLang both ship frequent updates for new model architectures.

SGLang Day-Zero Support and Prefill-Decode Disaggregation

SGLang's day-zero Inkling support is broad: BF16 and NVFP4 checkpoints, both NVIDIA and AMD execution paths, speculative decoding, prefill-decode disaggregation, multi-LoRA serving, and HiCache (source). Prefill-decode disaggregation matters specifically for Inkling's mixed workload profile: the 1M-token context window means prefill can be extremely long-running for document-heavy requests, and separating that phase onto dedicated GPUs keeps it from stalling decode throughput for concurrent short requests. That's the same tradeoff covered in more general terms for other large context-window MoEs.

Framework support announcements need the usual caveat before you build on them: confirm the exact checkpoint format, GPU generation, CUDA stack, container image, and kernel version in your environment before routing production traffic, since day-zero support claims can lag real-world stability by a few point releases.

Speculative Decoding with the Multi-Token Prediction Drafter

Inkling ships with Multi-Token Prediction (MTP) drafter layers built into the checkpoint for speculative decoding, no separate draft model required (source). The official vLLM recipe's benchmark on 4x GB200 with the MTP drafter enabled (MTP8 configuration) reaches about 380 tok/s/user (source). If your serving stack is already tuned for another MTP-equipped model, the mechanics carry over directly: the drafter proposes several tokens ahead, and the main model verifies them in a single forward pass, accepting however many match before falling back to standard autoregressive decoding.

Tuning reasoning_effort for Latency vs Quality

Inkling's reasoning_effort parameter takes one of seven values: none, minimal, low, medium, high, xhigh, or max (source). Each step up generates a longer chain-of-thought trace before the final answer, which is a direct trade of latency and reasoning-token spend against accuracy on hard problems.

A practical starting point: use none or minimal for classification, extraction, and routing calls where you don't need visible reasoning and speed matters more than a marginal accuracy gain. medium is a reasonable default for general chat and moderate-complexity coding. Reserve high and xhigh for math, multi-step agentic planning, and code review where the extra thinking tokens demonstrably move benchmark scores; Inkling's 97.1% on AIME 2026 and 77.6% on SWEBench Verified are the kind of results that come from real thinking budget, not a none-effort call. max should be the exception, not the default, since every level between high and max is a tradeoff you're paying for whether you need it or not.

This matters for cost as much as latency. On the hosted Tinker API, output tokens (which include reasoning tokens) are billed separately and at a higher rate than input tokens: $4.68/M output vs $1.87/M input at 64K context (source). Leaving reasoning_effort unset or defaulting to max on every call is the single easiest way to overspend on a workload that didn't need the extra reasoning depth.

Inkling vs Kimi K2.6, DeepSeek V4 Flash, and Nemotron 3 Ultra: Benchmark Comparison

Artificial Analysis ran Inkling against the current open-weights field on agentic and reasoning benchmarks. The headline numbers:

(Sources: Artificial Analysis)

The output-token efficiency number is the one worth sitting with. Inkling averages 25K output tokens per Intelligence Index task, against 38K for Kimi K2.6, 37K for DeepSeek V4 Pro Max, and 43K for GLM-5.2 Max. Artificial Analysis frames this as roughly one-third the token spend of Nemotron 3 Ultra for equal coding performance, which is a direct cost lever if you're paying per token on a hosted API or paying for GPU-seconds on self-hosted decode. Kimi K2.6 and DeepSeek V4-Flash both remain strong picks for agentic workloads where their respective tool-use ecosystems are already built out; Inkling's edge here is doing comparable or better agentic scoring with meaningfully fewer generated tokens per task.

One caveat worth flagging before you lean on Inkling for anything knowledge-dependent: on AA-Omniscience, a knowledge-accuracy benchmark, Inkling scores +2 against Nemotron 3 Ultra's -1, but that comes with 40% accuracy and a 63% hallucination rate (source). Strong agentic and coding scores don't imply strong factual recall; if your workload needs grounded knowledge answers, pair Inkling with retrieval rather than trusting parametric knowledge alone.

Cost to Self-Host Inkling on Spheron vs the Tinker API

Live Spheron on-demand pricing for the GPUs Inkling actually needs, fetched 03 Aug 2026:

Spot pricing was live for both GPUs at fetch time and runs meaningfully cheaper: H200 SXM5 spot at $3.31/GPU/hr (8x = $26.51/hr) and B300 SXM6 spot at $5.81/GPU/hr (4x = $23.24/hr). Spot capacity can be reclaimed without notice, so it fits batch and evaluation workloads better than latency-sensitive production serving.

Pricing fluctuates based on GPU availability. The prices above are based on 03 Aug 2026 and may have changed. Check current GPU pricing → for live rates.

Against the Tinker API's $1.87/M input and $4.68/M output at 64K context, or $3.74/M input and $9.36/M output at 256K context (source), the self-hosted math depends heavily on how much concurrent request batching your serving stack achieves, since output tokens dominate cost on both sides. The 380 tok/s/user figure from the official recipe's MTP-enabled benchmark is a single-stream decode number, not aggregate cluster throughput, so it isn't a safe input for a cost-per-token estimate on its own; batching multiple concurrent requests against the same $46.86/hr 8x H200 cluster is what determines whether self-hosting undercuts the Tinker API. Benchmark your own batched throughput before committing to either path. For occasional or bursty usage where GPUs would sit idle between calls, the Tinker API's per-token pricing is the more capital-efficient choice; provisioning a $34K/month cluster for sporadic calls doesn't pencil out.

Production Notes: Context Length, Multimodal Inputs, and When to Use Inkling-Small

The open-weight release supports up to 1,048,576 tokens of context, while the hosted Tinker API caps at 256K (source). If you need the full 1M window, plan KV cache budget carefully: even with 55 of 66 layers capped at a 512-token sliding window, the 11 global-attention layers still scale with sequence length, and at 1M tokens that's not a trivial allocation on top of the weight footprint already claiming your VRAM.

Native multimodal input (text, image, audio) is trained in, but current outputs are text-only (source), so plan your input pipeline for multimodal ingestion but don't expect multimodal generation. The 45-trillion-token pretraining mix across text, images, audio, and video is what backs the 73.5% MMMU Pro score, so this isn't a bolted-on vision adapter; it's trained natively into the same weights that handle text reasoning.

If the 600GB NVFP4 floor is more cluster than your workload justifies, Inkling-Small is the fallback: 276B total, 12B active parameters, same training recipe, with a NVFP4 checkpoint fitting in roughly 180GB (source). That's a single B300 or two H200s instead of a four-to-eight-GPU cluster, at the cost of ceiling performance on the hardest reasoning tasks. For teams evaluating whether the jump to full Inkling is worth it, starting on Inkling-Small and measuring the gap on your own eval set is a cheaper way to find out than provisioning the full cluster up front. The GB200 NVL72 page covers the reservation path if you specifically want to match the official recipe's default hardware rather than the H200/B300 alternatives covered here.

Inkling's 600GB NVFP4 floor means you're provisioning a multi-GPU cluster from day one, not scaling into one later.

H200 SXM5 on Spheron → | B300 SXM6 capacity → | View all GPU pricing →

Get started on Spheron →