Deploy NVIDIA Nemotron 3 Nano Omni on GPU Cloud: Self-Host the 30B-A3B Omni-Modal Perception Model for Multimodal AI Agents (2026 Guide)

NVIDIA released Nemotron 3 Nano Omni on April 28, 2026 under the NVIDIA Nemotron Open Model License. The defining numbers are 30B total parameters and 3B active per token: a MoE design that gives you omni-modal perception at a compute cost close to a 3B dense model. It accepts text, images, video, and audio in a single inference loop and produces text output, with a 300K token context window and an approximately 16K token reasoning budget built in. For anyone building multimodal agent stacks, this changes the cost math on the perception layer considerably.

Before diving in, two related deployment guides worth bookmarking: the Nemotron 3 Super deployment guide covers the 120B/12B hybrid Mamba-Transformer tier for single-GPU deployment, and the Qwen3.5-Omni guide is a useful comparison point for other omni models in the same weight class.

What Is NVIDIA Nemotron 3 Nano Omni

Nemotron 3 Nano Omni is a hybrid Mamba-Transformer MoE model with three distinct architectural components: Mamba SSM layers for recurrent state across the sequence, standard Transformer attention layers, and Conv3D layers that process video frames. The Conv3D layers are the distinguishing piece: the model keeps a vision encoder (C-RADIOv4-H) and uses Conv3D to fuse pairs of frames into tubelets before passing them to the encoder. Efficient Video Sampling (EVS) then prunes redundant static tokens. Together, the Conv3D + EVS path reduces encoder compute by roughly 9x compared to per-frame ViT-based approaches, per NVIDIA's published figures.

The NVIDIA Nemotron Open Model License is not Apache 2.0. It has specific terms around commercial use that differ from fully permissive open-source licenses. Check the model card license section before deploying commercially.

Why an Omni-Modal Perception Sub-Agent

The most practical use pattern for Nano Omni is not as a standalone assistant but as the perception layer in a two-model agent stack. Nano Omni receives raw video, audio, or images and returns structured text. A larger reasoning model gets that text and handles planning and decisions.

The numbers that make this work:

• ~9.2x effective system capacity for video reasoning vs alternative open omni models at a fixed per-user interactivity threshold, per NVIDIA's published benchmarks.
• ~9x lower compute for video reasoning because Conv3D fuses frame pairs into tubelets before the vision encoder and Efficient Video Sampling (EVS) prunes redundant static tokens, reducing total encoder compute compared to per-frame ViT-based approaches. This is NVIDIA's published figure for the Conv3D + EVS path.
• Single inference loop: audio, video, and text pass through one model call instead of an ASR pipeline feeding a VLM feeding an LLM.

At 3B active parameters, each Nano Omni inference call is cheap enough that you can call it on every incoming video clip or audio segment without the per-call cost dominating your infrastructure budget.

For teams scaling to multi-GPU setups, the vLLM-Omni disaggregated serving guide covers multi-node deployment patterns for omni models.

GPU and VRAM Requirements

The 30B weight count is the primary constraint. Here is how it breaks down by quantization:

BF16 (2 bytes/param): ~60 GB for weights. Add 2-4 GB for Mamba recurrent state (scales with model width, not sequence length) and ~2 GB for KV cache headers. You need an 80GB GPU: H100 SXM5 or A100 80GB.

FP8 (1 byte/param): ~30 GB for weights plus the same Mamba state overhead, leaving roughly 14-16 GB free on a 48GB card for KV cache. The L40S 48GB is viable at FP8 for moderate concurrency.

4-bit GGUF (~0.5 bytes/param): ~15-18 GB. Fits RTX 4090 24GB for development and testing. Not recommended for production serving under concurrent load.

The Mamba state overhead is worth calling out separately: unlike pure-attention models where VRAM cost scales with sequence length, the SSM recurrent state in Mamba layers adds a fixed overhead proportional to model width regardless of context length. Plan for ~2-4 GB of fixed VRAM reserved for Mamba state on top of your weight and KV cache budget.

Pricing fluctuates based on GPU availability. The prices above are based on 29 Jun 2026 and may have changed. Check current GPU pricing → for live rates.

For production multi-user serving, H100 SXM5 80GB on Spheron gives the most headroom at BF16 with per-minute billing and no contract. For FP8 development and testing, L40S 48GB instances are the cost-efficient option. For single-user production at BF16, the A100 80GB is the mid-tier choice.

Step-by-Step Deployment with vLLM

Step 1: Provision a GPU instance

Log into app.spheron.ai and provision your instance. For BF16 production, select H100 SXM5 80GB or A100 80GB. For FP8 development, L40S 48GB works. See docs.spheron.ai/getting-started for SSH setup instructions.

Verify your CUDA version after SSH:

nvidia-smi
nvcc --version

You need CUDA 12.4+ for H100 (Hopper). If you are using B-series Blackwell GPUs, CUDA 12.8+ is required.

Attach persistent storage at /data/models/ before downloading weights. The BF16 checkpoint is roughly 60 GB and you do not want to re-download on every restart.

Step 2: Install vLLM

Nemotron 3 Nano Omni uses hybrid Mamba-Transformer architecture and requires audio input support. The minimum version is 0.20.0, and the [audio] extra is required for audio input modality:

pip install "vllm[audio]==0.20.0"

# Verify installation
python -c "import vllm; print(vllm.__version__)"

Check NVIDIA's model card and vLLM release notes for any Nano Omni-specific notes before deploying, since Conv3D video layers are architecturally uncommon and may require version pinning.

Step 3: Download model weights

For BF16:

huggingface-cli download nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
  --local-dir /data/models/nemotron-3-nano-omni

For FP8:

huggingface-cli download nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --local-dir /data/models/nemotron-3-nano-omni

Always verify the exact HuggingFace repository name against NVIDIA's release page at https://huggingface.co/nvidia before running. The Nemotron family has inconsistent naming conventions across releases.

Step 4: Launch vLLM

BF16 on H100 (production):

vllm serve /data/models/nemotron-3-nano-omni \
  --dtype bfloat16 \
  --no-enable-chunked-prefill \
  --trust-remote-code \
  --port 8000

FP8 on H100 (higher throughput):

vllm serve /data/models/nemotron-3-nano-omni \
  --dtype bfloat16 \
  --quantization fp8 \
  --no-enable-chunked-prefill \
  --trust-remote-code \
  --port 8000

FP8 on L40S (development):

vllm serve /data/models/nemotron-3-nano-omni \
  --dtype bfloat16 \
  --quantization fp8 \
  --no-enable-chunked-prefill \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192 \
  --trust-remote-code \
  --port 8000

The --no-enable-chunked-prefill flag is required for SSM layer correctness. This is the same requirement as Nemotron 3 Super: chunked prefill can produce incorrect outputs on models with Mamba layers. Re-enable only after you have validated correct outputs on your specific workload and confirmed support in your vLLM version.

Step 5: Send multimodal requests

The server exposes an OpenAI-compatible /v1/chat/completions endpoint. Pass multimodal content using the content array format.

Text-only (baseline):

import openai

client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="none")

response = client.chat.completions.create(
    model="/data/models/nemotron-3-nano-omni",
    messages=[{"role": "user", "content": "Describe the steps to train a vision model."}]
)
print(response.choices[0].message.content)

Image input:

import openai
import base64

client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="none")

with open("frame.jpg", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="/data/models/nemotron-3-nano-omni",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
            {"type": "text", "text": "What is happening in this image?"}
        ]
    }]
)

Audio input (base64 WAV):

import openai
import base64

client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="none")

with open("clip.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="/data/models/nemotron-3-nano-omni",
    messages=[{
        "role": "user",
        "content": [
            {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}},
            {"type": "text", "text": "Transcribe and summarize this audio."}
        ]
    }]
)

Audio tokens accumulate at approximately 7 per second (illustrative; verify against your model card), so a 60-second clip generates roughly 420 audio tokens before any text context.

Video input:

For video, sample keyframes at 1-2fps and pass them as an image array, or use the video MIME type if the model card specifies support for it. The Conv3D processing happens inside the model, so the exact input format may differ from ViT-based models. Verify the video input convention in the Nano Omni model card before deploying.

import openai
from your_video_utils import extract_keyframes  # replace with your frame-extraction logic

client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="none")

# Recommended: extract keyframes at 1fps and pass as image array
# See model card for video MIME type support
frames = extract_keyframes("video.mp4", fps=1)
content = [{"type": "text", "text": "Describe what is happening in this video."}]
for frame_b64 in frames:
    content.append({"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{frame_b64}"}})

response = client.chat.completions.create(
    model="/data/models/nemotron-3-nano-omni",
    messages=[{"role": "user", "content": content}]
)

Step 6: Enable chain-of-thought reasoning

Nano Omni has an approximately 16K token reasoning budget. To activate it, pass enable_thinking: True in the extra body (or the equivalent parameter name per NVIDIA's model card):

import openai
client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="none")

response = client.chat.completions.create(
    model="/data/models/nemotron-3-nano-omni",
    messages=[{"role": "user", "content": "Analyze this complex scene: [video input]"}],
    extra_body={"enable_thinking": True}
)

Monitor generation length for reasoning requests: chain-of-thought can produce significantly more tokens than non-reasoning responses. Set --max-model-len to a value that accommodates your expected reasoning trace length plus the multimodal input token count.

Deployment with SGLang

For teams already running SGLang infrastructure, the launch pattern is similar. Check the SGLang production deployment guide for general SGLang setup, and the vLLM vs SGLang comparison if you are choosing between the two for this workload.

SGLang launch for Nemotron 3 Nano Omni:

python -m sglang.launch_server \
  --model-path /data/models/nemotron-3-nano-omni \
  --trust-remote-code \
  --port 8000

Conv3D video layer support in SGLang may require version verification against NVIDIA's compatibility matrix. Check SGLang release notes for Nemotron 3 Nano Omni support before deploying, and fall back to vLLM if Conv3D is not yet supported in your SGLang version.

Building a Multimodal Agent: Nano Omni as the Perception Layer

The two-model architecture separates concerns clearly:

1. Nano Omni (30B/3B active) handles perception: takes raw video, audio, or images and returns structured text
2. Larger reasoning model handles planning: receives Nano Omni's text and produces decisions or actions

import openai
from your_video_utils import extract_keyframes  # replace with your frame-extraction logic

nano_omni = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="none")
reasoning_model = openai.OpenAI(base_url="http://reasoning-node:8001/v1", api_key="none")

def perceive(video_frames_b64: list[str]) -> str:
    content = [{"type": "text", "text": "Describe what is happening in this video in detail."}]
    for frame in video_frames_b64:
        content.append({"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{frame}"}})

    response = nano_omni.chat.completions.create(
        model="/data/models/nemotron-3-nano-omni",
        messages=[{"role": "user", "content": content}]
    )
    return response.choices[0].message.content

def plan(perception_text: str, task: str) -> str:
    response = reasoning_model.chat.completions.create(
        model="/data/models/nemotron-3-ultra",
        messages=[
            {"role": "system", "content": "You are a planning agent. Use perception inputs to make decisions."},
            {"role": "user", "content": f"Scene description: {perception_text}\n\nTask: {task}\n\nWhat action should the agent take?"}
        ]
    )
    return response.choices[0].message.content

# Agent loop
frames = extract_keyframes("scene.mp4", fps=1)
perception = perceive(frames)
action = plan(perception, "Navigate around any obstacles and reach the exit.")

Nemotron 3 Ultra 550B as the reasoning layer and Nemotron Ultra 253B are both strong candidates for the second tier depending on your latency and cost requirements. For setups combining Nano Omni with retrieval-augmented generation, the agentic RAG infrastructure guide covers the storage and indexing side of that architecture.

The cost advantage of this pattern: at 3B active params, Nano Omni perception calls cost a fraction of running the same video through a 70B or 550B model. The perception step handles the expensive multimodal encoding; the reasoning step receives clean, compact text.

Cost Math: On-Demand vs Spot on Spheron

Monthly cost estimates assume 730 hours of continuous use.

Pricing fluctuates based on GPU availability. The prices above are based on 29 Jun 2026 and may have changed. Check current GPU pricing → for live rates.

Break-even against hosted multimodal APIs. Nano Omni's 3B active parameter count drives very high tokens-per-dollar compared to comparable omni models. If you are processing video and audio at scale, the break-even point against pay-per-token multimodal APIs arrives faster than it does for text-only LLMs, because multimodal API pricing typically carries a premium per image frame or audio second.

For a rough estimate: if a hosted multimodal API charges $0.015 per image frame equivalent and you are processing 100 frames per minute continuously, that is $1.50/min or $90/hr. An A100 at $1.43/hr starts paying for itself in less than 1 hour of that throughput rate.

Spot instances. H100 spot pricing drops to $2.91/hr vs $4.06 on-demand. For stateless perception tasks in a multi-agent pipeline, spot is viable: if a spot instance is reclaimed, you redirect perception calls to another instance. Build checkpointing into any stateful parts of your pipeline (the reasoning model session, not the Nano Omni calls) rather than relying on spot instance uptime. Spot can be reclaimed without notice.

Production Notes

NIM microservice option. NVIDIA packages Nemotron models as containerized NIM inference endpoints. For teams who want a managed container rather than raw vLLM setup, see the NVIDIA NIM self-host deployment guide for the containerized path. NIM handles some of the engine configuration automatically.

Batching. Start with --max-num-seqs 8 and increase based on observed cache utilization. Video inputs inflate KV cache proportionally to frame count at the input token level (though Conv3D reduces downstream compute). Monitor vllm:gpu_cache_usage_perc in Prometheus. If it stays above 80% consistently, reduce --max-num-seqs before increasing it.

Video latency. Conv3D preprocessing latency scales with frame count. At 1-2fps sampling for a 30-second clip, you are passing 30-60 frames. Measure total time-to-first-token with nvidia-smi dmon -s mu -d 2 and size your timeout accordingly.

Context management for long sessions. At 300K context with approximately 7 audio tokens per second of audio (verify against your model card) plus video frame tokens, sessions accumulate tokens fast. Implement a sliding window over media tokens while preserving the full text history, or set hard session limits on context length per session. Track token counts in your orchestration layer before they hit the model context limit.

Persistent storage. The BF16 checkpoint is roughly 60 GB. Always mount a persistent volume at /data/models/ so the model survives instance restarts without re-downloading. Use FP8 if storage cost is a constraint: the checkpoint drops to about 30 GB.

Monitoring stack. vLLM exposes Prometheus metrics at /metrics. Key metrics for Nano Omni: vllm:gpu_cache_usage_perc, vllm:num_running_seqs, and vllm:time_to_first_token_seconds. Track num_running_seqs separately for multimodal vs text-only requests if your workload mixes both, since multimodal requests carry more input tokens per request on average.

Setup documentation for Spheron instances is at docs.spheron.ai.

Nemotron 3 Nano Omni's 3B active parameter design makes it one of the most cost-efficient omni-modal models to self-host. Spheron's GPU cloud lets you go from weights on Hugging Face to a running multimodal endpoint without buying dedicated hardware.

H100 SXM5 on Spheron → | L40S for FP8 serving → | View all GPU pricing →

Get started on Spheron →