Deploy NVIDIA NeMo Retriever on GPU Cloud: Enterprise RAG with Multilingual Embeddings and Citation-Aware Reranking (2026 Guide)

A generic TEI deployment gives you fast embeddings. NeMo Retriever gives you fast embeddings with TensorRT-LLM-compiled kernels, a citation-aware reranker, and multilingual support across 26 languages, all in two Docker containers that pass NVIDIA AI Enterprise support entitlements. If your RAG system needs to cite sources at the passage level for compliance, handle non-English documents, or exceed what a PyTorch-backed server can deliver on H100 hardware, NeMo Retriever is the stack to run. This guide covers the full deployment on Spheron bare-metal GPU cloud. For context on deploying other NIM containers, the NVIDIA NIM deployment guide covers the broader NIM catalog including LLM NIMs.

One naming note before diving in: NeMo Retriever is specifically the retrieval NIM family (embedding + reranker). It is distinct from NeMo Curator (data curation), NeMo Guardrails (safety rails), and the broader NeMo training framework. This post covers the retrieval NIM stack only.

What NeMo Retriever Is (and What It Is Not)

NeMo Retriever ships two NIM containers designed to work together:

Embedding NIMs:

• nv-embedqa-e5-v5 for English-only workloads. Faster, smaller, optimized for English BEIR benchmarks.
• llama-3.2-nv-embedqa-1b-v2 for multilingual workloads. 26 languages, up to 2048-dimensional vectors (Matryoshka-configurable to 384/512/768/1024/2048), built on a multilingual Llama checkpoint.

Reranker NIM:

• nv-rerankqa-mistral-4b-v3: a 4B parameter cross-encoder reranker fine-tuned for passage relevance scoring. Returns per-passage attribution data alongside relevance scores.

What NeMo Retriever is NOT: it is not a vector database, not a document parser, and not a full LLM. It sits in the middle of your RAG pipeline, between your document chunker and your vector store on the indexing side, and between your ANN retrieval and your LLM on the query side.

The core difference from a TEI embedding and reranker deployment is the backend. TEI uses Flash Attention 2 and PyTorch. NeMo Retriever uses TensorRT-LLM compiled kernels, which are GPU-architecture specific and compiled at container start time. On an H100 SXM5, that compilation gives you roughly 1.8x embedding throughput over TEI with the same model class. The tradeoff is flexibility: TEI runs any Hugging Face embedding model; NeMo Retriever runs NVIDIA's curated set.

NeMo Retriever Architecture

The pipeline has two phases:

Indexing phase:

Document corpus
  → Chunker / Document Intelligence layer
  → Embedding NIM (nv-embedqa or multilingual NIM) → vector store (Milvus/Qdrant/Weaviate)

Query phase:

Query
  → Embedding NIM → ANN retrieval → top-K candidates
  → Reranker NIM → top-N ranked passages with attribution
  → LLM (any OpenAI-compatible endpoint)

The vector store is the component you deploy separately. NeMo Retriever does not bundle a vector database. For a full deployment guide covering Milvus CAGRA, HNSW tuning, and DiskANN, see the self-host vector database guide.

Hardware Requirements and GPU Sizing

NIM containers use TensorRT-LLM profiles compiled for specific GPU architectures. Running an H100 NIM on an A100 will fall back to a less-optimized profile; it works, but you lose some throughput. Match your GPU to the target deployment tier from the start.

Pricing fluctuates based on GPU availability. The prices above are based on 03 Jun 2026 and may have changed. Check current GPU pricing for live rates.

The reranker NIM (nv-rerankqa-mistral-4b-v3) needs at least 24GB VRAM. On an L40S (48GB), you can colocate both embedding and reranker NIMs on one GPU for dev workloads. For production, separate GPUs prevent the reranker's memory footprint from competing with embedding batch processing.

Deploying NeMo Retriever on Spheron: Step by Step

Prerequisites

• NGC API key from ngc.nvidia.com (NVIDIA Developer Program membership is free and grants evaluation access on up to 16 GPUs)
• NVIDIA Container Toolkit installed on your instance
• Docker and Docker Compose
• A Spheron bare-metal GPU instance: Spheron H100 instances for production, A100 for medium workloads

NVAIE licensing note: NVIDIA AI Enterprise (NVAIE) is required for production use beyond the Developer Program tier. Self-hosting on Spheron is fully supported under NVAIE. The license is per GPU regardless of which cloud provider the GPU runs on.

Docker Compose Configuration

The cleanest way to run both NIMs is with Docker Compose. Assign each NIM to its own GPU using device_ids.

version: "3.9"
services:
  nemo-embedding:
    image: nvcr.io/nim/nvidia/llama-3.2-nv-embedqa-1b-v2:latest
    # Pin to a specific digest for production stability; :latest changes without notice
    runtime: nvidia
    environment:
      - NGC_API_KEY=${NGC_API_KEY}
      - NIM_MAX_BATCH_SIZE=64
    volumes:
      - /nim-cache:/root/.cache/nim
    ports:
      - "8080:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

  nemo-reranker:
    image: nvcr.io/nim/nvidia/nv-rerankqa-mistral-4b-v3:latest
    # Pin to a specific digest for production stability; :latest changes without notice
    runtime: nvidia
    environment:
      - NGC_API_KEY=${NGC_API_KEY}
      - NIM_MAX_BATCH_SIZE=16
    volumes:
      - /nim-cache:/root/.cache/nim
    ports:
      - "8081:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]

Start both containers:

export NGC_API_KEY=<your-key>
docker compose up -d
docker logs nemo-embedding -f  # watch for "NIM ready to serve"
docker logs nemo-reranker -f

First start downloads model weights from NGC (~2-7GB per NIM). With a warm cache volume at /nim-cache, subsequent starts skip the download.

Verification

Test the embedding endpoint:

curl -s -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "nvidia/llama-3.2-nv-embedqa-1b-v2", "input": "What is the GPU memory bandwidth of the H100?"}' \
  | jq '.data[0].embedding | length'
# Expect: 2048

Test the reranker endpoint:

curl -s -X POST http://localhost:8081/v1/ranking \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nv-rerankqa-mistral-4b-v3",
    "query": {"text": "H100 memory bandwidth"},
    "passages": [
      {"text": "The NVIDIA H100 SXM5 delivers 3.35 TB/s HBM3 bandwidth."},
      {"text": "GPU VRAM determines model capacity."}
    ]
  }' | jq '.rankings'
# Expect: array with index and logit score per passage

Helm / Kubernetes Deployment

For Kubernetes, NVIDIA publishes Helm charts on NGC at https://helm.ngc.nvidia.com/nvidia/charts/. Pull the NIM chart for the embedding model:

helm fetch https://helm.ngc.nvidia.com/nvidia/charts/nim-llm-1.0.0.tgz

Override values.yaml to set your GPU resource request and NGC API key secret:

resources:
  requests:
    nvidia.com/gpu: 1
  limits:
    nvidia.com/gpu: 1

ngcAPIKey:
  secretName: ngc-api-key-secret
  secretKey: NGC_API_KEY

For the full Kubernetes GPU setup including cluster networking and node selectors, see the Kubernetes GPU orchestration guide.

Multilingual Embedding Pipelines

The llama-3.2-nv-embedqa-1b-v2 NIM is the multilingual variant. It handles 26 languages (English, Arabic, Bengali, Chinese, Czech, Danish, Dutch, Finnish, French, German, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Norwegian, Persian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, and Turkish) and supports cross-lingual retrieval: you can index a French corpus and query it in English without any translation layer. The English-only nv-embedqa-e5-v5 is approximately 20% faster but only handles English text.

When to use each:

• English-only corpus, latency-sensitive serving: use nv-embedqa-e5-v5
• Any non-English documents or mixed-language corpus: use llama-3.2-nv-embedqa-1b-v2

Testing multilingual embedding and verifying the vector dimensions:

import requests

resp = requests.post(
    "http://localhost:8080/v1/embeddings",
    json={"model": "nvidia/llama-3.2-nv-embedqa-1b-v2", "input": "¿Qué es el aprendizaje automático?"}
)
embedding = resp.json()["data"][0]["embedding"]
print(f"Dimensions: {len(embedding)}")  # expect 2048 (default; Matryoshka truncation configurable to 384/512/768/1024)

The two embedding models produce different vector dimensions: nv-embedqa-e5-v5 outputs 1024-dimensional vectors; llama-3.2-nv-embedqa-1b-v2 outputs up to 2048-dimensional vectors by default (Matryoshka-configurable to 384, 512, 768, or 1024). Both models also support Matryoshka truncation to smaller sizes when index footprint matters. Higher dimensionality improves retrieval quality on complex technical passages at the cost of a larger vector store index.

For a broader look at embedding model selection across vision and text modalities, the multimodal embedding guide covers SigLIP-2, JinaCLIP-v2, and Cohere Embed-v4.

Citation-Aware Reranking: Why It Matters for Enterprise

Standard cross-encoder rerankers score (query, passage) pairs and return a relevance ranking. That is useful but not sufficient for regulated industries.

Citation-aware reranking adds passage-level source attribution to the relevance scores. Each ranked result from the NeMo reranker NIM includes a document.index (matching back to your original passage array) and a logit relevance score. When you store the original document ID and passage byte offset alongside each chunk before indexing, the reranker output becomes a fully traceable citation chain: from query to passage to source document.

This matters for HIPAA, EU AI Act Article 13, and financial services compliance requirements where the LLM's output must be traceable to a specific, citable passage in a source document.

Contrast this with common alternatives:

• ms-marco-MiniLM cross-encoders: no built-in attribution, document-level only
• Cohere Rerank API: black-box, no self-hosted option, no passage-level offsets
• Custom cross-encoders via TEI: passage attribution requires custom pipeline wiring

The NeMo reranker NIM handles this out of the box.

Example reranker API call:

import requests

query = "What is the GPU memory bandwidth of the H100?"
passages = [
    "The NVIDIA H100 SXM5 delivers 3.35 TB/s of HBM3 memory bandwidth.",
    "GPU memory bandwidth is critical for embedding throughput.",
    "VRAM capacity determines how large a model you can load."
]

resp = requests.post(
    "http://localhost:8081/v1/ranking",
    json={
        "model": "nvidia/nv-rerankqa-mistral-4b-v3",
        "query": {"text": query},
        "passages": [{"text": p} for p in passages]
    }
)
rankings = resp.json()["rankings"]
for r in rankings:
    print(f"Passage {r['index']}: score {r['logit']:.3f}")

The index field maps back to your original passages array, so you can reconstruct the full source reference from your indexing metadata.

Vector Store Integration

NeMo Retriever is vector-store-agnostic. The embedding NIM exposes an OpenAI-compatible /v1/embeddings endpoint, so any client that supports OpenAI embeddings works with it.

For a full vector store deployment guide covering Milvus CAGRA, HNSW tuning, and DiskANN:

For Milvus CAGRA configuration, Qdrant payload filtering, and Weaviate hybrid search setup, see the self-host vector database guide.

Throughput and Latency Benchmarks: NeMo Retriever vs TEI vs vLLM Embedding

These figures are based on publicly available NVIDIA NIM benchmark data and community measurements. Conditions: single GPU, 512-token passages, batch size 64 for embedding, batch size 16 for reranking. Your numbers will vary by model, chunk size, and batch configuration.

The TRT-LLM throughput advantage comes from kernel compilation at container start, not runtime overhead. The first launch compiles the kernels for your specific GPU architecture; every subsequent request uses the pre-compiled path. For more on inference optimization approaches, see the vLLM production deployment guide and the LLM serving optimization guide.

Cost Comparison: Spheron vs NVIDIA NIM API vs Managed Embedding APIs

The table below uses live pricing from the Spheron API fetched on 03 Jun 2026. Monthly GPU cost is ($/hr) × 24 × 30. Per-1M-token cost is derived at 50% GPU utilization.

Baseline: 1B tokens/month, continuous serving

The per-token economics favor self-hosting at high utilization. A well-utilized A100 at $0.010/1M tokens undercuts every managed API in the table above. The catch is the fixed monthly base: the A100 costs $1,066/month whether you run 1M or 100B tokens through it. The break-even against Cohere ($0.12/1M) is around 9B tokens/month, and against OpenAI ($0.130/1M) around 9B tokens/month. Below those thresholds, managed APIs are cheaper. Above them, self-hosting on bare-metal GPU hardware wins on cost and keeps your embeddings off third-party APIs.

One factor that matters specifically for NeMo Retriever: embedding throughput is bandwidth-bound. Shared or virtualized GPU instances partition PCIe bandwidth across tenants, reducing throughput by 10-30% compared to dedicated hardware. Spheron's bare-metal instances give the embedding NIM dedicated HBM bandwidth, which is what the TRT-LLM kernels are compiled to exploit.

Pricing fluctuates based on GPU availability. The prices above are based on 03 Jun 2026 and may have changed. Check current GPU pricing for live rates.

Production Checklist

1. GPU memory tuning: set NIM_MAX_BATCH_SIZE based on your chunk size. For 512-token chunks, 64 is a safe starting point for H100. Increase until GPU memory hits 85% under peak load.
2. Batch sizing for indexing vs serving: use async batch embedding with batch sizes of 128-256 for offline corpus indexing. Use smaller batches (16-32) for online query embedding to keep p99 latency under control.
3. NGC cache volume: mount /root/.cache/nim to a persistent volume or fast NVMe disk. Model weights are 2-7GB per NIM; cold starts without cache take 2-10 minutes.
4. Prometheus observability: each NIM container exposes /metrics on the same port. Scrape it with Prometheus and alert on nim_request_duration_p99 and nim_batch_size_avg.
5. Health checks: add a Docker healthcheck for the /v1/health/ready endpoint before routing traffic. Both NIMs expose this endpoint.
6. Reranker pair limits: the reranker NIM accepts up to 256 passage pairs per request. For retrieval pipelines returning more than 256 candidates, paginate the reranking call across multiple requests.

For production LLM inference latency budgeting (TTFT and ITL targets for the generation layer), see the LLM inference SLO and latency guide. Embedding and reranking latency tuning is NIM-specific and covered in the production checklist above. To add safety rails to the LLM layer sitting downstream of the reranker, the NeMo Guardrails guide covers the guardrails NIM stack.

NeMo Retriever needs direct GPU memory bandwidth. Bare-metal instances prevent the PCIe sharing that cuts embedding throughput on virtualized GPUs. Spheron's H100 and A100 bare-metal instances let you run the full NIM stack at full hardware speed.

H100 GPU pricing → | A100 on Spheron → | View all GPU pricing →

Deploy NeMo Retriever on Spheron →