Self-Host AI Dubbing on GPU Cloud: Open-Source Localization Pipeline (2026)

AI dubbing is not one model. It's four models chained together: automatic speech recognition transcribes the source audio, machine translation converts the script, text-to-speech synthesizes a new voice track, and a lip-sync model redraws the mouth to match. If you want to self-host AI dubbing instead of paying per-minute SaaS rates, you have to size and deploy all four stages, and the fourth one runs on different hardware than the first three.

That last part trips up most first attempts at this. Teams provision an H100 because it's the biggest card on the price list, get through ASR, translation, and TTS without issue, then hit a wall on lip-sync because the GPU they picked doesn't have the video encode/decode hardware the sync model needs. This post covers the full pipeline: which open-source components replace ElevenLabs, Rask AI, and HeyGen at each stage, how to size GPUs for each one (and why they're not the same GPU), and what it actually costs per minute of localized video once you're running it yourself.

For contrast with a different voice AI architecture, dubbing is asynchronous and video-bound: you're not chasing sub-300ms turn-taking latency the way a real-time speech-to-speech agent has to. You're processing a fixed video file through four sequential stages and can batch as many languages as you want in parallel. That changes every sizing decision that follows.

The AI Dubbing Pipeline: ASR, Machine Translation, TTS, and Lip-Sync Explained

Dubbing takes a source video, its spoken audio, and a target language, and produces a new video where the audio (and ideally the mouth movements) match the target language instead of the original. Getting there means running the source through four distinct stages, each with its own model, license, and hardware profile.

Why Dubbing Is a Different Workflow Than Voice Cloning or Speech-to-Speech

Voice cloning is a single model: text and a short reference clip go in, audio in that speaker's voice comes out. It's one of dubbing's four stages, not a replacement for the pipeline. Speech-to-speech models like Moshi or Sesame CSM are built for live, synchronous conversation, audio in and audio out with no text bottleneck and no video involved at all.

Dubbing sits somewhere else entirely. It's batch, not real-time. It's video-bound, meaning the output has to line up with existing footage frame by frame if you want lip-sync. And it's a pipeline, not a model: transcription errors propagate into translation errors, translation length changes break the timing you'll try to sync TTS output against, and TTS pacing has to match the original speaker's cadence closely enough that a lip-sync model can plausibly redraw the mouth. Each stage's output quality bounds the next stage's ceiling.

The Four-Stage Pipeline: Transcribe, Translate, Synthesize, Sync

The pipeline runs in a fixed order:

1. Transcribe. ASR converts source audio to timestamped text, ideally with speaker diarization so multi-speaker content keeps voices separated.
2. Translate. Machine translation converts the transcript into the target language, sentence by sentence, preserving enough timing information to keep the dub roughly synced to the original's pacing.
3. Synthesize. TTS (often with voice cloning) generates new audio in the target language, matching the original speaker's voice and approximate delivery speed.
4. Sync. A lip-sync model takes the new audio and the source video and regenerates the mouth region frame by frame so the visual matches the new speech.

Steps 1 through 3 are audio-only and can run on almost any modern GPU. Step 4 is video generation and has a hardware requirement the first three don't share, which is the single most important sizing fact in this whole pipeline and gets its own section below.

Open-Source Components That Replace Paid Dubbing Vendors

Every stage above has open-weight options that run on rented GPUs instead of a per-minute API. Here's what's production-viable at each stage as of mid-2026.

ASR: faster-whisper and WhisperX for Transcription, Timing, and Diarization

faster-whisper (a CTranslate2 reimplementation of OpenAI Whisper) is the standard choice: 4x faster than stock Whisper at the same accuracy, with INT8 quantization support. WhisperX builds on it and adds word-level timestamp alignment via wav2vec2 and speaker diarization via pyannote-audio, both of which matter for dubbing since you need accurate per-line timing to know how much room the TTS stage has to fill, and diarization to keep multi-speaker scenes assigned to the right voice.

Combined VRAM for Whisper Large v3 plus wav2vec2 alignment plus pyannote diarization runs 5-6GB at FP16, comfortably inside any of the GPUs discussed below. On an L40S, faster-whisper processes audio at roughly 25-30x real time, so a 10-minute source clip transcribes in under 25 seconds. For the full deployment walkthrough, including streaming setup and a FastAPI reference server, see the Whisper v4 ASR production guide.

Machine Translation: NLLB-200 vs SeamlessM4T v2 (and the NLLB License Trap)

Meta's NLLB-200-distilled-600M is the most-downloaded open translation model on Hugging Face, supporting 200+ languages including many with little other open MT coverage. It's also, at 600M parameters and roughly 2.5GB, cheap to run: a few seconds of GPU time per minute of dialogue text on any card in this pipeline.

The catch is the license. NLLB-200 models ship under CC-BY-NC 4.0, which explicitly prohibits commercial deployment. If you're dubbing content for a commercial product, that rules NLLB out unless you secure a separate license from Meta.

Meta's SeamlessM4T v2 is the alternative worth knowing about. It's an open foundation model that handles automatic speech recognition, text-to-text, speech-to-text, text-to-speech, and direct speech-to-speech translation across roughly 100 languages, all in one architecture. It can, in principle, collapse the translate-and-synthesize steps into one hop for cross-lingual speech, though most production dubbing pipelines still keep the stages separate for QA and text-based review. SeamlessM4T v2 also ships under a non-commercial license, so check the current terms on whichever checkpoint you pull before committing a production pipeline to either model; license terms on both projects have shifted before and can shift again.

TTS and Voice Cloning: XTTS-2 and F5-TTS for the Dubbed Voice Track

The synthesis stage is where dubbing and voice cloning overlap directly. XTTS-2 (Coqui) supports cross-lingual cloning: clone a speaker from a few seconds of the original audio, then generate that same voice speaking the translated script in one of 17 supported languages. That's exactly the shape of the dubbing TTS problem. F5-TTS is the flow-matching alternative, better suited to longer passages since it avoids the error accumulation that autoregressive models like XTTS-2 can show on long text, though its language coverage (English and Chinese) is narrower.

Both models run comfortably on the same GPU tier as ASR and MT: 3-4GB VRAM for XTTS-2, 2-3GB for F5-TTS at FP16. The full model comparison, license details (XTTS-2 is CPML, not permissive for unrestricted commercial use without review), and a production FastAPI deployment walkthrough are in the self-hosted voice cloning guide. If your dub doesn't need to preserve the original speaker's exact voice, non-cloning TTS models like Kokoro or Fish Speech are lighter options, covered in the open-source TTS deployment guide.

Lip-Sync: Wav2Lip, MuseTalk, LatentSync, and NVIDIA's LipSync NIM

This is the stage that looks like the others on a diagram and behaves completely differently on hardware.

Wav2Lip is the oldest and lightest option, syncing lip movement to arbitrary target audio. It's restricted to personal, research, and non-commercial use in its open release; commercial use requires contacting the developers directly, and its LRS2-trained weights carry their own dataset licensing.

MuseTalk (Tencent) generates 256x256 lip-synced face regions at 30+ FPS on a single NVIDIA Tesla V100, using a UNet borrowed from Stable Diffusion v1.4 with a frozen Whisper-tiny audio encoder for conditioning. Code is MIT licensed and the trained weights are usable commercially, though dependencies (Whisper, the VAE, pose and face-detection models) carry their own licenses you need to comply with separately.

LatentSync (ByteDance) takes a different architectural approach: "audio-conditioned latent diffusion models without any intermediate motion representation," per the project's own description, meaning it skips the explicit facial-landmark or motion-warping step other methods rely on. LatentSync 1.6 trains at 512x512 to fix the blurriness earlier versions showed, and needs a minimum of 18GB VRAM; the earlier 1.5 checkpoint needs 8GB. It's Apache 2.0 licensed, the least restrictive of the pure lip-sync models here.

NVIDIA's LipSync NIM is the managed-container option, part of the Content Localization Blueprint NVIDIA released in May 2026, a modular reference architecture aimed at broadcast, OTT, and cloud media pipelines that also covers speech-to-speech translation and Active Speaker Detection with diarization. It integrates with speech-to-speech providers including NVIDIA Riva, CAMB.AI, and ElevenLabs. The important detail for GPU sizing is in NVIDIA's own support matrix: NIM Lipsync officially supports T4, A2, A10, A16, A40, L4, L40, L40S, B40, and the consumer RTX 4090, 5090, and 5080, and explicitly does not support A100, H100, or B100.

Full-Pipeline Projects: Open-Dubbing, Linly-Dubbing, and VideoLingo

Three open-source projects chain some or all of these stages together so you're not wiring four separate repos by hand:

Open Dubbing is the cleanest starting point for stages 1-3. It also runs Demucs for vocal separation and pyannote-audio for diarization before transcription, which improves ASR accuracy on content with background music. It's a single CLI command:

pip install open_dubbing
open-dubbing --input_file video.mp4 --target_language=cat --hugging_face_token=TOKEN

Source language is auto-detected from the first 30 seconds unless you set --source_language explicitly. Open Dubbing doesn't include a lip-sync stage, so you chain a sync model on afterward if you need visual matching, not just a new audio track.

Linly-Dubbing is the only one of the three that includes lip-sync out of the box, alongside UVR5-based vocal separation and GPT-SoVITS fine-tuning for voice naturalness, making it the closest thing to a true four-stage pipeline in one repo.

VideoLingo doesn't attempt frame-level lip-sync at all. Instead it invests heavily in a three-step "translate, reflect, adapt" LLM translation pass and speech-rate engineering to keep dubbed audio close to the original's pacing, useful when you're optimizing for translation quality and timing over visual mouth matching.

For teams building this themselves rather than starting from one of these repos, the voice AI GPU infrastructure guide covers the general pattern of co-locating multiple model stages on shared hardware, which is exactly the problem the next section works through for dubbing specifically.

GPU Sizing for Batch Localization Across Multiple Languages

The four-stage pipeline splits cleanly into two hardware groups: ASR, MT, and TTS need VRAM and general tensor throughput; lip-sync needs a GPU with NVENC/NVDEC hardware on top of that. Sizing wrong in either direction either wastes budget on a card that can't run your sync stage, or under-provisions VRAM for concurrent stages.

VRAM Footprint Per Pipeline Stage

Stack a full ASR + MT + TTS chain and you're looking at roughly 10-13GB depending on which translation and TTS models you picked, well within any 24GB+ card. Add a lip-sync pass and, if you're running LatentSync 1.6 for its sharper 512x512 output, you need at least 18GB dedicated to that stage alone.

One GPU vs a Pipeline of GPUs: Co-Locating ASR, MT, TTS, and Lip-Sync

The practical question is whether one GPU can run the whole chain, or whether you need to split stages across different instance types. The answer depends entirely on which card you pick.

An L40S on Spheron is the one card that clears every requirement at once: 48GB of VRAM, comfortably enough for ASR, MT, and TTS running concurrently plus a LatentSync 1.6 pass, and it's on NVIDIA's own NIM Lipsync support list. That means a single L40S instance can host all four stages without you needing to stand up and coordinate two separate GPU tiers.

An H100, by contrast, handles the first three stages faster than an L40S ever will, but can't run the fourth at all. If your organization already has H100 capacity reserved for LLM or training work and wants to route ASR/MT/TTS through it opportunistically, that's a reasonable split-architecture design: heavy compute on H100 for the audio stages, a separate L40S or RTX 4090 worker pool dedicated to lip-sync. For most teams starting from zero, though, a single L40S per pipeline worker is simpler to operate and cheaper to provision than juggling two GPU pools for one job.

Why Lip-Sync Models Need NVENC/NVDEC GPUs, Not H100 or A100

This is worth restating plainly because it's the single most common mis-provisioning mistake in this pipeline: NVENC and NVDEC are dedicated hardware video encoders and decoders present on NVIDIA's professional and consumer GPU lines (Blackwell, Ada, Ampere, and Turing architectures), but absent from the data-center A100, H100, and B100 parts, which trade that silicon for more tensor cores and memory bandwidth instead. NVIDIA's own LipSync NIM documentation states this directly: GPUs without NVENC/NVDEC hardware are not supported.

Open-weight lip-sync models like Wav2Lip, MuseTalk, and LatentSync don't have that same hard vendor exclusion baked into their code, but the underlying constraint is the same: lip-sync is video generation, and encoding/decoding video frame-by-frame at scale is dramatically faster with dedicated hardware than software fallback paths. There's no upside to running the sync stage on an A100 or H100 even where it's technically possible; you're paying data-center pricing for a workload that runs just as well, or better, on an L40S, L4, or RTX 4090.

Batch Throughput: Localizing One Video Into 10+ Languages in Parallel

The pipeline's real economics show up once you localize into more than one language, because the four stages don't scale the same way. ASR runs once per source video, regardless of how many target languages you're producing. Translation, synthesis, and lip-sync all run once per target language.

That means the fan-out point is stage 2, not stage 1. A 10-minute source video transcribed once costs the same whether you're dubbing it into 2 languages or 20. Here's the math for a representative job, assuming an L40S running the whole chain:

• Transcription (one-time): at ~25x real time, a 10-minute source clip transcribes in about 24 seconds of GPU time, regardless of target language count.
• Translation (per language): NLLB-200 or SeamlessM4T v2 processes a script this length in low single-digit seconds of GPU time per language.
• Synthesis (per language): at XTTS-2's roughly 420 characters/second on L40S, a 10-minute dub script (around 7,500 characters at typical speaking pace) synthesizes in about 18 seconds per language.
• Lip-sync (per language): MuseTalk's 30+ FPS generation rate on V100-class hardware means a 10-minute, 25fps video (15,000 frames) takes roughly 500 seconds of GPU time per language on an L40S, which is faster than V100 for this workload, so that's a conservative upper bound.

Run 10 target languages on one L40S worker sequentially, and lip-sync dominates total wall-clock time by a wide margin, roughly 80-90% of the job. That's the stage worth parallelizing across multiple GPU workers if turnaround time matters more than raw cost, since ASR and translation are cheap enough in absolute terms that running them redundantly per worker barely moves the bill.

Cost Per Minute of Localized Video: Self-Hosted vs SaaS Dubbing Tools

The AI dubbing tools market is projected to reach $2.56 billion by 2030 at a 17.3% CAGR, according to one industry estimate; a separate report puts it at $1.88 billion by 2030 growing at 14.2% annually. The two firms disagree by close to $700 million on where this lands, which mostly tells you the category is still young enough that nobody has a settled number, and it's growing either way. Pricing in that market reflects the immaturity: per-minute rates vary widely by vendor and by whether lip-sync is switched on.

What ElevenLabs, Rask AI, and HeyGen Charge Per Dubbed Minute

SaaS AI dubbing in 2026 runs roughly $0.20 to $2.40 or more per dubbed minute depending on the platform, plan tier, and whether lip-sync is switched on, and enabling lip-sync is consistently one of the biggest levers on that price. HeyGen's own developer pricing shows the split clearly: audio-only dubbing and standard-tier video translation with lip-sync both bill at $0.0333 per second (about $2/min) via the API, but the higher-precision lip-sync mode doubles that to roughly $4/min.

Pricing fluctuates based on GPU availability. The prices above are current as of 02 Aug 2026 and may have changed. Check current GPU pricing for live rates on Spheron.

Self-Hosted Cost Per Minute on Spheron GPU Cloud

Combining the throughput assumptions from the batch section above with live L40S pricing puts self-hosted marginal compute cost, per target language, once a source video's one-time transcription pass is done, at roughly the following:

The estimate assumes NLLB-200-class translation (a few seconds of GPU time per language), XTTS-2 synthesis (~420 chars/sec), and MuseTalk-class lip-sync at roughly 1x real time as a conservative bound on L40S-class hardware. The RTX 4090 row applies that same conservative per-stage time budget for simplicity; a 4090 will typically run somewhat slower than an L40S on the lip-sync stage in practice, but its lower hourly rate keeps the per-minute figure in the same ballpark either way. That works out to somewhere around a penny or two per minute of localized video per target language, against $0.20 to $2.40+ per minute on SaaS platforms.

That gap holds up even accounting for the platform's operational realities. Spheron's instance types documentation notes a 20-minute minimum runtime per instance and that spot instances, typically 30-60% cheaper than dedicated, aren't meant for anything without checkpointing since they can be reclaimed at any time. For a batch dubbing job with dozens of languages queued, that's a non-issue: spot is exactly the right fit for a fault-tolerant, restartable batch workload like this. For a single one-off 3-minute clip into one language, you're still billed the 20-minute floor, on an RTX 4090 on-demand that's about $0.18 total, which is still far below what any SaaS vendor would charge for the same job.

Break-Even Volume and When a SaaS Dubbing Vendor Still Makes Sense

The raw compute math favors self-hosting at almost any real volume; a pipeline running a handful of languages a week pays for the engineering time to build it within the first few batches. The variable that actually decides this isn't GPU cost, it's operational overhead. A four-stage chained pipeline with translation QA, voice-quality review, and lip-sync artifact checking takes real engineering time to build and keep healthy, in a way a single voice-cloning or TTS endpoint doesn't.

SaaS dubbing still earns its markup when:

• Volume is genuinely occasional. A handful of videos a month into one or two languages doesn't justify building and maintaining a pipeline, regardless of the per-minute math.
• You need a contractual uptime or quality SLA. Self-hosted GPU infrastructure carries spot preemption risk and no vendor-backed accuracy guarantee on translation or lip-sync output.
• Your team has no GPU operations experience. A pipeline that silently drops diarization on a multi-speaker scene, or ships an unsynced mouth, costs more in brand damage than the SaaS bill would have.
• You need built-in review and approval workflows. Managed dubbing platforms increasingly bundle human-in-the-loop QA steps that a raw self-hosted pipeline doesn't include unless you build it yourself.

For teams localizing a recurring content catalog, a weekly episodic show going out in 8-12 languages, a product demo library that needs refreshing every release, an internal training archive serving a global team, the math tips decisively toward self-hosting once you're past the first month of amortized engineering time.

If you're building a dubbing pipeline that chains ASR, translation, TTS, and lip-sync, an L40S gives you enough VRAM to run all four stages on one instance and the NVENC/NVDEC hardware the sync stage needs.

Spheron L40S instances →