TL;DR
We're building a script that takes a video in English and produces the same video narrated in Spanish, in a cloned version of the original speaker's voice. Stack: faster-whisper for timestamped transcription, an LLM (or any MT engine) for translation, XTTS-v2 for voice-cloned synthesis, FFmpeg for surgery. We'll also handle the problem every demo skips: translated audio that doesn't fit its time slot.
If you'd rather start from a finished system, Softcatala's open-dubbing and KrillinAI are full pipelines behind one CLI. This post builds the minimal version by hand so you understand what those tools are doing, and where they break.
0. Setup and a licensing warning ⚠️
Python 3.10–3.12. The original Coqui company shut down in early 2024; the maintained fork of their TTS library is published by Idiap as coqui-tts:
