AI-Native Series · Reverse Engineering
Build Your Own HeyGen From Open Source
1-minute takeaway — what you'll walk away with
HeyGen is a ~$100M-ARR AI-video company. I took it apart module by module and mapped each piece to its best open-source replacement — avatar, lip-sync, voice, dubbing, real-time. Every module has a credible OSS equivalent, but no fully commercial-safe single-repo clone exists, and the best-sounding models are non-commercial. The real moat isn't the model; it's the product layer.
HeyGen is a ~$100M-ARR AI-video company, and its avatars are genuinely good. So I did the obvious founder thing: I took it apart, module by module, and asked — for each piece, what's the best open-source replacement? The answer is encouraging and humbling at the same time.
It's a pipeline, not a magic box
Every "AI avatar video" product is the same six-stage pipeline. Once you see the stages, HeyGen stops being a monolith and becomes a shopping list:
- Avatar synthesis — turn a photo or reference video into a talking head.
- Lip-sync — drive the mouth from audio.
- Voice / TTS — clone or synthesize the voice.
- Translation / dubbing — localize into other languages, voice preserved.
- Real-time streaming — do all of it live, under a second of latency.
- Orchestration — the app that ties it together for a non-technical user.
The module-by-module map
| HeyGen module | Best OSS (commercial-safe) | License | The gap |
|---|---|---|---|
| Avatar synthesis | MuseTalk, EchoMimicV2 | MIT / Apache | full-body & identity consistency |
| Lip-sync | LatentSync, MuseTalk | Apache / MIT | extreme head angles |
| Voice / TTS | GPT-SoVITS, F5-TTS, OpenVoice | MIT | best-sounding models are non-commercial |
| Translation / dubbing | faster-whisper + WhisperX + SoniTranslate | MIT / BSD / Apache | one-click QA vs assemble-it-yourself |
| Real-time streaming | LiveTalking (MuseTalk backend) | Apache | latency, GPU cost, scaling |
| Orchestration (whole) | HeyGem / Duix-Avatar | custom, non-OSI | no permissive single-repo clone |
Star counts and licenses verified against the GitHub API on 2026-07-22. Licenses change; check before you build.
Two traps that will bite you
The license trap. The most-forked, best-sounding models are the ones you can't ship. Wav2Lip, the classic lip-sync baseline, is trained on a research-only dataset. The best voice cloners — fish-speech, Coqui XTTS-v2 — are non-commercial, and Coqui shut down, so there's nobody to buy a license from. A commercial-safe path exists (LatentSync, GPT-SoVITS, F5-TTS), but it's the second-best-sounding path, and you'll spend the money you saved on tuning.
The product trap. You can assemble every model for free and still not have HeyGen. The avatar library, the script editor, the dubbing QA loop, the render queue, the consent and moderation layer, the support org — none of that is open source. That's the real moat.
HeyGen's defensibility isn't any single model — every module now has a credible open-source equivalent. It's integration, realism-at-the-margin, a cheap pay-as-you-go API, and the product layer around the models. If you're building here, don't try to out-model them. Out-product a narrow slice they under-serve.
How I did it — and the honest scoreboard of the tool that did it
I ran this through a reverse-engineering method I've been building (private repo, by design). The discipline is the interesting part: gather only public evidence, tag every claim as fact / inference / speculation, keep a provenance trail, and never let an inference render as a fact. Same method whether the target is a product, a paper, or a market.
Then I graded the tool honestly, because a teardown tool that can't be torn down isn't trustworthy. What worked: it browsed the live site, kept every claim typed and sourced, and enforced its own gates (it refuses to reproduce protected IP without a human sign-off). What broke — and this is the useful part: when I first ran it, it reported 0.96 confidence. Too high. It had been counting my inferences about HeyGen's internals as if they were facts. I fixed that so evidence type flows into the confidence math, re-ran, and it honestly dropped to 0.74. A lower, truer number is the whole point.
The honest gap at the time: the tool was a strong honesty scaffold that didn't yet do the reverse-engineering for you. Since I first drafted this, I've closed most of that gap. It now browses the live target itself, its confidence math respects evidence type by construction, and — the one that matters here — it produces the module-by-module OSS map above by itself, refusing to invent a repo where none exists (it flags the un-cloneable product layer as an honest "unknown" rather than fabricate one). The weakness became a feature, with the same receipts.
What's next
The engine stays private for now. What's done: live browsing, evidence-typed confidence, and native module-to-OSS mapping — the map above is generated, not hand-built. What's left: full model-driven decomposition in the default path, so the tool proposes the module breakdown itself rather than working from one I hand it. The goal is unchanged — make a rigorous, provenance-backed teardown cheap to produce for any target.
If you're deciding build-vs-buy in AI video: buy the realism and the API, and build the narrow product wedge. The models are a commodity now. The product isn't.
AI-Native Series · Paul Jialiang Wu · love12xfuture · Public sources only; not affiliated with HeyGen. Analysis for understanding and interoperability, not for reproducing anyone's protected models. Figures dated 2026-07-22 and move fast.