Source
r/u_Interesting-Pen-3023•
Interesting-Pen-3023

I built an AI dubbing service in 1 day using Claude Code — now tackling lip-sync. What's your experience with Wav2Lip/HeyGen?

The Challenge

I wanted to build an AI dubbing service that converts audio/video files into different languages automatically. The pipeline needed:

  • Speech-to-Text (STT)
  • Translation
  • Text-to-Speech (TTS)
  • Video processing with audio sync

Normally, this would take weeks. But with Claude Code, I built the entire MVP in a single day.

The Tech Stack

  • Framework: Next.js 15 (App Router)
  • STT: ElevenLabs Scribe v1
  • Translation: Google Cloud Translation API
  • TTS: ElevenLabs Multilingual v2
  • Video Processing: ffmpeg.wasm (browser-side)
  • Auth: NextAuth.js + Google OAuth
  • Storage: Vercel Blob
  • DB: Turso (for whitelist management)
  • Deployment: Vercel

How Claude Code Made It Fast

1. Architectural Design in Minutes Instead of spending hours planning, I described the requirements and Claude Code designed the entire file structure and API routes in one go. Next.js App Router conventions were followed perfectly.

2. API Integration Pattern Rather than figuring out each API separately, Claude Code created a consistent pattern for integrating ElevenLabs, Google Translate, and NextAuth simultaneously. One coherent system, not scattered code.

3. Complex UI Features

  • Dark glassmorphism design
  • Real-time soundwave animation
  • Side-by-side video player comparison
  • Drag-and-drop file upload

Claude Code generated all of these without me having to design each component from scratch.

4. Browser-Side Video Processing Implementing ffmpeg.wasm to merge video + dubbed audio into MP4 in the browser is non-trivial. Claude Code handled the FFmpeg API calls and WebAssembly context setup seamlessly.

5. Debugging Speed When I hit errors, I just copy-pasted the browser console error directly into the chat. Claude Code analyzed the error and gave me the fix immediately. No long explanations, just working code.

The Experience: Key Takeaways

✅ Error messages are gold — Copy-paste the actual error log, not a description. Claude Code fixes it faster.

✅ Be specific in requests — "Make a file upload" is vague. "Create a /api/dub route that accepts FormData with MP4 files" gets better results.

✅ One feature at a time — Requesting multiple features simultaneously causes conflicts. Sequential requests = cleaner code.

✅ Read the generated code — AI isn't perfect. Always review, especially auth and security-related code.

The Next Challenge: Lip-Sync

Now I'm facing the hard part—lip-sync. The current pipeline syncs audio length to video duration by adjusting playbackRate globally. But true lip-sync requires:

  • Extracting phonemes from the dubbed audio
  • Detecting facial landmarks (mouth position) in the video
  • Re-generating the mouth shape to match phoneme timing
  • Stitching it back into the original video

The Options I'm Considering:

  1. Wav2Lip (Open-source, free)

    • Pros: No API costs, can run locally or in the browser
    • Cons: Slow, requires GPU, might be too heavy for production

  2. HeyGen API (Proprietary)

    • Pros: Fast, production-ready
    • Cons: Costs money per minute
    • Question: What's the actual price per video minute?

  3. Sync Labs (synclabs.ai)

    • Similar to HeyGen, REST API available
    • Free tier limited, paid plan unknown

What I'm Asking the Community

  1. Have you implemented Wav2Lip in production? How's the performance and quality? Is it worth the computational cost?

  2. HeyGen or Sync Labs users? Pricing, quality, API reliability — any gotchas?

  3. Architecture question: Should I process lip-sync after TTS (video + audio file sent together), or integrate it into the pipeline differently?

  4. Claude Code workflow: For someone else wanting to build complex AI pipelines fast, what would you ask Claude Code to do first?

  5. Prompt engineering tips: What prompts worked best for you when working with AI code assistants on video/audio processing?

The full code is open-source: https://github.com/hyosikkk/ai-voice-bot

Would love to hear your experiences and suggestions. Also, if anyone wants to contribute (especially on the lip-sync front), DMs are open!

Demo: https://ai-voice-bot-rose.vercel.app

Posted Mar 14, 2026, 6:33 AM UTC

00
Loading comments…