Ordinary lipsync maps audio to a mouth region. The cheap version is visemes: a handful of mouth shapes (open, closed, wide, round) that approximate groups of sounds. Phoneme-level alignment is finer. Each phone — the /p/ closure, the /s/ slit, the vowel hold — gets a time and a shape. You see it on stops and fricatives. A viseme model will chew them. A phoneme model will hit them. Seedance 2.0 is the catalog example that advertises phoneme-level lip-sync across 8+ languages in the same generate as the picture.
Forced alignment is the clock underneath: a known transcript mapped onto the waveform so each phone has a timestamp. Dubbing that changes language has to re-align, not just re-voice, or the new phones will sit on the old mouth. Native-audio video models that generate speech and picture together can skip the second pass because both came from one sample. A silent generate plus a TTS track plus a lipsync pass is the older pipeline, and it is only as good as the alignment in the middle.
The face still sets the ceiling. Profile, hands on the jaw, motion blur, a microphone in front of the mouth — no phoneme clock will save those. Audio with music bled in will confuse the alignment the same way it confuses a captioner. Trim the take to the speech before you generate. The extra two seconds of room tone at the end is where mouths start to wander.
In practice
- Prefer a frontal, well-lit face with the mouth visible. The phoneme clock cannot track what it cannot see.
- If you dub, re-align. New language, new phones, new mouth shapes — not a time-stretch of the old take.
- Keep music out of the driving audio. Alignment wants speech, not the bed.
The mistake to avoid
Calling any talking generate phoneme-level. If the model only flaps visemes, /p/ and /b/ will look the same and you will see it.
Where you will run into it
- Lipsync a Photo or Video to Audio — One photo, one audio file, one talking clip.
- Dub a Video Into Another Language — Same voice, same face, new language.
- Seedance 2.0 — Seedance 2.0 generates Hollywood-grade cinematic videos from text prompts with native audio-visual synchronization, director-level camera and lighting control, and exceptional motion stability
Related terms
Lipsync
Lipsync meaning: an AI model moves a face's mouth to match an audio track, so the words look spoken by that face instead of dubbed over the top.
Forced alignment
Forced alignment meaning: matching a known transcript to its audio to find exactly when each word was spoken.
AI dubbing
AI dubbing meaning: replacing spoken audio with another language, usually keeping the speaker voice and optionally re-syncing lips.
Native audio
Native audio meaning: a video model that generates soundtrack (dialogue, effects, ambience) in the same pass as the picture.
Text-to-speech
Text-to-speech converts written text into spoken audio using a synthetic voice you choose before generating.
The all-in-one AI studio for creators. 60+ models for video, image, voice, music and lipsync, in your browser or on your phone.
Free on the web and on iPhone, one account.