Captions on lipsync AI video

On lipsync video the mouth is generated from the audio. Captions still follow the audio, not the pixels of the jaw. If those two ever disagree, the viewer believes the mouth. Recaption after a lipsync pass. The previous burn was timed to a take that no longer exists.

Lipsync drives a face's mouth from a track so the speech reads as spoken. Three input shapes exist: audio plus a still, audio plus existing footage (mouth region only), and text plus a face that generates voice and mouth together. Captions do not care which shape you used. They care whether the audio in the file you are captioning is the audio you will publish.

The mouth is generated; the caption clock is the track

Caption timing is alignment to speech in the file. Forced alignment maps known words onto that audio. Auto captions run speech-to-text and alignment together. Neither step looks at visemes. A captioner that "watched" the mouth would still be guessing; the trustworthy clock is the waveform.

That is why a good lipsync can still look wrong under captions. The model painted a plausible mouth. The cue is late, early, or on the wrong word. The viewer does not parse the pipeline. They see a closed mouth under a line, or a line that hangs after the face has moved on, and they file the whole clip as fake.

Two clocks, one picture:

• Audio clock. When the phoneme is in the track. This is what both the lipsync model and the captioner should obey.
• Mouth clock. When the generated jaw actually opens. This is an approximation. Frontal, evenly lit, mouth visible: it usually lands. Profile, shadow, a hand across the chin, a music bed bleeding into the vocal: it smears.

When the mouth clock misses, do not "fix" it by sliding captions off the audio to chase the jaw. You will unsync the one clock the rest of the edit still uses, and you will be wrong again on the next generate. Fix the lipsync input (cleaner audio, a better still, a shorter take) or recaption to the audio you kept.

Word-level captions (karaoke / highlight families) are harsher on this mismatch than a two-line block. A highlight that pops on a closed mouth is a viseme error made visible. If the lipsync is only "good enough," use a static family so the line can be read without pointing at each syllable.

Native audio from a video model is a different path: the clip arrives with speech already in the file. Caption that file as you would any talking-head. The recaption rule still applies if you then run a lipsync pass over it to change the language or the voice. New audio, new captions. Always.

Recaption after every lipsync pass

A lipsync generate is a new performance. Duration, pacing, and even small holds in the audio can shift. The burned-in caption file from the previous take is pixels of the previous take. There is no ripple.

Recaption when any of these happened:

• You generated a new talking clip from a still and a track.
• You replaced the voice (clone, TTS, a reread) and re-ran lipsync.
• You dubbed and lipsynced to the new language. The captioner transcribes; it does not translate. A Spanish mouth with English cues is a new kind of wrong.
• You trimmed the talking clip after the generate. Trim first, then caption. A post-caption trim orphans every cue after the cut.
• You extended or retimed the audio. Phoneme-level models still have to fit a new clock.

You can skip recaption only when the audio did not change and you only swapped a visual that is not the talking plate — a cutaway, a product insert, a background. The speech clock is the same. The cues still match. The moment the mouth plate is a new generate, assume the clock moved.

Order of operations that survives a retake:

1. Lock the script.
2. Lock or generate the audio. Trim silence. Music beds belong under the vocal, not in it — bleed confuses lipsync and transcription.
3. Run lipsync (or the talking-head model that already syncs) on that audio.
4. Watch the mouth on mute, then with sound. If the jaw is the problem, regenerate lipsync. Do not caption a take you are about to throw away.
5. Transcribe and burn captions on the take you will ship.
6. Watch on mute again. The line should land with the audible word, which should land with a plausible mouth.

Skip step 4 and you will caption a reject. Skip step 6 and you will ship a closed mouth under a karaoke pop.

Seedance 2.0 is one of the audio-native rows that does phoneme-level lip-sync in multiple languages in a single generate. That does not retire the caption step. Native sync is still a mouth approximation. Captions are still a typesetting job on the track that came back. If you recut that generate, recaption.

What "off" looks like when mouth and type disagree

Name the miss so you recaption the right layer.

Caption late, mouth on time. Alignment lag, or you captioned a file that still had a pre-roll the lipsync take does not. Offset the cues or recaption the trimmed file. Do not regenerate the face.

Caption on time, mouth late or smeared. Lipsync input problem: profile angle, occlusion, noisy audio, too much head turn. Regenerating captions will make a readable wrong. Fix the generate.

Caption on the wrong word, mouth plausible. Transcription. Product names, numbers, homophones. Read the transcript; recaption with the corrected words. The jaw is fine.

Caption in language A, mouth in language B. You captioned the source tape and lipsynced a dub, or the reverse. Always caption the audio that is in the shipped file. A dubbed cut is a new caption job in the new language.

Highlight dancing on a still jaw. Word-level preset on a take whose visemes are soft. Switch to a static family, or get a cleaner frontal take. Do not add more animation to "sell" the sync.

Cues from the still-image generate sitting on a video-input lipsync. Different takes. The still-plus-audio generate invented a performance; video-input lipsync kept your original body and repainted the mouth. Their clocks are not interchangeable. Caption the file you export.

Placement is a separate miss and it shows up more on talking faces than on B-roll. A lower-third that covers the generated mouth asks the viewer to choose between reading and believing the face. Keep the block off the jaw. The safe area still applies; so does the face.

Do not caption the still, then lipsync

The failure mode that looks like a shortcut: typeset a line on the portrait, then animate the portrait to speak. The overlay is a fixed headline. It will not follow phonemes. The lipsync model does not know it is there and will paint a mouth through it or leave a stale sentence on screen for the whole take.

Authored overlays and transcribed captions are different jobs. A hook you wrote can sit on the lipsync clip as a text overlay after the generate, timed for the first second. Spoken captions belong after the generate, transcribed from the audio you used to drive the mouth.

Photo-input lipsync invents head motion. That motion can carry a burned line out of the safe area or through a product you later composite. Caption last. Product plates and captions both go on the talking file you will publish, not on the still you fed the model.

Video-input lipsync preserves the original body. If that original already had burned captions, they are now pixels under a new mouth, timed to the old audio. Strip or recut from a clean plate, lipsync, then caption. Do not lipsync a captioned master.

The AI caption generator is the burn-in step for the take you kept. The AI lipsync tool is the mouth step. Run them in that order: mouth, then type. The agent job transcribe-and-caption-my-video should see the talking clip, not the portrait.

If Article 50 or a platform AI label applies because the face is a realistic synthetic person, that mark is not a caption cue. Put it at first exposure, then caption the speech as you would any other talking-head.

Mouth first, captions second, same audio. Recaption whenever the take changes.

Caption the lipsync take in Versely.

FAQ

If the audio did not change, can I keep the old captions after a lipsync retake?

Only if you truly exported the same track at the same length and you are sure the model did not pad or trim. When in doubt, recaption. A burned file is cheap next to a closed mouth under a late line.

Should karaoke captions be the default on talking avatars?

No. Word-level highlight makes viseme errors obvious. Use it when the mouth is clean and the line is the performance. Use a static family when the face is only "good enough."

Do I caption before translating a lipsync dub?

Caption after the dub, in the dubbed language. The captioner transcribes the audio in the file. English cues on a French mouth are a miss you cannot offset away.