Caption Accuracy Targets and a Spot-Check
Looks fine is not a rate. Set a word-error-rate target, count punctuation and speaker misses, and sample a 200-clip library instead of reading every file.
"Looks fine" is a reaction to the first thirty seconds. It is not a rate, it does not count speaker errors, and it does not scale to two hundred files. A caption track can be word-perfect and still fail readability. It can also read cleanly and still be wrong: a missing "not," a speaker label stuck to the other person, a dollar figure that ASR guessed. Those are different defects. They need different targets, and they need a sampling method so you are not sitting through the whole library.
Word error rate is the word layer only
Word error rate (WER) is the standard ASR metric: the number of substitutions, deletions, and insertions needed to turn the hypothesis into a reference transcript, divided by the number of words in the reference.
WER = (S + D + I) / N
S is substitutions, D deletions, I insertions, N the reference word count. A clip with 200 reference words, 6 substitutions, 2 deletions and 2 insertions is 10 / 200 = 0.05, or 5% WER, which is 95% word accuracy if you prefer that framing. Do the arithmetic on the clip, not in your head: S, D and I are added, not averaged, and N is the reference length. Inflating N with filler you did not actually transcribe will make any engine look better than it is.
What WER will not see:
- Punctuation and capitalization. "Let's eat, grandma" and "let's eat grandma" can be WER 0.
- Speaker identity. WCAG's definition of captions includes who is speaking. A two-hander with no labels can be WER 0.
- Non-speech audio. Music, laughter, an alarm. Same criterion, same gap.
- Homophones that are spelled as a different real word. Some of those do show up as substitutions; "there" for "their" does. A number written as words instead of digits often does not, depending on how you normalise the reference.
- Reading speed, line breaks, and contrast. That is the readability layer, not this one.
So WER is necessary and not sufficient. Use it as the word-layer target. Keep three extra tallies next to it: punctuation-or-meaning, speaker, and non-speech. A file can pass WER and still fail the caption's actual job.
What the law actually requires (and does not)
47 CFR § 79.1(j)(2) is the FCC's caption quality rule for televised video programming. It is qualitative. Captions must be accurate, synchronous, complete, and appropriately placed. Accuracy includes matching spoken words and lyrics in order, without paraphrasing except to resolve time constraints, plus spelling, punctuation, capitalization, speaker identity, music, and sound effects. The 2014 order that adopted those standards (FCC 14-12) described them as non-quantitative. There is no WER number in § 79.1.
Vendor pages that say "the FCC requires 99%" are collapsing an internal target onto a rule that deliberately did not pick one. Do not put 99% in a client report as if it were the regulation. Put it in, if you use it, as your word-layer bar.
WCAG 2.2 SC 1.2.2 (Level A) is also non-numeric. Captions are required for prerecorded synchronized media. They include dialogue, speaker identity, and meaningful non-speech sound. Failure F8 is captions that omit some dialogue or important sound effects. Passing WER and failing F8 is a documented way to fail the criterion.
For a working internal bar on prerecorded brand and training video, use this unless a buyer has specified something tighter:
| Layer | Target | How you score it |
|---|---|---|
| Words | WER at or under 0.05 on the sampled files, against a human reference | (S+D+I)/N, numbers and proper nouns counted as words |
| Meaning punctuation | Zero errors that reverse or hide a claim (negation, decimal, unit) | Tally separately. A comma error that does not change meaning is logged, not failed |
| Speaker | Every speaker change labelled, or visually obvious in a one-person piece | Missed or swapped label = fail the file, even at WER 0 |
| Non-speech | Every sound that carries plot, warning, or reaction has a cue | Missing laugh / alarm / music-that-matters = fail |
| Completeness | First audible speech to last audible speech | Gap at either end = fail |
| Sync | Cue in the same breath as the word, readable on screen | A file that is uniformly late by a constant offset is a shift, not a recaption. Progressive drift is a clock problem |
Live and near-live stay on a different product, with a different check. Do not apply this table to a stream that has not ended.
Spot-check a 200-clip library without watching 200 clips
Full-file WER on every clip is how this work never ships. Sampling is how it does. The point is not a confidence interval. The point is to stop pretending the ten nicest talking heads represent the month.
1. Bucket the library before you draw. Five buckets, not one random list. A 200-clip month will not split evenly, and that is the point: you want risk, not equality.
| Bucket | What goes in it | Why it is a bucket |
|---|---|---|
| A | One speaker, quiet room, no bed | The easy case. If this fails, the pipeline is broken |
| B | Music under the voice | ASR substitutions cluster here |
| C | Two or more speakers, or a handoff | Speaker-label failures live here |
| D | Product names, SKUs, prices, version numbers | Proper-noun and figure failures |
| E | Anything else: noisy location, heavy accent plus fast speech, non-English, very long (>8 min) | Drift and completeness |
If a clip belongs in two buckets, put it in the harder one. D beats B. C beats A.
2. Draw 20 full-file WER passes (10% of 200). Four from each bucket if the buckets can support it. If bucket E only has seven files, take all seven and move the leftover draws into D and C, not into A. You are sampling risk.
3. Draw 20 more window checks, not full files. One 30-second window per extra file, taken at about 50% of duration (or at 75% if the file is longer than eight minutes). This is the cheap screen for the drift that shows up after the first two minutes. A window is not a WER of the file. It is a tripwire. If the window is dirty, promote that file to a full-file pass and recaption it.
4. Always add the last five files you shipped, regardless of the draw. Recency bias in production is real: the week you changed a preset, a model, or a ducking level is the week silent failures land. Those five sit on top of the 20 + 20, they do not replace them. 20 full + 20 windows + 5 recency = 45 sits, not 200.
5. Fail the bucket, not just the file. If two of the four full-file passes in bucket D miss prices, you do not "fix those two." You recaption the bucket, or you change the input (a glossary of SKUs, a pre-duck of the bed) and redraw. A single miss in A is a file problem. Repeated misses in D are a process problem.
How to take the measurement
You need a reference. Auto-transcript against auto-transcript is a circle.
- Generate a working transcript from the audio with speech-to-text via get a video transcript. That is the hypothesis, not the reference.
- Produce the reference by correcting that text against the audio, or from the script if the script is what was actually said. Scripts that were rewritten on the day are not references.
- Normalise both sides before you count: expand or collapse numbers the same way, drop filled pauses you have decided not to caption, keep every negation and every proper noun.
- Count S, D and I. Write the three numbers down. Then compute WER. Then tally punctuation-or-meaning, speaker, and non-speech on the same pass.
If the hypothesis is already burned in, you are scoring pixels against audio, which is slower. Prefer scoring the text file, then burning. Versely's burn-in path (add captions to a video, or the caption generator) is the styled output. The QC target sits on the transcript that feeds it.
A uniform timing miss (every cue 1.2 seconds late after a trimmed intro) is not a WER miss. Shift the file. The in-browser subtitle timing shifter moves every cue by a constant offset on your machine and does not call a model. Progressive drift is not a shift. That is alignment or frame-rate, and forced alignment against the corrected reference is the actual fix.
Finish the sit by opening two frames against the picture, not just the text: one mid-sentence on a busy shot, and the last spoken line. Completeness and placement do not show up in a spreadsheet of S, D and I.
FAQ
Is 95% word accuracy the same as "good captions"?
No. 95% word accuracy is WER 0.05, which is only the word layer. A track at that rate can still omit speaker labels, skip an alarm, and reverse a claim with one punctuation miss. Report the extra tallies next to WER or you are describing a different object than a caption.
Do I need a new reference transcript for every sample?
You need a reference that matches the audio of that file. Reusing last month's script on this month's take will manufacture substitutions. For scripted VO that did not change, the script can be the reference. For anything ad-libbed, correct a transcript against the take.
What if a bucket has fewer than four files?
Take all of them as full-file passes. Do not refill from bucket A to make the spreadsheet look balanced. The empty-ish bucket is information: that risk is rare this month, so spend the leftover sits on the bucket that is full of prices and names.
Can I automate the WER count?
The arithmetic is automatable once both texts exist. The reference is not. Do not pipe two ASR outputs into a WER script and call it QC. That measures agreement between engines, which is a useful engineering metric and a useless caption metric.