Sampling QA when you ship hundreds of assets
A batch-level sampling scheme, the checks that must stay at 100 percent, and the stop-the-line rule that turns a sample into real coverage.
Four hundred assets in a week, ninety seconds of honest human attention each, and you have spent ten hours reviewing. That is a full day and a quarter of your most expensive person's time, every week, on a task that gets measurably worse after about the fortieth asset. Reviewing everything at volume is not rigour. It is a plan to review nothing carefully.
Reviewing none is obviously worse. Somewhere in four hundred assets is a stripped disclosure label, a competitor's logo in a background, or a claim nobody cleared.
The way out is not "review harder." It is to sort checks by cost, run the cheap ones on everything, and sample the expensive ones in a way that matches how generated defects actually distribute — which is not randomly.
Sort every check into one of three cost classes
Before designing a sampling scheme, split your checklist. Most teams sample the wrong things because they treat all checks as equally expensive, when the spread is enormous.
Class A: machine-cheap. A script or an automated pass answers it in under a second, at no meaningful marginal cost per asset. Duration, aspect ratio, resolution, frame rate, file size, loudness, caption track present, filename convention, provenance metadata present. These get sampled by nobody. They run on 100% forever.
Class B: human-fast. A trained person answers it in fifteen to thirty seconds with the asset in front of them. Does the product look right, does the text read, is the grade consistent with the batch, does the motion hold up. These are the ones you sample.
Class C: high-consequence. Claim substantiation, likeness clearance, legal read, competitor conflict, disclosure adequacy. The downside of missing one is not a mediocre asset but an incident. These are never sampled. They are scoped instead — you decide which assets need them, then run them on 100% of that set.
The mistake behind most sampling failures is treating a Class C check as Class B because it is quick to eyeball. "Is there a real person's face in this" takes two seconds to answer and still belongs in Class C, because the consequence of the miss is not aesthetic.
What stays at 100 percent
| Check | Class | Why it never gets sampled |
|---|---|---|
| Duration, aspect ratio, resolution, frame rate | A | Machine-checkable, and a wrong spec is a rejected placement |
| Loudness and audio present | A | Silent or blown-out audio is the most common silent failure |
| Captions present and burned in where required | A | Platform requirement, trivially checkable |
| Provenance metadata survived the export | A | Delivery spec, and it is silently stripped by re-encodes |
| Disclosure label present where required | A/C | Binding requirement, not a taste question |
| Real identifiable person or voice appears | C | Rights exposure, and the answer routes the job |
| On-screen claim or price | C | Substantiation is a legal question, not a QC one |
| Competitor or third-party mark visible | C | Cheap to miss, expensive to explain |
Class A rows should be a script, not a person with a checklist. If you are asking a human to confirm a file is 1080×1920, you have already lost the capacity argument. An automated pre-publish pass handles the mechanical layer and hands a human a severity rating instead of a raw asset — automated QA before a paid placement goes live covers what it can and cannot resolve.
Class C rows are 100% of the assets that trigger them, usually a small fraction. Ask the trigger question at intake, tag the batch, and review load stays proportional to real risk rather than to volume.
Why the batch is the sampling unit
Here is the part that most QA schemes get wrong. Classic sampling assumes defects are independent — that asset 47 being broken tells you nothing about asset 48. In generated output that assumption is false, and usefully so.
Assets in one batch share a prompt structure, a model, a reference image, a seed lineage, a caption template and an export preset. When something is wrong, it is usually wrong systematically: the reference had a flaw, the prompt lacked a negative constraint, the preset dropped the metadata. Defects cluster by batch far more than they scatter across a day's output. Two consequences follow.
Sample within batches, not across the day. Ten assets pulled from ten different batches tells you less than ten from one batch, because you learn nothing complete about any of them. Sample every batch, proportional to its size.
A confirmed defect implies its batch, not just itself. If the sample turns up a broken caption template, the next action is not "fix that asset" but "check the whole batch," because the cause is upstream. This is the property that makes small samples work.
The scheme
Sample size is a coverage decision, and the arithmetic is not intuitive. For a batch with a given defect rate, here is the chance a random sample of a given size finds at least one defective asset, assuming independence:
| Defect rate in the batch | Sample of 5 | Sample of 10 | Sample of 20 |
|---|---|---|---|
| 20% | 67% | 89% | 99% |
| 10% | 41% | 65% | 88% |
| 5% | 23% | 40% | 64% |
| 2% | 10% | 18% | 33% |
Read the bottom row honestly. At a 2% defect rate, a sample of 20 misses the problem two times out of three. Sampling does not catch rare defects. It catches systematic ones, which is exactly what batch-clustered defects are, and it is why the rare-but-serious checks live in Class C at 100%. A workable scheme:
| Batch size | Class B sample | Notes |
|---|---|---|
| Under 10 | All of them | Sampling saves nothing at this size |
| 10–30 | 5 | Include the first and last generated |
| 31–100 | 10 | Stratify across sub-prompts or variants |
| Over 100 | 10, plus 5 per additional 100 | Diminishing returns past about 25 |
Two selection rules matter more than the count.
Always include the first and last asset in the batch. Drift, credential stripping and template errors show up at the edges. The first tells you whether the setup was right; the last tells you whether it stayed right.
Stratify by variant, not by position. If a batch of 60 covers three product variants, take four from each rather than twelve at random. Variant is the dimension along which prompts differ, so it is the dimension along which defects differ. Batch generation testing across 20 creatives and one instruction across forty assets both produce this variant structure.
Anything going to paid media, a landing page, or a launch is exempt from sampling and gets the full pass regardless of batch size. Write that next to the scheme so nobody makes the judgement call under deadline.
The escalation rule
A sampling scheme without a stop-the-line rule is a slower way of shipping the same defects. State it in one sentence and apply it mechanically:
Two defects of any severity in one sample, or one blocking defect, escalates the whole batch to 100% review and blocks the batch from delivery until the cause is named.
Three things that rule is doing.
It uses two rather than one, because a single cosmetic reject in a batch of 60 is ordinary variance and stopping the line for it trains people to under-report. It escalates on one blocking defect regardless of count, because a rights or disclosure failure is never variance. And it requires a named cause before release, not just a fixed asset, because the whole premise of batch sampling is that defects have shared causes.
When a batch escalates, the output of the escalation is a change to the input: a negative constraint added to the prompt, a fixed reference image, a corrected export preset, a new line on the checklist. If a batch escalates twice for the same cause, the fix went in the wrong place.
Log every escalation with the batch identifier, sample size, defect found, and cause named. That log is the only source of the number this entire scheme depends on — your actual defect rate. There is no published benchmark for it, and your rate depends on your models, your prompts and your brand's tolerance, none of which generalise. Thirty logged batches turns the sample sizes above from a guess into a decision. Asset naming and version discipline is what makes a batch identifier worth logging against.
For the checks themselves, quality control for AI-generated marketing assets has the gate structure, the brand safety checklist covers the Class C questions, and auditing generated output against a brand manual is the batch-level brand pass. You can also run an AI check on a video against the brief as a first filter, which shifts some Class B load toward Class A.
FAQ
What sample rate should I start with if I have no data?
Ten per batch, plus the first and last, and the two-defect escalation rule. Run it for thirty batches, then read your escalation log. Escalating more than about one batch in four means your defect rate is high enough that the sample is not the problem — fix the inputs. Never escalating means either the process is genuinely clean or reviewers are not recording minor defects, and the second is more likely.
Does sampling work for images as well as video?
Better, because images are faster to assess and batches are larger, so the same sample size costs less. The adjustment is that image batches vary along more dimensions at once — product, background, angle, crop — so stratification matters more than sample size.
How do I stop reviewers from just approving the sample?
Give the verdict more than two states. A binary pass/fail collapses toward pass under volume. Pass, pass-with-note, and reject-with-class gives a reviewer somewhere to put "this is fine but the hands are odd," and those notes are what populate your defect data. A reviewer who never records a minor defect is not finding zero, they are recording zero.
Should the person who generated the batch review it?
They should run the Class A pass, which is mechanical and immediate. They should not be the only person on the Class B sample for a batch they built, because you cannot reliably see the flaw you prompted for. Rotating a second pair of eyes across ten assets costs little and catches exactly that category.