Claim Substantiation: Where Synthetic Data Stops

Enterprise · 11 min read

TL;DR: A synthetic panel cannot substantiate an advertising claim. Substantiation is a statement about people, and the FTC requires a reasonable basis held before the claim runs. The National Advertising Division, the ASA and the courts read consumer perception evidence the same way: a sample drawn from the population the claim addresses, made of respondents who exist. A synthetic panel samples no population, carries less variance than the human data it stands in for, and leaves no fieldwork record. It has three real jobs upstream of the study that becomes evidence: screening which claims are worth paying to test, pretesting the questionnaire, and mapping the unintended takeaways that decide most claims challenges.

Can synthetic research substantiate an advertising claim?

No. Substantiation is a statement about people, and every framework that grades it asks who was surveyed, how they were selected, and what they understood. A synthetic panel answers none of those questions with a person. Use it to decide which claim is worth testing and how to word the test, then field that test with human respondents.

The question arrives in a specific form. Legal has asked for backup on a preference claim, the survey house has quoted six weeks and a five-figure fee, and someone has noticed that a synthetic panel returns the same numbers by lunchtime. The numbers look alike. The evidentiary status is not close.

An advertising claim carries two separate burdens. The first is that the claim is true. The second is that you held proof of it before the ad ran. A synthetic study can inform the first. It cannot discharge the second, because the proof a regulator asks for is a record of people being asked.

That is a boundary, not a verdict on the method. The same panel that cannot sit in an evidence file is well suited to the work that happens before one exists.

What do regulators actually require as claim evidence?

A reasonable basis held before the claim runs. The FTC's substantiation policy states that advertisers must have that basis at the time a claim is disseminated, and objective claims call for competent and reliable evidence. In the UK, CAP Code rule 3.7 requires documentary evidence held before an ad appears. Neither regime accepts proof assembled after a challenge.

The FTC's Policy Statement Regarding Advertising Substantiation sets the American baseline: a reasonable basis for objective claims, existing before dissemination rather than produced in response to an inquiry¹. The standard is about the quality of the support and about its timing at once.

Most disputes never reach the FTC. They reach the National Advertising Division at BBB National Programs, the industry self regulatory forum where a competitor challenges your claim and you file your support³. A case there is decided on the documents. What you hand over is the study, its method, and the sample it rests on.

The UK route runs through the ASA. CAP Code rule 3.7 requires marketers to hold documentary evidence for objective claims before publication⁴. CAP's advice on consumer surveys and sample claims goes further into practice, setting out expectations for who was surveyed, how many people, and how the question was put⁵.

Read the three together and a pattern shows. None of them asks whether your number is plausible. They ask who produced it and under what conditions. That is a provenance test, and provenance is precisely where a synthetic panel has nothing to file.

Why does a synthetic panel fail that test specifically?

Three reasons, and only the first concerns accuracy. A synthetic panel samples no population, so there is no sampling frame to describe. Its responses carry markedly less variance than the human data they stand in for. And no respondent can be re-contacted, which removes the fieldwork record an evidence file is built from.

Start with the sampling frame. A claims survey describes the population the claim addresses, then explains how respondents were drawn from it. A census-grounded synthetic panel matches the marginal distributions of a national population, which is a calibration, not a draw. There is no frame, no response rate, and no non-response analysis to submit.

The second reason is measured. Bisbee and colleagues reported in Political Analysis that simulated survey responses carried substantially less variance than the human data they were meant to replace, and shifted when the prompt was reworded⁶. A preference claim lives on a proportion, and a compressed distribution moves proportions in a direction nobody can bound.

The third reason is the one that survives every future improvement in accuracy. Even a panel that matched human results exactly would still have no consent record, no screening log, no field dates and no respondent to verify. Those artefacts are not a formality around the evidence. In a challenge, they are the evidence.

This is the same distinction that separates a synthetic purchase intent score from a sales forecast. One is a reading of a model. The other is a claim about a market, and only one of them has to survive cross examination.

Is AI-generated consumer voice in an ad a separate violation?

Yes, and it is the sharper exposure. The FTC's Trade Regulation Rule on the Use of Consumer Reviews and Testimonials, finalised in 2024, reaches reviews and testimonials that misrepresent a real person's experience, including material generated by AI. Internal synthetic research sits outside it. A synthetic verbatim republished as a customer quote does not.

16 CFR Part 465 addresses fake and false consumer reviews and testimonials, among them those attributed to people who do not exist or who never used the product². Read against a synthetic research programme, it draws a line most teams have not drawn for themselves.

On one side sits a synthetic panel used inside the research function to plan a study. Nothing there is represented to a consumer. On the other sits any synthetic output that reaches an audience wearing the clothes of a real customer: a quote in a landing page, a testimonial card, a persona line lifted into a campaign deck that later becomes a press release.

The failure mode is drift rather than intent. A well written synthetic verbatim moves from an appendix to a summary slide, loses its label in the third revision, and arrives in a creative brief as something a customer said. Nobody decided to mislead, and the format did the work. The same drift is why synthetic open-ends need a rule about what counts as a finding.

ESOMAR's buyer guidance on augmented synthetic data puts how the data was generated, and how that is disclosed, among the questions to settle before commissioning⁷. Settling it at commissioning is cheap. Settling it during a challenge is not.

What is a synthetic panel actually for in a claims workflow?

Three jobs, all upstream of the study that becomes evidence. It screens a long claim list down to the few worth paying to test. It pretests the questionnaire that will carry the claim. And it maps the unintended takeaways an execution can produce, which is the ground most claims challenges are fought on.

Screening comes first and saves the most money. Marketing arrives with fourteen candidate claims. A claims survey can carry three or four before it becomes unwieldy. Ranking fourteen by how a census-grounded panel responds costs an afternoon, and the ranking is a hypothesis order rather than a result, which is all the decision needs.

Questionnaire pretesting is the underrated one. Claims surveys lose on instrument defects more often than on findings: a double-barrelled question, an order effect, a scale that invites agreement. Those defects are detectable without human respondents, and the checks that catch them are the same ones used in multi-market survey pretesting.

Takeaway mapping is where the method earns its place. A claims challenge usually turns on what the ad conveyed rather than on what it said. Running the execution past a panel and collecting every interpretation produces an inventory of possible readings, including the ones your team is too close to the brief to see. You then measure incidence of those readings with human respondents.

PersonaHive runs these panels across nine countries, each grounded in its own national census data and built from aggregated public statistics, with every response shipping a written rationale so a reviewer can read why a persona landed where it did. The free tier includes 250 credits and no card. The smallest useful test: take a claim already in market and run the takeaway map on a matched panel, then compare it against the objections your last human study surfaced.

How do you keep the synthetic stage out of the evidence file?

Label it at creation rather than at review. Every synthetic output carries a method line naming panel, country, model version, run date and a sentence stating the responses were generated by synthetic personas. The claims file then holds the human study alone, and legal reviews one document set instead of two that look alike.

Four rules hold the separation in practice.

First, physical separation. Synthetic outputs live in a research working folder, never in the claims substantiation folder. Copying rather than moving is how a file crosses the line.

Second, the method line travels with the artefact. Panel, country, model version, run date, question wording and replicate count, written into the document rather than into an email around it.

Third, verbs are disciplined. A synthetic panel surfaced an interpretation. Consumers did not raise it. The first is supportable and the second is a claim about incidence your data cannot carry.

Fourth, the brief to the survey house names what came before. Telling the vendor which takeaways were identified synthetically improves the instrument and creates a clean record of how the study was designed.

This sits alongside, not instead of, the obligations attached to the AI system itself. Those run on a different track, covered in what the EU AI Act actually applies to. Advertising law grades your claim evidence. AI law grades your system. Meeting one says nothing about the other.

What else do brand and legal teams ask?

Five questions recur in claims reviews: whether a larger synthetic sample changes the answer, whether a hybrid design helps, whether internal claims are treated differently, whether comparative claims raise the bar, and what to tell an agency partner. Short answers follow, with the reasoning in the sections above.

Does a bigger synthetic sample help? No. More personas draw more responses from the same generator, which tightens the centre and leaves the provenance problem untouched. Sample size is not the objection.

Does a hybrid design work, synthetic plus a small human cell? For research purposes, sometimes. For substantiation, the human cell is the evidence and it has to stand alone at a defensible size. Pooling does not let a small human sample borrow authority from a large synthetic one.

Are internal or B2B claims treated differently? The forum changes and the standard does not. A claim made to business buyers is still an objective claim requiring a reasonable basis held in advance¹.

Do comparative claims raise the bar? In practice yes. Comparative claims attract competitor challenges, which is how most matters reach the National Advertising Division³, and a challenged advertiser files method as well as results.

What should you tell an agency partner? That synthetic work informs the brief and never the footnote. If an agency proposes a synthetic study as claim support, the answer is that the study is the wrong instrument for that job, not that the numbers are wrong.

The practical next step is small. Take one claim you are preparing to test and run the takeaway map on a matched panel before you brief the survey house. You will either find an interpretation worth adding to the instrument, which pays for the exercise, or confirm the instrument is complete, which is also worth knowing before the invoice.

Sources

• FTC Policy Statement Regarding Advertising Substantiation — Federal Trade Commission, Federal Trade Commission
• Trade Regulation Rule on the Use of Consumer Reviews and Testimonials, 16 CFR Part 465 (Final Rule) — Federal Trade Commission, Federal Trade Commission
• National Advertising Division — BBB National Programs, BBB National Programs
• CAP Code rule 3.7: Substantiation — Committee of Advertising Practice, Committee of Advertising Practice
• Substantiation: Consumer surveys and sample claims — ASA and CAP, Advertising Standards Authority and Committee of Advertising Practice
• Synthetic Replacements for Human Survey Data? The Perils of Large Language Models — James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, Jennifer M. Larson, Political Analysis, Cambridge University Press
• 5 Topics of Discussion to Help Buyers of Augmented Synthetic Data — ESOMAR, ESOMAR