Synthetic Focus Groups Converge: Six Design Rules

Methodology · 11 min read

TL;DR: A synthetic focus group is not a survey with more personas. It is a multi-agent system, and multi-agent systems converge. Peer-reviewed work through 2025 and 2026 documents the pattern: language model agents adopt the positions of other agents, shed disagreement across discussion rounds, and conform more sharply when they are uncertain. Left alone, that turns a twelve-persona debate into one opinion repeated twelve times. The fix is design, not model choice. Field the independent round first, cap the interaction, seed real disagreement from the panel rather than the prompt, and read rationales before you read consensus. This guide sets out the mechanism, the six design rules, and the questions where a synthetic group is the wrong instrument.

Why do synthetic focus group participants agree with each other?

Because a synthetic focus group is a multi-agent system, and language model agents conform to each other. Peer-reviewed work through 2025 and 2026 documents agents adopting the stated positions of other agents, shedding disagreement across discussion rounds, and conforming more sharply when they are uncertain. The agreement is an interaction artefact, not a finding.

Three mechanisms are at work, and they compound.

Agents adopt each other's positions. An ACL Findings 2025 study of group conformity in multi-agent systems shows the effect is systematic rather than incidental to one setup.¹ Put agents in a room and the room starts having a single opinion.

Discussion rounds shed diversity. Work on emergent convergence in multi-agent annotation finds agreement rising as agents exchange reasoning, which is the opposite of what a focus group is for.² The value of a group is the spread it reveals, not the consensus it reaches.

Uncertainty makes it worse. Research on the drivers of language model social conformity describes the effect as uncertainty-moderated: the less anchored an agent is on a question, the more it defers to the group.³ A new product concept is exactly the low-anchor case.

The conformity is not shallow. A controlled trial using the Asch paradigm found models shifting stated judgements under peer disagreement in a clinical assessment setting,⁵ and related work reports the same direction across tasks.⁴

None of this rules out synthetic groups. It rules out running one on the defaults and reading the consensus.

How is convergence different from sycophancy or neutrality bias?

Sycophancy is a persona agreeing with you, the researcher. Neutrality bias is a persona hedging toward the middle of a scale on its own. Convergence is a third thing: personas agreeing with each other. It survives the controls built for the first two, because it enters through the interaction layer rather than the individual response.

Three failure modes get collapsed into one complaint, which is why the fixes so often miss.

Sycophancy and acquiescence live between the persona and the instrument. A persona agrees with the framing of the question, or with whatever the researcher appears to want. Those are wording problems with wording controls.

Neutrality bias lives inside a single response. A model trained to hedge lands near the middle of the scale regardless of the persona behind it. That is a base-model problem answered by platform-level controls.

Convergence lives in the transcript. It appears only once one persona can see what another persona said, which makes it invisible in every single-response check you run beforehand. A panel that scores clean on both other tests can still collapse the moment you let it talk.

That is the practical reason to test group behaviour separately. The controls do not transfer.

Does a real focus group have the same problem?

Yes, and the qualitative methods literature documented it decades before synthetic panels existed. Real groups produce dominant speakers, social desirability pressure, and a public account rather than a private one. The difference is speed and detectability. Human groups conform unevenly and audibly. Synthetic groups conform fast, quietly, and without the cues a moderator reads.

The honest comparison is between two imperfect instruments, not between a flawed one and a clean one.

Smithson's review of focus group limitations sets out the human failure modes: dominant participants, the group producing a collective public account, and quieter members settling onto it.⁶ Anyone who has watched a group through the glass has seen it inside ten minutes.

What a human moderator has is friction. A pause, a folded arm, a participant who agrees out loud and scores differently on the sheet. Those cues are how a moderator knows a consensus is soft.

A synthetic group offers none of that. It converges in text, at speed, with every persona producing fluent agreement that reads exactly like conviction. Detection has to be designed in, because it will not announce itself.

The method choice itself, cost and speed and scale against depth, is a separate comparison. This section is only about what happens inside the group once you have chosen one.

What does convergence cost you in a real study?

It costs you the variance you paid for. When twelve personas converge, the study reports one opinion with a narrow spread, and a narrow spread reads as a strong signal. The topline looks confident while the disagreement that predicts adoption risk has been erased. You end up most certain exactly where you should be least.

Take a concept screen you could run this week.

You field six concepts to a group of twelve personas, let them discuss each one, and collect ratings after the discussion. Concept C comes back top with the tightest spread on the panel. It looks like the safe bet, so it goes forward.

The tight spread is the artefact. If the first two speakers set the frame and the rest settled onto it, you have measured how persuasive the opening two responses were, not how the market splits.

The damage is specific. Concept screens exist to surface polarization, because a concept that splits a market hard is a different commercial bet from one that mildly pleases everyone. Convergence deletes that distinction while making the report look more confident.

Run the independent round and you can see it directly. Compare the pre-discussion spread to the post-discussion spread. If variance collapsed, the discussion did not add information, it removed it.

What are the design rules for a synthetic focus group that holds signal?

Six rules: field an independent round before any interaction, cap the discussion rounds, seed disagreement through panel composition rather than prompt instructions, randomize speaking order, track variance across rounds as a stopping rule, and read the written rationales before the ratings. Each one targets a documented mechanism rather than a general worry.

Order matters here. Rule one does most of the work.

1. Field the independent round first. Every persona answers alone, with no visibility of any other response, and you record that reading before discussion starts. This is your uncontaminated baseline, and it is the number you report when the two disagree.

2. Cap the discussion rounds. Convergence grows with exchange.² Two or three rounds surface the arguments. Ten rounds manufacture a consensus. Set the cap before the run, not after you read the transcript.

3. Seed disagreement through the panel, not the prompt. Telling a persona to disagree produces performed disagreement. Composing a panel that genuinely contains opposed positions produces the real thing.

4. Randomize speaking order per concept. First speakers anchor the group. If the same persona opens every discussion, you have built one respondent's opinion into six results.

5. Track variance across rounds and use it as a stopping rule. Measure the spread after each round. A sharp drop is your convergence alarm, and it costs nothing to compute.

6. Read rationales before ratings. Two personas can reach the same score for unrelated reasons, which is real agreement, or one can restate the other's argument, which is not. The written justification is what separates them.

Some of this is platform work rather than researcher work. PersonaHive personas are grounded in national census data on a country-specific basis, built from aggregated public statistics and validated against real surveys, which makes rule three a composition step instead of a prompt instruction. The platform also ships anti-mimicry prompting, hardcoded behavioral traits and sampling of extremes, and every response arrives with a written rationale generated before the rating, which makes rule six practical rather than aspirational.

Smallest useful next step: take one question from your last group study, field it independently to a panel, then field it again with discussion, and compare the two spreads. A free PersonaHive account includes 250 credits, which covers that comparison.

When should you not run a synthetic focus group at all?

When the research question depends on genuine social influence, on lived sensory experience, or on a category where public statistics carry no signal about behaviour. A synthetic group simulates deliberation; it does not observe it. If the finding you need is how real people persuade each other, you need real people in the room.

Three cases where the answer is no, and one where it is only partly no.

When the mechanism is the finding. If you are studying how opinion actually spreads through a real social group, a simulation of that process is not evidence about it. You would be measuring the model.

When the response depends on the body. Taste, smell, texture, fatigue, physical discomfort in a store aisle. A persona can describe those. It has not had them.

When public statistics carry no signal. Narrow professional niches, very new categories, and behaviours that census and survey data do not touch. The panel has nothing to be grounded in. That sits alongside the broader validity limits worth knowing before you field anything.

The partial case is recruitment screening. A synthetic group is a reasonable way to sharpen a discussion guide and kill weak stimuli before you spend on live recruitment. It is a poor substitute for the live group itself.

Disclosure applies in every case. ESOMAR's Congress 2024 work on synthetic data in marketing studies treats method transparency as a condition of use rather than a courtesy.⁸ Say in the report that the group was simulated, and say which round you are reporting.

What else do teams ask about synthetic focus groups?

The recurring questions are about group size, whether larger models conform less, what personas should see of each other, whether a synthetic group can replace recruitment, and disclosure obligations. Short answers follow. The through-line is that a synthetic group is a designed instrument, and most of its failure modes are design choices you already control.

How many personas belong in a synthetic focus group? Smaller than you think, and for a different reason than cost. Past roughly eight to twelve, added personas mostly restate positions already in the room once discussion starts. Scale the independent round instead, where more personas genuinely tighten the estimate.

Do larger models conform less? Do not assume it. Work on conformity dynamics finds the effect shaped by network topology, meaning who can see whom, as much as by model choice.⁷ Changing the discussion structure is more reliable than changing the model.

Should personas see each other's ratings, or only the reasoning? Reasoning only, and preferably a subset of it. Sharing numeric ratings gives every persona an explicit target to move toward, which is the fastest route to a collapsed spread.

Can a synthetic group replace recruiting a real one? For stimulus screening and guide development, often yes. For the decision that justified the recruitment budget, no. Treat it as the round before, not the round instead.

Do we have to disclose that the group was simulated? Yes. Disclosure of method, data sources and limitations is the standing expectation in industry codes, and the discussion round you report is part of the method.⁸

So run the comparison once on something real. Field one question independently, field it again with discussion, and look at what the discussion did to the spread. A free PersonaHive account covers it, and the answer will tell you more about your own group design than another vendor page will.

Sources

• An Empirical Study of Group Conformity in Multi-Agent Systems — ACL Findings 2025, Findings of the Association for Computational Linguistics 2025
• Emergent Convergence in Multi-Agent LLM Annotation — BlackboxNLP Workshop, ACL Anthology, BlackboxNLP 2025
• Disentangling the Drivers of LLM Social Conformity: An Uncertainty-Moderated Dual-Process Mechanism — arXiv preprint 2508.14918, arXiv
• Do as We Do, Not as You Think: the Conformity of Large Language Models — arXiv preprint 2501.13381, arXiv
• A controlled trial examining large language model conformity in psychiatric assessment using the Asch paradigm — BMC Psychiatry 2025, BMC Psychiatry
• Using and analysing focus groups: limitations and possibilities — Janet Smithson, International Journal of Social Research Methodology
• Conformity Dynamics in Multi-Agent Systems: A Network Topology Perspective — OpenReview submission, OpenReview
• Synthetic Data in Marketing Studies — ESOMAR, ESOMAR Congress 2024