How to Write Survey Questions for Synthetic Personas

Playbook · 11 min read

TL;DR: Synthetic panels fail more often on the instrument than on the model. Writing survey questions for synthetic personas is a different craft from writing them for people, because peer-reviewed work shows large language models pick answers labelled A, flip when option order reverses, and shift on paraphrases that leave humans unmoved. Those are questionnaire defects, not persona defects, and the researcher controls all of them. This playbook sets out the rules that hold: randomize response order, replace agree-disagree batteries with forced choices, keep instruments short enough to stay coherent, control question order, and pretest by reading written rationales before you trust a number.

Why do survey questions for synthetic personas need different rules?

Large language models react to question format in ways humans do not. Across nine models tested in a 2024 TACL study, models generally failed to reproduce human response biases, and shifted on wording changes that leave people unmoved. The instrument, not the persona, is where most synthetic studies go wrong first.

A human questionnaire carries assumptions that do not transfer. It assumes the respondent reads the option list in a stable way and has a fixed opinion to report. A synthetic panel breaks both.

Tjuatja and colleagues tested nine open and commercial models against known human response biases drawn from the survey methodology literature.¹ Models trained with reinforcement learning from human feedback matched worst. Even where a model moved in the same direction as people, it shifted on perturbations that produce no significant change in human samples.¹

Dominguez-Olmedo and colleagues found the same fragility from another angle. Across 43 models tested on American Community Survey questions, they documented ordering and labelling bias, including a pull toward whichever option was labelled A.²

The useful consequence is that most of this sits under your control. It lives in the wording, the option list, and the order, not in the persona engine. Instrument defects also sit alongside deeper validity limits that no wording fix repairs.

Should you randomize response option order in a synthetic study?

Yes, always, and it matters more than in human surveys. Models show a documented pull toward the first labelled option, and one 2025 study found that moving a semantically identical option to the last position changed selection frequency by more than 2,000 percent in a small model. Randomize order per persona and treat any study that did not as unreadable.

Pew Research Center randomizes most response options in its own surveys, because in self-administered surveys people pick items near the top and in telephone surveys they pick items heard later.⁴ In humans those effects are modest. In models they can be extreme.

Rupprecht and colleagues ran 167,400 simulated interviews across nine models on World Values Survey items, applying eleven perturbations to questions and answer lists.³ Recency dominated. Making the last option semantically identical to the first raised selection of that last option by over 2,000 percent for Llama-3.1-8B.³ Larger models held better: Llama-3.3-70B and Gemini-1.5-Pro reproduced their original answer in more than half of perturbed runs, against under 5 percent for a 1B model.³

There is a second reason to randomize. Dominguez-Olmedo and colleagues found that once ordering bias is removed, models drift toward uniformly random answers regardless of size or training data.² A result that survives only one fixed option order is an artifact of that order.

Randomization is therefore both a fix and a test.

Which question formats break with AI personas, and what should you use instead?

Agree-disagree batteries, long unordered option lists, and double-barreled items break hardest. Replace them with forced choices between balanced alternatives, short randomized lists, and one idea per question. These are the same repairs survey methodologists recommend for human respondents, applied more strictly, because model sensitivity to framing is larger and less predictable than human sensitivity.

Format decides more of the answer than most people expect. Pew reports that in a 2008 election poll, 58 percent named the economy when it appeared as a listed option, against 35 percent who volunteered it unprompted.⁴ That gap belongs to the instrument. Ask the closed question first and the open follow-up second, never the reverse.

Agree-disagree formats need their own warning. Pew recommends a choice between alternative statements instead, because agreement formats produce acquiescence bias in people.⁴ In models the effect exists but runs the other way. Braun's 2025 study of five models across 37,975 question variations in three languages found a bias toward answering no, regardless of whether no meant agreement, with option-B responses rising 31 to 203 percent when neutral A/B questions became yes/no items.⁵

The direction of the bias is not the one you were trained to correct for. A forced choice between two balanced statements removes the problem rather than adjusting for it.

Platform-level controls for agreement and neutrality bias handle part of this. None of them repair a double-barreled question.

How many questions should a synthetic questionnaire contain?

Between eight and fifteen for most studies. Short instruments keep answers internally coherent and keep cost legible, since on PersonaHive one credit buys one persona answering one question. A 150-persona panel on a 12-question instrument costs 1,800 credits, which fits inside the Growth plan's 12,500 monthly credits with room to re-run several times.

Question count is a budget line. One credit is one persona answering one question, so instrument length and panel size multiply. Going from 12 questions to 24 doubles the cost of every re-run, and you will want several.

Length also affects coherence, for a reason specific to how the answers are produced. PersonaHive generates between two and eight complete questionnaire scenarios per persona and samples one whole scenario, so related answers reflect the same respondent state instead of being drawn item by item. That property is easier to hold across a compact instrument.

The published robustness work is also short-form. The World Values Survey items used in the 167,400-interview perturbation study are single questions, not 40-item batteries.³ There is no published evidence that long synthetic instruments hold up. Absent it, keep them short.

PersonaHive's July 2026 validation report, which benchmarked a panel grounded in national census data blind against a published national survey, used a 12-question instrument. That is a fair working ceiling.

Panel size is a separate calculation with its own logic.

Does question order matter when a persona answers the whole questionnaire?

Yes, and more than in human fieldwork. Earlier questions set context for later ones, an effect Pew documents in human samples, and the effect compounds when a persona produces one coherent set of answers in a single pass. Put unaided and open questions before aided ones, and never place a concept description before the question measuring unprompted awareness.

Pew documents a clean case in human data. Support for legal agreements for same-sex couples ran at 45 percent when the question followed a marriage question, and 37 percent when it did not.⁴ Eight points, from sequence alone.

Synthetic panels inherit that and add a mechanism of their own. Because the engine draws one complete questionnaire scenario per persona rather than answering each item independently, the whole instrument acts as a single context. Everything read early is present at question ten.

That cuts both ways. It is why synthetic answers hold together across a questionnaire instead of contradicting each other item by item. It is also why a leading question early on contaminates everything after it, not just the item beside it.

Two rules follow. Ask unaided before aided. And when you need a clean read on two concepts, split them into separate runs, because the first will colour the second.

How do you pretest a synthetic questionnaire before trusting the results?

Run the instrument on a small panel first and read the written rationales, not the numbers. Every PersonaHive response ships with a justification grounded in the persona's own background, so a misread question shows up as a rationale that answers something you did not ask. Then re-run with shuffled option order and compare toplines.

A synthetic pretest costs almost nothing. Twenty personas on a 10-question draft is 200 credits, inside the 250 credits a new free account receives.

Read the rationales for three things. Does the rationale reference what you asked, or something adjacent, which means the wording is ambiguous. Do rationales across personas cite different reasons, or repeat one phrase, which means the question is not discriminating. Does any rationale answer only half the question, which means a double-barrel survived.

Then run the order test. Re-run with response options shuffled and compare toplines. A stable question moves within noise. A fragile one moves a lot, and Dominguez-Olmedo and colleagues explain why: strip the ordering signal and answers drift toward random.²

The 2025 fifth edition of the ICC/ESOMAR International Code places overall responsibility for research on the researcher regardless of the technology applied, and requires disclosure of methods, data sources and limitations so clients can assess validity.⁶ A documented pretest and an order test are what that obligation looks like for a synthetic study.

Smallest useful next step: open a free PersonaHive account, which includes 250 credits and needs no card, and pretest one 10-question draft on a 20-persona panel before committing to a full run.

What else do teams ask about writing questions for synthetic panels?

The recurring questions are about scales, sample size, whether to reuse a human questionnaire verbatim, and how to report synthetic evidence to a skeptical stakeholder. Short answers follow. The through-line is that a synthetic instrument is a human instrument with the ambiguity removed, the option order randomized, and the length cut.

Can I reuse a questionnaire I already fielded with human respondents? Yes, and it is the best starting point, because you can compare the two sets of answers directly. Randomize the option order before you run it, and expect the differences to be largest in small subgroups rather than in the topline.

Should I use 5-point or 7-point scales? Either works if every point is labelled. Unlabelled scales leave the middle point undefined, which is where uncertainty pools. Where the decision hinges on a rank, ask for a forced ranking instead of a rating.

How many personas do I need? That depends on how much variation you need to observe, not on instrument length. Sample size for synthetic panels follows its own logic and deserves a separate calculation.

Do I need to disclose that responses were synthetic? Yes. The 2025 ICC/ESOMAR International Code requires disclosure of methods, data sources and limitations.⁶ State the panel construction, the country whose census grounds it, and the fact that answers are model-generated, in the report itself rather than a footnote.

Will better wording make synthetic research valid for any question? No. Wording fixes instrument defects. It does not extend synthetic research to low-incidence populations, sensory response, or regulated claims, where live respondents remain necessary.

Write the instrument first, pretest it on 20 personas, then scale. A free PersonaHive account includes 250 credits, which covers a full pretest before you spend anything.

Sources

• Do LLMs Exhibit Human-like Response Biases? A Case Study in Survey Design — Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkar, Graham Neubig, Transactions of the Association for Computational Linguistics
• Questioning the Survey Responses of Large Language Models — Ricardo Dominguez-Olmedo, Moritz Hardt, Celestine Mendler-Dunner, NeurIPS 2024
• Prompt Perturbations Reveal Human-Like Biases in LLM Survey Responses — Jens Rupprecht, Georg Ahnert, Markus Strohmaier, arXiv
• Writing Survey Questions — Pew Research Center Methods, Pew Research Center
• Acquiescence Bias in Large Language Models — Daniel Braun, arXiv
• ICC/ESOMAR International Code on Market, Opinion and Social Research and Data Analytics — ICC and ESOMAR, International Chamber of Commerce