Back to Blog
Methodology10 min readNov 21, 2026

Preregistering a Synthetic Study: What to Fix First

By PersonaHive Team

TL;DR

To preregister a synthetic study is to write down, before the first confirmatory run, the decision it serves, the configuration it will use, and the rule that turns output into a finding. A synthetic panel needs this more than a human survey does, because a rerun costs minutes. A 2025 study of 13 million text-annotation labels from 18 language models found configuration choices produced wrong conclusions on about 31 percent of hypotheses for state-of-the-art models, and false positives were feasible for 94.4 percent of null hypotheses when the choices were made on purpose. In Pew Research Center's 2026 synthetic poll work, model choice alone moved average error from 11.4 to 17.4 points. Write one dated page with six entries, pilot freely, report every run, and log every deviation.

What does it mean to preregister a synthetic study?

To preregister a synthetic study, you write down and date three things before the first confirmatory run. The decision the study serves. The configuration it will run. The rule that turns output into a finding. Pilots can still come first. What changes is that every later choice is either in the plan or logged as a deviation.

Preregistration comes from human-subjects research, where it separates a test from a search. A plan fixed in advance makes a result a prediction. A plan written after the data arrives makes it a description of whatever turned up.

The problem it addresses does not need bad faith. A 2024 post on Columbia's statistical modeling blog makes the forking-paths point. Multiple comparisons can mislead "even when there is no 'fishing expedition' or 'p-hacking' and the research hypothesis was posited ahead of time."¹ A careful analyst makes small, defensible choices after seeing the data. The series of them is what misleads.

A synthetic study adds choices a human survey never had. Michelle Vaccaro's 2026 ICML paper on preregistering AI-agent experiments lists prompt wording, model selection, decoding parameters, retry policies and response parsing.² It adds refusal handling: exclude, impute, retry, or code a refusal as its own category.² None of these exist when a person fills in a form.

Thomas Kosch and Sebastian Feger give the failure a name. Prompt-hacking is "the strategic tweaking of prompts to elicit desirable outputs from LLMs."³ Their remedy: preregister prompts and protocols, and document every later change that departs from them.³ They go further than this post, recommending that LLMs be used for data analysis "only with caution, oversight, and justification."³

Why does a synthetic panel need preregistration more than a human survey?

Because reruns are nearly free. A human wave costs weeks and a field budget, which caps how many versions of a study anyone can try. A synthetic run costs credits and minutes. The number of paths is limited only by patience, and the one behind the headline stays invisible unless someone wrote the others down.

Vaccaro names the mechanism. Cheap iteration and missing reporting norms make these choices "both easy to exploit and difficult to detect."²

The best measurement of how far configuration can move a conclusion comes from an adjacent field. Joachim Baumann and colleagues at Bocconi University, GESIS and three other institutions replicated 37 annotation tasks from 21 datasets.⁴ They collected 13 million labels from 18 language models across 2,361 hypotheses.⁴ Their name for the result is LLM hacking: configuration choices that lead to incorrect conclusions.

Their numbers are blunt. State-of-the-art models reached a wrong conclusion on about 31 percent of hypotheses, and smaller models on about half.⁴ Made on purpose, the choices could produce a false positive for 94.4 percent of null hypotheses and hide 98.1 percent of true effects.⁴ The authors' summary: "with just a handful of prompt paraphrases, virtually anything can be presented as statistically significant."⁴

Two cautions keep that honest. The study measures models labeling text, not personas answering a questionnaire. It shows the size of the configuration space, not the error rate of any synthetic panel. And its sharpest finding is about thresholds: near p = 0.05, error rates approached 70 percent.⁴ Close calls are where configuration decides the answer, and most briefs hinge on close calls. It is one more reason significance tests mislead on a synthetic panel.

In a human survey the budget limits how many versions of a study get tried. In a synthetic one, only a written plan does.

How much do configuration choices move a synthetic result?

Enough to change a headline. Pew Research Center tuned a 2026 synthetic replication of its own surveys. The choice of model moved average absolute error between 11.4 and 17.4 percentage points. The profile detail given to each persona moved it from 16 to 13.1. Every option was defensible. Only a record of all of them shows how the final one was chosen.

Pew's six authors published the configuration search in full.⁵ They tested three models, three reasoning levels and three levels of profile detail. The test data was an American Trends Panel wave fielded 20 to 25 January 2026.⁵

They could not test every combination. So they tested one dimension at a time, "with the winner of each round advancing to the next."⁵ The winner was Claude Opus 4.6 with low reasoning, extended profile information and an expert reflection step.⁵

Some choices made no difference. High, medium and low reasoning won 43, 39 and 37 questions, and no configuration did noticeably better or worse at the question level.⁵ Others made a large difference, as the table shows. The model rows compare on a 6,700-panelist subsample.⁵

A team that ran only the nano row, or only the standard-profile row, would have reported a different study. Pew tuned on one wave and carried the winner into its replications of later waves. That is the honest order: tune on one benchmark, score on data the tuning never saw. A configuration chosen on a benchmark scores well on that benchmark by construction.

Configuration choiceOption testedAverage absolute error
ModelClaude Opus 4.611.4 points
ModelGPT-5.113.3 points
ModelGPT-5 nano17.4 points
Profile detailStandard profile only16 points
Profile detailExtended profile plus expert reflection13.1 points
Reasoning levelLow, medium or highNo noticeable difference at question level

How do you preregister a synthetic study?

Write one dated page with six entries before the confirmatory run. The decision and its threshold. The primary outcome. The full configuration. Where piloting stops. How refusals and malformed answers are handled. How many runs you will make and report. Share it with whoever will act on the result, and log anything changed afterwards as a deviation.

The six entries map onto the degrees of freedom the research above identifies. The table gives each one and the shortcut it closes.

If you keep a run record, most of the configuration entry already exists. The six fields that make a synthetic study reproducible cover model version, settings and panel definition. The difference is timing. A reproducibility record is written so someone can rerun the study. A preregistration is written so nobody, including you, can quietly choose which run to believe.

Two entries do most of the work. The decision threshold turns a reading into an action before you see it. Concept B goes forward if it beats concept A on top-two-box in every replicate run, say. The reporting rule closes the last door. Vaccaro requires results from every model, "not just those yielding preferred outcomes."² Baumann's team recommends preregistering configuration choices and reporting all tested combinations.⁴

Set the run count in advance. If the plan says five replicate runs, report five, including the odd one. Name the model by exact version: a silent upgrade between pilot and confirmatory run is a deviation whether or not anyone chose it.

EntryWhat you write downWhat it prevents
Decision and thresholdThe business decision and the result that would change itReading a threshold into the data after it arrives
Primary outcomeOne measure and one comparison, named in advancePromoting whichever measure moved
ConfigurationModel version, settings, persona specification, prompts and questionnaireSwapping the model or prompt until the answer improves
Pilot boundaryWhat the pilot tested and the date it endedTreating the last pilot as the confirmatory run
ExclusionsWhether refusals and malformed answers are excluded, retried or codedDropping inconvenient responses after the fact
Runs and reportingThe number of replicate runs and a commitment to report all of themShowing the best run of several
A reproducibility record lets someone rerun the study. A preregistration stops anyone choosing which run to believe.

Can you still pilot and explore after you preregister?

Yes, and you should. Vaccaro's 2026 paper calls pilot testing to refine prompts and select models "both legitimate and encouraged." The condition is that the plan marks where piloting ends and confirmatory testing begins. Exploration after the confirmatory run is allowed too. It is labeled exploratory, reported as such, and tested again before anyone acts on it.

A plan fixes what you promised, not everything you may do. The Columbia post puts it in one line: "Preregistration is a list of things you plan to do, that's all; it does not stop you from doing more."¹

Piloting is where synthetic research is cheapest and most useful. A pilot finds the ambiguous question, the option order that drags answers toward the first choice, the persona specification that collapses into one voice. Vaccaro asks only that you disclose it: how many prompt iterations you explored, and which models you tested.²

Kosch and Feger add a stability check that belongs in the pilot: repeat prompts and see whether the results hold.³ A finding that moves between two identical pilot runs is not ready for a confirmatory one.

What counts as success belongs in the plan too. A November 2026 framework from Enrico D'Ecclesiis, Fulvio Castellacci and Arash Ahmadi separates predictive, distributional and substantive validity.⁶ Testing four off-the-shelf models on Eurobarometer data, they found results varied by dimension and by model.⁶ Their advice is to judge a synthetic sample against the properties its intended use requires. So name that property in advance. A rank order and a level estimate are different tests.

What does a preregistered synthetic study cost on PersonaHive?

The plan costs an hour of writing. The pilot costs credits. On PersonaHive, one credit is one persona answering one survey question, and each new persona costs two. A 20-persona pilot of a 10-question manual questionnaire is 240 credits. The confirmatory run is priced by the same rule, so its cost can go in the plan as well.

Pricing the confirmatory run in advance puts back the discipline a field budget used to supply. Five replicate runs of 150 personas on 10 questions is 7,500 response credits.⁷ Written into the plan, a sixth run becomes a visible line rather than a quiet retry.

PersonaHive grounds each panel in country-specific census data, validates against real surveys, and runs panels in nine countries.⁷ Its published validation report is described as a held-out benchmark against the US CFPB 2024 National Age-Friendly Banking Survey.⁷ That is the principle this post argues for at study level: the score that counts comes from data the configuration was not tuned on.

The smallest useful next step is to write the page before your next study. Six entries, dated, shared with the person who will act on the result. Run the pilot on the free credits a new account receives, granted once with no card required.⁷ Then write down where the pilot ended, before the first confirmatory run starts.

Frequently asked questions about preregistering a synthetic study

Five questions come up when a team writes its first plan. Is preregistration only for academic work? Does a fixed temperature or seed replace it? What if the first run looks wrong? Where should the plan live? And is a preregistered study automatically a valid one?

Is preregistration only for academic research? No. The mechanism is the same in a brand team as in a journal: a result chosen after several tries reads stronger than it is. Commercial work does not need a public registry. A dated document shared with the decision owner before the run does the job.

Does a fixed temperature or seed replace a plan? No. Fixed settings make runs comparable, which is a reproducibility property. They do not stop anyone from choosing among configurations, which is the problem a plan solves.

What if the first confirmatory run looks wrong? Report it, then investigate. Vaccaro's template treats deviations from the registered plan as acceptable "when clearly disclosed and justified."² Log what you changed, why, and the result before and after. A rerun that silently replaces the first one is the failure the plan exists to prevent.

Where should the plan live? With the study files, next to the run record, in a form nobody edits once the run starts. Kosch and Feger recommend recording the whole prompt creation process, including the specific model used.³

Does a preregistered study give a valid answer? Not on its own. Preregistration makes a finding honest about how it was produced. Whether the panel represents the population is a separate question, answered by a validation study against a live benchmark.

Preregistration does not make a synthetic finding true. It makes it honest about how many tries it took.

Sources

Run trusted AI market research in minutes.

Put this method to work on a census-aligned synthetic panel.

No credit card · Instant access · free credits

© 2026 PersonaHive. All rights reserved.