Non-English Synthetic Panels: What Language Changes

Methodology · 10 min read

TL;DR: A synthetic panel run in a non-English market does not hold language constant. Prompt language is a variable in its own right, and three mechanisms move it: models express different values depending on the language they are asked in, capability is not equal across European languages, and tokenisation differs by script. Census grounding fixes who is in the sample, not this. The practical consequence is that a Finnish reading and a US reading are two instruments, not one scale used twice. Rank within each market, run a short bilingual control cell to size the offset, and record the run language with every result.

Does a synthetic panel give the same answer in every language?

No. The language a synthetic study runs in is a variable in its own right. Work published in Computational Linguistics in 2026 found that prompt language shifts the cultural values a model expresses, independent of any persona instruction. A Finnish run and a US run are two instruments reporting on two populations, not one scale applied twice.

Ask a Hungarian persona a question in Hungarian and you get a Hungarian answer. Ask the same persona the same question in English and you get a different one. Not a translation of the first. A different answer.

That is not a defect in the panel. It is a documented property of the models underneath it. Research published in Computational Linguistics in 2026 tested how prompt language and explicit cultural framing move the values a model expresses, and found the language of the question doing work on its own.¹

Most multi-market research treats language as a transport layer. You write the questionnaire once, translate it, and assume the instrument survives the trip. Survey methodology has spent decades showing that assumption needs checking even with human respondents, which is why documented translation protocols exist at all.⁷

On a synthetic panel the assumption fails one step earlier. The respondent and the translator are the same system.

What does the language of a run actually change?

Three things, and they stack. The values a model expresses move with the prompt language. Its competence is not equal across languages, which European benchmarking work now measures directly. And tokenisation differs by script, so the same question arrives as a different quantity of text. None of the three is visible in the output.

Separate the three channels, because the controls differ.

The first is value tilt. The language you ask in shifts what the model treats as an ordinary opinion. Moving a run from English to Czech does not simply relocate the respondent. It moves the distribution the answer is drawn from.¹

The second is capability. European evaluation work has built benchmarks across EU languages precisely because performance is not flat across them.³ A 2026 study of multi-turn performance across EU-24 languages put the finding in its title: language equality has a price.² The European Commission announced EU MMLU in July 2026 for the same reason, to measure the gap rather than assume it away.⁶

The third is encoding. Tokenisers split text differently by language and script, so the same sentence arrives as a different quantity of tokens depending on the language it was written in.⁴ That changes how much of your questionnaire fits in context and how much of it competes for attention.

Cross-lingual inconsistency is the measured result of all three. Models return different answers to the same question posed in different languages, and the effect has its own literature.⁵

Why does census grounding not fix this?

Because census grounding sets who is in the sample, not how the model reasons once it is there. PersonaHive builds each country panel from that country's own national census data, which makes a Hungarian panel demographically Hungarian. The language layer sits above it, inside the model, and no amount of demographic calibration reaches it.

A census tells you how many people in Finland are 35 to 44, what they earn and where they live. It does not tell a model how a Finn uses a rating scale.

PersonaHive grounds each panel in its own national census across nine countries: the United States, Germany, France, Austria, the Czech Republic, Hungary, Romania, Denmark and Finland. That is the part public statistics can carry, and why national grounding beats one global panel sets out what it buys you.

The language layer sits above it. Two panels can each be demographically correct in their own country and still return readings that are not on one scale, because the model answering them is not the same instrument in Finnish as it is in English.

This is the honest version of a country coverage claim. Nine national panels is a statement about sampling. It is not a statement that nine readings are interchangeable, and a vendor who presents a nine-market chart without that distinction is selling you a comparison the method does not support.

Which readings from a non-English synthetic panel are comparable?

Rank orders inside one market survive best, because every option carried the same language tilt. Absolute levels and market-to-market gaps survive worst, because the language shift and the real cultural difference arrive inside one number with no way to separate them. Open text sits in between and tracks how well the model handles that language.

Sort your outputs by how much of the language effect each one carries.

A rank order inside one market carries least. Every claim in the set was asked in the same language, so whatever tilt that language introduced applies to all of them. The ordering still reads, which is part of why message testing is the job a synthetic panel does best.

A gap between two markets carries most. Suppose Danish personas score a concept eight points above Romanian ones. Part of that difference is Denmark and Romania. Part of it is Danish and Romanian. The output hands you one number and no way to split it.

Open text sits in between, and its quality tracks how well the model handles that language. In a less-covered language you tend to get shorter and more generic rationales. That is a signal about the run, not a finding about the market.

How do you run a multi-market synthetic study that holds up?

Six steps. Freeze one source questionnaire, translate it outside the model with a documented protocol, run a short bilingual control cell in every non-English market, compare ranks within markets instead of levels across them, record the run language as a field, and send any cross-market gap that drives a decision to live fieldwork.

1. Write one source questionnaire and freeze it. Every market version descends from that document, never from the previous market's version.

2. Translate outside the model. Use independent translation, review and adjudication, and keep the record. The European Social Survey publishes its translation guidelines openly, which is a reasonable template even for commercial work.⁷ Asking the panel to translate its own stimulus merges the instrument with the respondent.

3. Run a bilingual control cell. In each non-English market, field one short fixed block twice, once in the local language and once in English, against the same panel specification. The distance between those two readings is your language effect for that run. On PersonaHive a survey question costs one credit per persona, so a six-item block run twice costs twelve credits per persona, which is cheap enough to repeat every wave.

4. Compare inside markets. Rank in Finland. Rank in Romania. Then compare the two rankings, not the two scores.

5. Record the run language as a field. It belongs next to the model version and the date, alongside the six fields worth recording on every run.

6. Send cross-market gaps to people. Any difference between markets that changes a decision needs human respondents before anyone acts on it.

The smallest useful next step is step 3. Take one block of six items from a study you have already run, field it twice in a single market, and look at how far the two readings sit apart. That one comparison tells you more about your multi-market data than another afternoon spent reading vendor coverage claims.

When does a non-English study still need human respondents?

Four cases. Any decision resting on a market-to-market difference, any study in a language with thin published evaluation, any construct without a settled equivalent across the markets, and any number that will leave the research team as a level rather than a rank.

The first case is the common one. Teams field nine markets because the panel makes nine markets affordable, then build a market prioritisation out of the differences between them. That ordering is the single output the method cannot support.

The second is coverage. Czech, Hungarian, Romanian, Danish and Finnish are all better served than they were two years ago, and all less evaluated than English. Before fielding in one of them, check what published evaluation exists for that language and size your claim to it.³ ⁶

The third is equivalence. Some constructs have no settled translation. Trust, value for money and convenience carry different freight in different markets, and pretesting the questionnaire in every market before you field is the cheapest way to find out which ones travel.

The fourth is the number that leaves the room. Once a figure reaches a board pack it stops carrying its method with it. If a result will be quoted as a level rather than a rank, it needs human respondents behind it.

Frequently asked questions about synthetic research in non-English markets

Short answers to five questions that come up when a synthetic study crosses a language border: whether to run everything in English, whether the model should translate, how large a control cell has to be, what a flat market reading usually means, and whether better models will settle the problem.

**Should I run every market in English to keep the scale consistent?** That makes the readings comparable and makes them about something else. An English run against a Finnish demographic profile measures a model's idea of a Finn in English. If the output is a within-market ranking, English is defensible and the methods note should say so. If the output is meant to describe how a market talks, it is not.

**Can the panel translate the questionnaire itself?** Not if the result has to be defended. Translating inside the model makes the respondent and the translator one system, so a wording error and a finding become hard to tell apart. Translate outside, with a protocol you can show a reviewer.⁷

**How large does the bilingual control cell need to be?** Small. A fixed block of five or six items is enough to see the offset, because you are measuring a systematic shift rather than estimating a population value. Repeat it every wave so the offset stays current.

**One market came back flat while the others differentiated. What happened?** Usually the language, not the market. Short, low-variance, generic answers are the typical signature of a model working in a language it handles less well. Check the open text for length and specificity before concluding that the market is indifferent.

**Will better models settle this?** The capability gap is narrowing. The value tilt is not obviously narrowing with it, because it reflects what the training data contains in each language rather than raw model strength.¹ ⁵ Plan for it to persist, and keep the control cell.

Sources

• LLMs and Cultural Values: The Impact of Prompt Language and Explicit Cultural Framing — ACL Anthology, Computational Linguistics
• Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+ — arXiv
• Towards Multilingual LLM Evaluation for European Languages — arXiv
• Language Model Tokenizers Introduce Unfairness Between Languages — Petrov et al., arXiv
• Understanding Cross-Lingual Inconsistency in Large Language Models — arXiv
• Towards fair multilingual AI: EU MMLU, a new EU benchmark for LLMs — Directorate-General for Translation, European Commission
• ESS Round 11 Translation Guidelines — European Social Survey ERIC, European Social Survey