Synthetic Usability Testing: Nobody Clicked Anything

Methodology · 10 min read

TL;DR: Synthetic usability testing runs into a definition. A usability test measures what someone does with an interface, so task success and time on task are properties of the doing rather than opinions about it, and a panel that answers questions cannot produce them. A census-grounded persona has no session, no pointer and no elapsed time, which puts every behavioral measure out of reach. A second instrument does operate software: a language model agent driving the interface. That is a different tool with its own published error, and the 2026 benchmarks show results moving by double digits when only the simulated user changes. Product teams still get real value from a synthetic panel before the build. Keep the session with people.

Can synthetic users replace usability testing?

No. A usability test is an observation of behavior, and its headline measures are task success and time on task. Those are properties of a person doing something inside your product. A synthetic persona answers questions, which is a different act. Run the synthetic pass before the build, and keep the session with people.

The confusion starts with the word user.

A synthetic user is a language model answering as a described person. A usability test participant is someone handed a task and watched while attempting it. The first produces text. The second produces a record of behavior, and that record is the measurement.

Synthetic usability testing is sold as one thing and covers both, which is how a product team buys a synthetic panel expecting a usability finding and receives an opinion instead.

Sorting it out takes five questions. What a usability test measures, why a persona panel reaches none of it, where language model agents that operate software fit, what the 2026 benchmarks report about them, and which product questions a synthetic panel does answer.

What does a usability test actually measure?

Behavior under a task. ISO 9241-11:2018 frames usability as an outcome of use rather than a property of a screen.¹ Nielsen Norman Group describes the session as a facilitator giving task scenarios to a participant and watching, with task success and time on task the metrics most commonly collected in quantitative tests.²

Read the standard and the method together and the instrument becomes clear.

ISO 9241-11:2018 defines usability and its concepts, and was last reviewed and confirmed in 2023. It explains usability as an outcome of use, which means the measurement needs a use to occur.¹ The standard prescribes no method on purpose.

Nielsen Norman Group supplies the method most teams run. In Kate Moran's primer, the facilitator gives instructions and task scenarios to a participant, then observes. The two metrics named as most commonly collected in quantitative usability testing are task success and time on task, and qualitative sessions are positioned as the way to discover problems rather than to score them.²

Neither metric is an opinion. Task success is a fact about an attempt. Time on task is a duration. Both are recorded by the observer rather than reported by the participant, and that is where the synthetic question arises.

Why can a persona panel not produce those measures?

Because the measure is the behavior itself. A census-grounded persona has no session, no pointer and no elapsed time, so there is no attempt to score and no duration to record. What comes back instead is a stated reaction to a description of your product, which is a different variable carrying its own error.

This is the boundary that already removes an experience question from a synthetic panel. A satisfaction score asks someone to grade an event, and nothing happened to a persona, which is set out in why a customer experience score needs an episode. A usability measure is stricter, because the event has to happen during the interview itself.

The stated layer that remains is not free of error either. Pew Research Center had a model play American Trends Panel members across three 2026 waves and reported an average absolute error of 12 percentage points.³ The pattern mattered more than the size. Synthetic respondents exaggerated differences between groups, putting 97 percent of synthetic Hispanic respondents as at least somewhat likely to follow the World Cup against 42 percent of real Hispanic adults.³

So the behavioral measures are unavailable and the attitudinal ones carry a published offset. Neither is a reason to skip synthetic work. Both are reasons to name the variable you actually bought.

Are language model agents that operate software a usability test?

No, and they are a different instrument from a persona panel. An agent simulator drives the interface and leaves a trace, which looks like usability data. The 2026 benchmarks show that trace moving when only the simulated user changes, so the result grades the simulator as much as the product.

Agentic evaluation is where simulated users actually do things, and the field has started measuring its own reliability.

Seshadri and colleagues evaluated agents on retail tasks with participants in the United States, India, Kenya and Nigeria, and reported agent success rates varying by up to 9 percentage points depending on which model played the user.⁴ Simulated users were systematically miscalibrated, underestimating agent performance on hard tasks and overestimating it on moderate ones. They were also a worse proxy for speakers of African American Vernacular English and Indian English than for speakers of Standard American English.⁴

Jain and Sandhu built UserProxyBench on the same family of tasks. Holding the agent fixed and varying only the user proxy across 375 enterprise tasks moved mean task reward by 15.2 points, and 24.4 percent of successful episodes contained a user specification violation.⁵

A number that moves 15 points when you swap the simulated participant is not yet a product measurement.

Is there anything simulated users do measurably well?

Yes, surfacing candidate problems to investigate. Chowdhury and colleagues report that giving a simulated agent visual access to the page lifted recommendation quality to 0.758 NDCG@3 against 0.359 for text-only simulation, at 89 percent lower simulation cost.⁶ That is triage. It is not evidence about your users.

The honest case for simulation is cheap breadth ahead of expensive depth.

The AMUSER result is the clearest published version of it. Visual access during simulation produced richer traces and better recommendations, and the same authors note that visual input at the recommendation step can modestly degrade quality, which is a useful warning against assuming more context always helps.⁶

Scale is the whole argument. A simulation running at 89 percent lower cost covers far more of a product than a research budget reaches, and what it returns is a ranked list of places to look.⁶

Treat that list as a queue of hypotheses, not a set of findings. Pang and colleagues reviewed 153 CHI papers from 2020 to 2024 that engage with large language models, found authors using them in five roles including as simulated users, and reported that those authors often raise validity and reproducibility concerns while overwhelmingly studying closed models.⁷ The field is using the method and documenting its doubts at the same time, which is the right order.

Which product-research questions does a synthetic panel answer?

The ones that ask what people think rather than what they did. Concept choice, message comprehension, feature trade-offs, naming and price sensitivity all stay within reach, along with checking your own questionnaire. Anything that names a task, a screen flow or an elapsed time belongs in a session with people.

Sort your product research by whether the question needs an event to have happened.

Before the build, most of it does not. Which of nine concepts reads strongest, whether the value proposition lands without a demo, how a feature set trades against price, and whether your questionnaire is written cleanly are judgments about text and meaning. Those are the questions a synthetic interview can carry once you know what probing adds, and the outer boundary is mapped in where synthetic research stops being valid.

PersonaHive runs that pass. Personas are grounded in national census data, country-specific, built from aggregated public statistics and validated against real surveys. A study moves through audience, survey, review and results, and a survey question costs one credit per persona with no run fee. Image stimulus is supported at two credits per study, so a persona can react to a screenshot of your product. It cannot operate one, and PersonaHive does not describe that reaction as a usability finding. Showing an interface is not staging a session, which is the same distinction that governs what an ad test cannot measure without an exposure.

The smallest useful next step is one comparison. Take the last concept test you ran with people, load the same stimulus on a free panel, and check whether the synthetic ranking separates the same top group before the method goes anywhere near a live decision.

Frequently asked questions about synthetic users and usability testing

Short answers to five questions that come up when a product team weighs a synthetic panel against a usability session: whether a screenshot changes the answer, whether a larger panel helps, whether agent simulators count as testing, how discovery interviews differ, and which standard to cite in a methods note.

**Can a synthetic persona give a usability verdict if I show it a screenshot?** No. Showing an image gives the model something to read, not something to use. The reaction is a comprehension finding and worth having. Task success and time on task remain properties of an attempt that did not take place.

**Does a bigger synthetic panel make the usability answer more reliable?** No. Sample size narrows sampling error, and sample is not what is missing here. There is no behavioral measurement at any n, so a larger panel returns the same unavailable variable with a tighter interval around it.

**Are agent simulators usable for anything today?** For regression checks on known flows, and for generating a queue of places to inspect. UserProxyBench reported mean task reward moving 15.2 points across 375 enterprise tasks when only the user proxy changed, so an agent run tells you about the simulator as well as about the product.⁵

**What about discovery interviews rather than usability tests?** Those travel better, because the subject is attitude rather than behavior. A synthetic discussion returns language, objections and framing you can test later. It does not return what someone will do once the screen is in front of them.

**Which standard should a methods note cite?** ISO 9241-11:2018 for the definition of usability as an outcome of use, followed by a plain statement of what your synthetic run measured instead.¹ Name the variable. A reviewer's objection is almost always that the label and the measurement do not match.

Sources

• ISO 9241-11:2018 Ergonomics of human-system interaction, Part 11: Usability: Definitions and concepts — International Organization for Standardization
• Usability (User) Testing 101 — Kate Moran, Nielsen Norman Group
• How Well Synthetic Samples Replicate Public Opinion — Athena Chapekis, Arnold Lau and Samuel Bestvater, Pew Research Center
• Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations — Preethi Seshadri, Samuel Cahyawijaya and Ayomide Odumakinde, arXiv
• UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training — Ashish Jain and Armaan Sandhu, arXiv
• Automatic Multimodal UX Improvement Recommendations from LLM Agent User Simulations — Anu Chowdhury, Bin Wu and Hossein A. Rahmani, arXiv
• Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review — Rock Yuren Pang, Hope Schroeder and Kynnedy Simone Smith, arXiv