03/08 · Essay
There is a growing literature on using language models as stand-ins for research participants. The idea has a name, silicon sampling, and a respectable origin in political science, where models conditioned on demographic profiles reproduced the distribution of survey responses across groups with surprising fidelity. In 2026 a major survey organisation ran a careful replication with digital twins of real respondents. The results were mixed and the methodology was honest, which is more than can be said for some of the enthusiasm that followed.
I have been asked, more than once, whether we could use this for formative research. Instead of forty interviews with pregnant women in Assam, prompt a model forty times with forty profiles and read what comes back. It would be faster. It would be cheaper. It would never require a field vehicle.
Plausible but not valid
A psychometric audit published last year gave the problem its best name: plausible but not valid. Model-generated respondents produce answers that read correctly. They say the things a pregnant woman in a rural district might say, in the register a researcher expects. What they do not do is vary the way real people vary. The audit found that synthetic samples compress variance, drift toward the positive, and reproduce the central tendency of whatever the model learned rather than the particular distribution of the population you care about.
Another paper put it more bluntly, borrowing a term from philosophy: the model is das Man, the generalised anyone. It speaks as everyone and therefore as no one. A third group, at a human-computer interaction conference, called the output a simulacrum of stories.
For formative research, this is disqualifying. The whole point of formative work is to find the thing you did not expect. The woman who skips antenatal visits not because she does not know they matter but because the auxiliary nurse midwife at the sub-centre once shamed her in front of her mother-in-law. The household where the decision about a facility birth belongs to a brother-in-law who lives in another state. These are not central tendencies. They are the outliers that a program has to be designed around, and they are exactly what a model trained on the average of the internet will smooth away.
Where a model could help
I am not against the tools. I think they have a narrow and useful role in formative work, as long as it is honest about what they are. A model is a good sparring partner for an interview guide: ask it to find the leading questions, the assumptions, the places where a Hindi phrasing carries a connotation the English draft did not. A model can help a small team pre-test a survey for comprehension problems before it costs a day of fieldwork to discover them. A model can summarise the published literature on a community before you arrive, and a good one will tell you how thin that literature is.
None of that is sampling. It is preparation. The moment a model’s output becomes a data point in a results table, something has gone wrong with the study.
The test I use
My test is simple. If a method could not surprise me, it is not formative research. Real participants surprise me every time. Synthetic ones have not yet, and the audit literature suggests they are built not to.
Sources
- Argyle LP, et al. Out of one, many: using language models to simulate human samples. Political Analysis, 2023.
- Pew Research Center. Methodology: silicon samples and digital twins. 2026. https://www.pewresearch.org/data-labs/2026/09/30/methodology-silicon-samples/
- Plausible but not valid: a psychometric audit of LLM-simulated respondents. 2026. https://arxiv.org/pdf/2608.14606
- ChatGPT is das Man: on the generic voice of language models. 2025. https://arxiv.org/pdf/2507.02919
- Kapania S, et al. Simulacrum of stories: examining large language models as qualitative research participants. CHI, 2025. https://arxiv.org/pdf/2409.19430