Surveys and polls help societies understand what people think about issues in politics, health, education and much more. But fewer people these days tend to respond, so pollsters have to reach out more widely, which raises cost considerably. One survey provider prices a 10 minute survey of 1,000 people in the tens of thousands of dollars.
Could AI models stand in for hundreds or thousands of people, emulating the range of answers humans would provide? This practice, known as synthetic surveys or silicon sampling, is already happening, and it’s far less expensive. But are the results trustworthy?
I am a machine learning researcher. I study large language models and their uses in medicine and science. These systems change constantly as companies update them. Different prompts, settings and model versions can produce very different answers to questions. That trait can make models difficult to use reliably in social science research, but it can help simulate replies of many humans, what researchers call “synthetic respondents.”
To create 10,000 answers from ChatGPT, for example, a pollster would prompt the model with some basic respondent demographics and context, such as “You are a young college-going urban voter with conservative political views. Respond to the following questions.” Researchers can change the demographic settings to elicit many different responses from ChatGPT for the same query.
The model also has its own internal randomness, so it naturally generates different replies to the same question asked repeatedly. In this way, researchers can combine prompting and randomness to create 10,000 different synthetic responses.
Simulations are not opinions
Pollsters have long used statistical models to generalize results from a finite number of replies. And analysts can reach different conclusions from the same survey data. Studies of synthetic respondents suggest they may be even more sensitive than people to small changes in prompts or settings, producing sharply different results.
But the use of synthetic respondents raises a deeper issue. Surveys are not just prediction tools. They are measurement tools meant to capture what people actually think. A thermometer measures your temperature directly. You would not trust one that estimated your temperature by consulting an AI model instead.
Jose Carlos Cerdeno Martinez via Getty Images
Large language models and other AI tools inherit biases and blind spots from the data they train on. For example, AI can oversimplify or distort opinions from groups of people who are underrepresented online. Traditional polling also has biases, but many biases in modern AI systems are hidden from public view inside closed proprietary models. To make matters worse, pollsters may present results from synthetic respondents to the public as if they came from surveys of people.
These shortcomings can erode trust in polls and survey research. They also raise an interesting paradox. Synthetic data, created by computers or simulations, is widely used in modern AI. It helps train AI systems for medicine, finance, robotics, self-driving cars and other disciplines. So why do synthetic survey responses seem more problematic?
The key difference is that synthetic data is checked against reality. A self-driving car may train on synthetic images and videos of different road conditions, but an automaker would never deploy the car on public roads without extensive real-world testing. If synthetic data hurts performance, engineers can correct, retrain or replace the system.
Researchers may treat synthetic survey responses as public opinion itself, but the system is not measuring public opinion. It is running a simulation of public opinion based on data it was trained on. If the simulated opinions distort reality, researchers may not realize it until flawed conclusions have already shaped public policy, business decisions or scientific research.
More efficient design and analysis
Nevertheless, there are ways AI can help survey research without weakening the measurement of public opinion. AI tools can help survey researchers write clearer questions by simplifying wording, reducing ambiguity and eliminating repetition. They can help avoid unnecessary questions, making it easier for people to respond. These tools can also adapt surveys across languages.
Once a survey is done, AI can help researchers organize large volumes of open-ended responses, summarize recurring themes and handle incomplete surveys more efficiently than human analysts. Some researchers are exploring hybrid approaches that combine smaller human surveys with AI-assisted analysis.
Decision makers use surveys and polls to listen to and understand the voices of people affected by their decisions. Replacing human respondents with synthetic respondents risks weakening that connection. At the same time, falling response rates and rising costs are real survey challenges.
I’m confident that further research can find ways to use AI transparently and effectively, in a scientifically defensible way, without replacing people.
Ambuj Tewari receives funding from NSF and NIH.
