Three principles for emerging uses of AI by survey researchers.
By Kevin Collins
Generative AI tools in survey research are what everyone in the industry is talking about, from AI interviewer firm Listen Labs’ $2 billion acquisition by Salesforce to Pew Research’s blistering takedown of synthetic samples. Synthetic samples (or silicon samples, or digital twins, or personas) have gotten the most attention, in no small part because they are being rolled out as a commercial product very quickly. Firms with their own panel assets have been quick to role out such products as another strategy to monetize the underlying panel asset, including YouGov, Gallup, Ipsos, Qualtrics, and others, and other new firms offering parallel products have launched, sometimes with eye-popping valuations.
But as we pointed out in an essay last year, the public research does not suggest that these products can do what they claim, nor that frontier models are doing better at this task as they develop. And research since that time has largely agreed with our more skeptical takeaway.
Moreover, the proliferation of these tools (somewhat paradoxically) creates value of survey and other first-party data, because there is demand for these tools, they offer secondary ways to monetize that data asset. Rather than silicon samples undermining the value of actual survey measurement, they may well increase it.
Furthermore the failure of silicon samples to live up to their promoters’ hype to date does not mean that this technological advancement is irrelevant to the work of survey research; far from it. We are focused on how LLMs can improve productivity and performance across a variety of steps of the survey process. To that end, we offer three principles we recommend to the survey research community as you think about integrating generative AI into your workflows.
Principle 1: Keep the researcher in the loop
The idea of keeping a “human in the loop” is broadly recommended as a recognition that LLMs make errors and humans – expert humans, even – need to catch them. But we view the “researcher in the loop” not only as a mechanism of quality control, but division of labor to leverage the comparative advantages of both human researcher and generative AI assistant. Some examples of how we use generative AI at Survey 160 illustrate this process
Coding open-ended responses. We have found that current frontier models do a sufficiently good job coding open ended responses that they can be relied upon – with appropriate checks. As we describe in detail here, we have standardized (using a Claude skill) a process by which humans do the meaning-making, and the machines do the counting.
Advising on questionnaire design. With the iterative assistance of Claude, we have developed a 59 point best practices guide, drawing on 91 separate journal articles, for writing survey questionnaires. We then built a Claude skill to review a written questionnaire against this set of best practices and provide suggestions for possible improvements. LLMs are good at comparing text to a large corpus, but human experts make final judgments about when those suggestions are appropriate and when they are not.
Assisting in quality-control checks. One thing machines do well is clicking through a web survey many times very quickly. This comes in handy when doing quality control review, making sure that the programmed survey text matches the existing questionnaire. This is particularly valuable when checking skiplogic and randomized survey elements, which require many different runs through the survey to evaluate. Again, we have built a Claude skill to help identify discrepancies between the survey as written and survey as programmed. Sometimes the “errors” it flags are false positives, and it is never the final review. But adding this as an initial layer of review has expedited our extensive quality control process.
Principle 2: Advance the Pareto frontier of data quality and respondent user experience.
Depending on how you look at it, data quality and respondent user experience can often be viewed as a tradeoff. For example, a longer battery can measure a concept more precisely but impose greater burdens on survey respondents. As researchers incorporate generative AI into the survey-taking experience directly, the goal should be to simultaneously optimize data quality without compromising user experience and optimize user experience without compromising data quality. Newly-published scholarship illustrates how researchers are navigating this tradeoff.
AI-assisted speech-to-text offers richer responses. Open-ended responses can be data goldmines, but they can be burdensome to type out. In a newly published paper, Camille Landesvatter and Paul C Bauer find that offering a speech-to-text option for open-ended responses produces longer and more informative answers. AI technologies like Wispr Flow create far higher fidelity transcription than you find in your typical smartphone, and offer a path for better open-ended responses.
Tailoring surveys with AI to make them more relevant to respondents. In a series of papers, Yamil Velez and co-authors have found ways to use LLMs to incorporate information from open-ended responses to make subsequent questions more relevant to the particular concerns of those survey-takers. For example, Velez has used information from open-ends to tailor the content of conjoint experiments later in the survey, and with co-author Patrick Liu, to experimentally test count-arguments to issue positions expressed by the respondent. In another study, Velez and co-author Alec Ewig use retrieval augmented generation to ask respondents about specific bills relevant to the policy concerns they named earlier in the survey. In this way, surveys can be more relevant, shorter, and more informative with these types of uses of generative AI.
AI-generated prompts to respondents can make responses better, but also annoy respondents. Amanda Austin and colleagues tested AI-generated follow-ups to open-ended questions, and found that these prompts generated richer and more specific responses, and while self-reported satisfaction was lower, those differences were not statistically significant. But they also found that these treatments produced more polarization in later questions, raising concerns they affected how respondents answered questions. Soubhik Barari and his NORC colleagues tested a similar sort of AI generated probing, looking at three types of prompts: confirmation probes, elaboration probes, and “relevance probes” that asked respondents to re-word unintelligible answers. They similarly found improvements in data quality, but in their case there were significant drops in respondent satisfaction (except for the confirmation probes), and for one question, increases in survey drop off. This research highlights how AI-generated probes in surveys can face the dilemma between data quality and respondent user experience. But it also points at future, yet-untested directions such as distilling multiple options from open ends (as opposed to simple yes/no confirmation).
Principle 3: Move from “Most Powerful Model” to “Best Model for the Job”
We are looking for ways to consistently choose the best model for the job. Sometimes those will be large, general language models, which are comparatively slow and expensive, but increasingly they are more task-specific models which are faster and cheaper, while being equally good at some tasks. To be sure we have found that for some needs, such as classifying responses to open-ended questions, frontier models outperform their more diminutive cousins, this is not always true for all cases. For example, demographer Chris Soria has found that ensembles of lower-tier models can beat frontier models for some open end response classifications.
Furthermore, newer models like Jev from TypeSafe AI that are designed for specific tasks (in Jev’s case, selection of options rather than generation of text) show much faster speed and lower cost for comparable performance on relatively simple classification tasks. As we move forward, we are looking at how lightweight and post-trained open-weights models can solve problems faster and cheaper without sacrificing quality. What’s more, the frontier models of today that work well will continue to work well when those models are no longer the frontier.
We endorse the disclosure recommendations from AAPOR’s taskforce report “Responsible AI Integration in Survey Research,” which call for researchers to disclose which models they use (along with many other aspects of the use of AI in survey research). Only by sharing what models work – and which work “well enough” – can we collectively advance the industry’s use of this promising if imperfect technology.
