Research We Read in 2025 And Just Can’t Stop Thinking About

By Kevin Collins

Happy New Year! Before we start 2026 in earnest, we wanted to share with you some research from 2025 that will continue to be relevant in the coming year

LLMs and Survey Research

One persistent theme of methodological research in 2025 – in fact, the theme of the 2025 meeting of the American Association for Public Opinion Research – was the use of LLMs in surveys. LLMs has been a focus for us as well. Last year, we conducted a survey on the public’s view of generative artificial intelligence, presented at the AAPOR conference about respondent’s views of automated interviewers, wrote a lengthy blog post synthesizing research on LLM-powered synthetic respondents, and did a podcast interview on the topic. But others’ work on the topic has really stuck with us as well. 

First, G. Elliott Morris and Verasight have done a number of studies this year on synthetic respondents. The first in their series, in particular, really sticks with us. It illustrates how poorly models do at “out of sample” questions – questions that haven’t been asked before and thus cannot be in the preexisting model training data. 

Second, Sean Westwood’s investigation into how LLM-powered respondents can successfully masquerade as online panelists alarmed many in the survey industry. This study illustrated how many conventional tools to identify respondents, such as attention checks, reverse shibboleths and reCAPTCHAs, can be defeated by LLM-respondents, highlighting the need for innovation in the face of this challenge. However, there still appear to be some effective tools for researchers using panels. Specifically, Westwood concluded his article with several suggestions, such as IP-based location verification, a practice also suggested by another study that came out earlier this year from James Matherus and co-authors (all associated with Morning Consult). That other study also recommended copy-paste detection and device fingerprinting, practices that are in routine use by some vendors in the field.

Third, we have also been following closely the development of LLM-based tools for coding open-ended responses. Two papers, one pre-print by demographer Chris Soria and a just-published paper in Survey Research Methods by a team of German sociologists, both offer examples of this approach, but also show that performance differs by model selection, fine-tuning, and characteristics of the question being studied.

Sample Quality

Aside from LLMs, a second pervasive challenge facing the survey industry is ensuring accurate samples. In an era of low-response rates, probability samples suffer from growing concerns about the reliability of some sources of opt-in panels, education polarization, and associated non-ignorable non-response. These are concerns that have motivated our own work, such as research on Dynamic Response-rate Adjusted Stratified Sampling and the integration of opt-in panel responses with probability text-to-web respondents.

A number of papers tackling these questions, from a variety of approaches, in 2025.  Michael Stagnaro and co-authors conducted a detailed comparison of nine different opt-in online panels, evaluating not only data quality both in terms of response validity metrics as well as sample composition and respondent professionalism metrics.  

Respondent professionalism was the topic of another paper published in Political Analysis last year by Bernhard Clemm von Hohenberga and co-authors. They used web-browsing data to evaluate professionalism directly rather than via self reports, and found that in 2019 when the data were collected, the prominent panelist marketplace Lucid (now Cint) had professional respondents comprising anywhere from between 34% to 72% of its sample, depending on the measure evaluated. 

More optimistically, Michael Bailey and Jing Peng, in a paper posted at SSRN, have developed a method for for leveraging panel data (meaning surveys with repeated attempts to interview the same respondents, rather than companies for sourcing those respondents) to empirically estimate the association between interest in survey participation and the outcomes of interest within a random-effects framework to at least partially address the pervasive issue of non-ignorable non-response biasing estimates of election polling. 

Beyond statistical methods to address under-represented groups, there is also the question of how to ensure the inclusion of those groups at the design stage. Newly up at Survey Practice, Arina Goyle, Susan Sherr, and Cameron McPhee have a paper investigating a new way to increase the proportion of respondents from populations who are under-represented in typical Address-Based Sample surveys, such as low-education respondents, Hispanic and Black respondents, and younger adults. They find that adding pre-paid phone sample to address-based samples can help correct some of these imbalances.

Questionnaire Design

Finally, we are always on the lookout for new research for the best way to design questionnaires. A few studies from the past year stood out from that category.

First, we cannot recommend strongly enough Nicholas Carnes and Geoffrey Henderson’s study in the British Journal of Political Science on the importance of attention in moderating in-survey-experiment treatment effects and translating results to the field. After failing to generate an effect in a mail experiment on climate change attitude, they did another round of survey experiments and found that making content skippable dramatically reduced the treatment effects, closer to the magnitude of effects observed in the field. In doing so, they show that conditioning on attention in survey experiments risks answering the wrong question; applied researchers must also address the threshold question of what gets voters (and respondents) attention in the first place.

We also loved Mads Andreas Elkjær and Christopher Wlezien’s paper on the use of “don’t know” options published in 2025 in Political Science Research and Methods. Contrary to conventional wisdom arguing against such options, the researchers  find that including such options increases respondents’ confidence in their own answers, and also alter estimates of majoritarian support for policies by not forcing people who do not have an opinion to express one.

And last but not least, we recommend Matthew Graham’s continued contributions to the “how to write good surveys” literature with his study on catch questions, published last year in Public Opinion Quarterly. Using web browsing data, he shows that offering “catch questions” in political knowledge batteries – questions that no reasonable person could answer, and correct answers to which would demonstrate cheating – actually induce cheating, or at least increase the proportion of respondents who switch windows during the survey .

Did you read survey methods research last year you just cannot stop thinking about? Email us at info@survey160.com and let us know what captured your attention.