How long should we stay in the field? (Part I)
In an earlier newsletter, we wrote about when to start fielding surveys. Today we want to share some research, presented at last year’s AAPOR conference, on when to stop fielding.
The persistence and asynchronous nature of text message communication makes this a compound question: how long should we leave open the online component for a text-to-web survey, and should we send follow-up requests for participation among non-responders? More formally, we asked two questions:
What are the costs and benefits of multiple rounds of attempts to recruit respondents by texting, as opposed to a single round?
What are the costs and benefits of allowing longer response windows?
In an experiment in the lead up to the 2023 Kentucky Governor election, we fielded an experiment randomizing a voter-file-sampled list to either receive text messages (or for landlines, IVR calls) or live interviewer calls. We wrote about some other findings from this survey previously. In the first round we let the initial text messages last a couple of days before sending a follow-up to non-responders.
Takeaway #1: On SMS, most respondents come from the first attempt, unlike phones where the responses are more evenly balanced across attempts. That’s somewhat intuitive, since unlike phones, you don’t need to immediately answer the survey the moment you are contacted in order to answer it on the first round. This means that a second round of text messages does increase the overall cost per interview (while still being far, far less expensive than phones were, at least in this particular project).
| SMS | Round 1 | Round 2 | Combined |
|---|---|---|---|
| N | 448 | 117 | 565 |
| Response Rate | 1.5% | 0.5% | 1.9% |
| Cost per complete | $6.35 | $15.65 | $8.19 |
Takeaway #2: People who respond on the second attempt have a somewhat lower survey response quality. In particular, we found that second-attempt respondents are somewhat more likely to straightline on favorability questions (the effect is small but statistically significant).
So two rounds of survey attempts on SMS produce more costly data and add lower-quality data. But are those costs worth it? Often, yes. Here’s why.
Takeaway #3: Multiple rounds produced more representative and accurate data. People interviewed on the second attempt by SMS disproportionately come from harder-to-reach populations: younger and less politically engaged. These differences are small and non-significant on phones.
Furthermore, the absolute error and bias is lower in the weighted sample when adding in additional responses from the second attempt.
To summarize, we found that multiple rounds of attempts produce more representative and more accurate data, though, somewhat higher costs, slightly more straightlining. This was notably different from phones, which have more similar response rates and resulting sample characteristics across rounds of attempts in our study.
This research has not only informed our fielding best practices, but also our software development. For example, we are building tools to streamline the process of re-attempts within SMS surveys.
