How long should we stay in the field? (Part II)

In our last blog post we summarized the result of half one of our presentations at last year’s American Association for Public Opinion Research (AAPOR) annual conference. Today we return with part II, looking at how long to leave text to web surveys open.
In an experiment in the lead up to the 2023 Kentucky Governor election, we fielded an experiment randomizing a voter-file-sampled list to either receive text messages (or for landlines, IVR calls) or live interviewer calls. We wrote about some other findings from this survey previously. In the first round we let the initial text messages last a couple of days before sending a follow-up to non-responders. 

Takeaway #4: Most responses come in quite quickly. The dotted line here is 10 hours, which is the field period of the first day, so anyone who responded later than that responded to the survey after the initial day in the field

But how do the people who answer later differ from those who answer quickly?

Takeaway #5: Leaving the survey open longer adds more “easy to reach” respondents. In this case, we compared responses within 10 hours of the initial contact versus those who responded later. The later respondents were significantly  whiter and more engaged respondents (the latter at p<0.1 marginal level of statistical significance) .

However, these additional responses lower cost per interview – because no additional texting is required to obtain them – and they do help produce more accurate estimates after weighting as well.

In summary, longer response windows produce less representative raw data, but lower costs and (with the additional responses) improve accuracy after weighting.