When not to use MMS for surveys

An increasingly common way to use text messaging for surveys is not SMS (short message service) but MMS (multimedia message service). Some believe that MMS allows survey firms to share an image like a logo to increase their credibility. Others are attracted to the longer number of characters within a single MMS segment, despite generally higher costs per segment (though, that may be a highlight for other firms selling MMS services). And others may use MMS out of a misguided belief that it increases deliverability rates; several years ago MMS could be sent via lines not registered with the carriers’ 10DLC system, though that no longer is true. 

What this discussion has largely lacked is an empirical discussion about whether MMS is more effective at encouraging survey participation than SMS – and if it is more effective, whether that efficacy offsets increased costs. 

At Survey 160 we have performed such experiments, repeatedly. On the basis of those findings, we recommend against using MMS, or at least against using it to show a brand logo of the survey firm. 

A case in point is an experiment we ran in the run up to the 2024 presidential elections in three states, Arizona, North Carolina, and Pennsylvania. We included an MMS vs. SMS experiment in the first few days of the fielding, in which we sent otherwise identical messages with or without a logo for the brand identified as fielding the survey. Importantly, the name of the brand was listed in the message; all that was changed was whether a logo was also attached. 

Results
The reason that this experiment was only included in the first few days of fielding is that we chose to end the survey early after seeing the dramatic negative effects of MMS on fielding performance. The results (marginal effects from a regression with state and day fixed effects) are in the graphs below.

We are attentive not only to average treatment effects on participation, but also on any distributional effects resulting from heterogeneous treatment effects on response rate across groups. To evaluate this possibility, we also ran a version of this regression in which treatment was fully interacted with four traits measured with voter file measures: non-voting in 2020, being white, being male, and being a Gen Z or Millennial. Each of these binaries represented about half of the samples, except for non voting which was about 35%. Of these, only gender showed a significant interaction with the MMS treatment in predicting completion rate, with women showing approximately zero effect and men showing a sharply negative effect of the MMS message relative to the SMS message. 

Meta-Analysis

We do not make blanket recommendations based on a single experiment, but rather through repeated experimentation and systematic review of those experiments. The best way to do such an assessment is through a meta-analysis. Below, we document the results of a random-effects meta-analysis of four experiments we can report publicly (the experiment listed above is Study 4). Each blue box represents a point estimate and associated confidence interval for a specific experiment. The green diamond shows the range of plausible effects we can estimate from combining the experiments. In order to justify the additional cost, the effects of MMS would need to be meaningfully positive, which that green diamond shows we can rule out.

Recommendations and Next Steps

We are always open to trying further experiments – your brand logo with your population of interest might work better – but based on our past experience we doubt it will have enough of an effect to offset the greater cost of MMS messages. 

However, as always, the research continues. Going forward we will be examining other types of images, such as infographics or images of people taking surveys, to see if those can make requests for survey participation more effective. And to be sure, there may be particular use cases where the benefits for MMS over SMS are both warranted and empirically observable. 

But in any case, we recommend all survey researchers using texting – whether with us or with other vendors – to ask for data on the effectiveness of MMS before agreeing to spend more to send images, whether those images are brands or something else entirely.