MarketTools Blog

Experienced researchers have long assumed that online survey design, respondent engagement, and online research data quality are interrelated.  For example, it seems obvious that long, complex questionnaires will likely be viewed unfavorably by respondents, and will increase the likelihood of “bad” respondent behaviors such as speeding and partial completes.  Along the same lines, it is reasonable to suppose that sub-optimal respondent behavior and experience may expose potential issues with the quality of data in these surveys.  For these reasons, it is commonly accepted that the primary objective of a well-designed survey is to elicit the most accurate and representative possible responses from the respondents.

Recent studies conducted by MarketTools clearly show that survey design influences the respondents’ experience in a survey, which manifests itself in how they behave while answering survey questions – and that the level of respondent engagement directly impacts the quality of respondent data.  If respondent engagement is low, data quality is also more likely to be low.  Additionally, complex survey design can cause respondents to become unengaged and increase the odds of unreliable data. 

How can you tell if an online survey is well-designed?
TrueSample has developed tools that provide a comprehensive, objective measure of survey engagement to help researchers see the impact that their survey design has on respondent engagement.  TrueSample SurveyScore® uses a proprietary mathematical formula for looking holistically at respondent engagement – using both quantitative behavioral metrics (such as survey abandonment and speeding) and qualitative experiential metrics (such as respondents’ ratings of the survey-taking experience) in a given survey. 

• Survey Rating:  how the respondent perceived the survey experience, self-reported based on a standard scale.
• Partial Rate:  the percentage of respondents who abandoned the survey before completing it.  
• Speeding:  the level of speeding behavior exhibited by the survey respondents.

Together, these metrics comprise a SurveyScore, which is assigned to every TrueSample-enabled survey. The SurveyScore also provides a way to consistently evaluate respondent engagement across surveys, in varying product categories, and across different research methods.  You can compare the results of two surveys and know that the one with the lower SurveyScore probably has lower data quality. 

Can you check survey design quality before launching a survey?
TrueSample SurveyScore Predictor helps researchers optimize survey design before it is launched, allowing them to estimate and improve respondent engagement for an online survey.  This tool calculates an estimated SurveyScore based on survey design elements such as types of questions asked and complexity of questions – and then will suggest which particular elements should change to improve your SurveyScore and lead to better engagement levels. 

In addition to keeping “good” respondents engaged, good survey design also has the additional benefits of lowering field times, reducing costs and, given the finite supply of available online panelists, helping to maintain online panel health throughout the industry.

We believe that survey engagement measurement tools such as TrueSample SurveyScore are critical for researchers making business decisions based upon their survey results.  By measuring and benchmarking the respondent engagement level of each survey before and after the survey is launched, researchers can improve survey design to increase respondent engagement and enhance data quality.

For more information on the MarketTools respondent engagement study, see the article reprint from Quirk’s Marketing Research Review: Designed to engage: What is the impact of survey design on respondent engagement?