MarketTools Blog

MarketTools research has shown that survey design parameters directly impact respondent engagement.  This essentially means that by modifying the survey design parameters, we can have an effect on engagement and, correspondingly, impact data quality – and turn a “bad survey” into a “good” one. 

A previous blog post from Brenton Wells talked about SurveyScore, an important component of the MarketTools TrueSample data quality process, and how it measures respondent engagement level to help researchers optimize the design of surveys (learn more about SurveyScore).  SurveyScore Predictor is a tool that gives researchers a means to adjust and improve design parameters in order to optimize the SurveyScore results for a survey.  When the SurveyScore is high, it’s more likely your survey respondents will complete your survey and give your questions the considered response you’re looking for, leading to higher-quality survey results.

A SurveyScore is scaled from 0-100, with 100 being the highest possible score, representing the highest relative level of engagement, and 0 being the lowest possible score, with the lowest relative level of respondent engagement. The chart here gives a typical distribution of SurveyScore for the 1600+ surveys that were used to build the score.  (Note: this distribution may change over time as more surveys are added to the system.)

We’re often asked what type of surveys typically tend to score high and what type of surveys score low.  While the problem is multi-dimensional, which means that the answer to that question cannot be answered by one parameter alone, we can nonetheless try to look at the partial dependencies of the score on some of the main parameters.  The following table provides some typical design parameter values in three ranges of SurveyScore results:

As you might expect, the more complicated the survey, the lower the score.  A large number of survey questions, longer questions, and very cumbersome matrix questions all can lower the engagement level of the individual taking the survey.  However, it is interesting to note that as the SurveyScore result number increases, the mean number of matrix attributes in the surveys does not decrease correspondingly.  But the total number of matrix attributes does indeed decrease with the increase in score, which is more a function of the overall number of matrix questions in the survey.  That means that in order to improve a survey so it scores in the mid-range rather than at the bottom, it’s OK for the typical matrix question to have as many attributes as those in the low-scoring surveys, as long as there are not too many matrix questions overall.  (Note: This analysis is by no means an indictment of matrix questions, which are essential to researchers – it is more a guideline for how to create a survey with limitations on these questions while keeping the respondents satisfied and engaged.)

For surveys that receive low SurveyScores due to length, consider conducting multiple shorter studies instead of one long survey, and weigh the cost implications against the improved data quality implications that come with higher SurveyScores.  In typical scenarios, you can conduct pilot projects to evaluate the differences in data between the two formats to help you arrive at the final decision.

As always, the goal of a well-designed survey is to elicit the most accurate and representative possible responses from the respondents.  With SurveyScore to indicate what’s “good” and “bad” about a survey, researchers are better able to improve the survey design to increase respondent engagement and enhance data quality.

While Security has long been near the top of priority lists for most companies dealing with technology, Privacy has been slower to climb the priority ladder. Fortunately, this is starting to change: the growth of the International Association of Privacy Professionals (IAPP) is one indicator of this. For the Market Research industry this new focus on privacy has also been clear, and the Global Privacy Webinar Series sponsored by CASRO (Council of American Survey Research Organizations) has been catering to the increased interest. But what do we mean when we talk about “Privacy”? And why should there be growing interest?

What is Privacy?
The American Institute of Certified Public Accountants defines Privacy as encompassing “the rights and obligations of individuals and organizations with respect to the collection, use, retention, disclosure, and disposal of personal information.” For a firm operating in the Market Research industry, key aspects of those privacy obligations will usually depend upon whether the firm is acting as a data controller or data processor, the nature of the data itself (e.g., health, financial, children, basic contact and/or service usage) and the purposes for which the data is being used (e.g., fraud detection). Unfortunately, few regions share exactly the same definitions of these and other important privacy concepts within their laws, so great expertise is required when evaluating compliance in a global setting.

As a side note, those new to Privacy may easily confuse it with Security, a separate but related discipline. Whereas Security is needed to ensure Privacy, the reverse is not true. For example, a Privacy audit may be concerned with whether one is storing health information, giving proper notice about this to users, etc., while a Security audit would be more focused on how that information is actually transferred and stored.

Why the Interest in Privacy?
The interest in Privacy is due to more than just the public perception issues raised by high-profile data breaches (see the consumer education website operated by the Privacy Rights Clearinghouse). It is also driven by concern for protecting consumers from identify theft, meeting client expectations, regulatory compliance, insurance and other liability considerations, etc.

We on the TrueSample Team also take into account privacy considerations in the context of the need for market research data quality – as researchers need to ensure that the respondents taking their surveys are who they say they are.  These researchers need a stamp of proof for each respondent's authenticity – proving that he or she is "real" and is qualified to participate in market research studies – and TrueSample uses personally identifiable information (PII), such as name, address etc., to validate sample.  The TrueSample team maintains a commitment to handling PII in the most responsible way to protect privacy and confidential information.   

To learn more about Privacy, see the following resources:

• International Association of Privacy Professionals (IAPP):  https://www.privacyassociation.org/
• CASRO Global Privacy Webinar Series:  http://www.casro.org/techform/2011-global-webinar-purchasing.cfm
• Privacy Rights Clearinghouse:  http://www.privacyrights.org
• Bureau of Consumer Protection – Safe Harbor Program:  http://business.ftc.gov/privacy-and-security/children%E2%80%99s-online-privacy/safe-harbor-program
• Data Privacy Day:  http://dataprivacyday2011.org/

Experienced researchers have long assumed that online survey design, respondent engagement, and online research data quality are interrelated.  For example, it seems obvious that long, complex questionnaires will likely be viewed unfavorably by respondents, and will increase the likelihood of “bad” respondent behaviors such as speeding and partial completes.  Along the same lines, it is reasonable to suppose that sub-optimal respondent behavior and experience may expose potential issues with the quality of data in these surveys.  For these reasons, it is commonly accepted that the primary objective of a well-designed survey is to elicit the most accurate and representative possible responses from the respondents.

Recent studies conducted by MarketTools clearly show that survey design influences the respondents’ experience in a survey, which manifests itself in how they behave while answering survey questions – and that the level of respondent engagement directly impacts the quality of respondent data.  If respondent engagement is low, data quality is also more likely to be low.  Additionally, complex survey design can cause respondents to become unengaged and increase the odds of unreliable data. 

How can you tell if an online survey is well-designed?
TrueSample has developed tools that provide a comprehensive, objective measure of survey engagement to help researchers see the impact that their survey design has on respondent engagement.  TrueSample SurveyScore® uses a proprietary mathematical formula for looking holistically at respondent engagement – using both quantitative behavioral metrics (such as survey abandonment and speeding) and qualitative experiential metrics (such as respondents’ ratings of the survey-taking experience) in a given survey. 

• Survey Rating:  how the respondent perceived the survey experience, self-reported based on a standard scale.
• Partial Rate:  the percentage of respondents who abandoned the survey before completing it.  
• Speeding:  the level of speeding behavior exhibited by the survey respondents.

Together, these metrics comprise a SurveyScore, which is assigned to every TrueSample-enabled survey. The SurveyScore also provides a way to consistently evaluate respondent engagement across surveys, in varying product categories, and across different research methods.  You can compare the results of two surveys and know that the one with the lower SurveyScore probably has lower data quality. 

Can you check survey design quality before launching a survey?
TrueSample SurveyScore Predictor helps researchers optimize survey design before it is launched, allowing them to estimate and improve respondent engagement for an online survey.  This tool calculates an estimated SurveyScore based on survey design elements such as types of questions asked and complexity of questions – and then will suggest which particular elements should change to improve your SurveyScore and lead to better engagement levels. 

In addition to keeping “good” respondents engaged, good survey design also has the additional benefits of lowering field times, reducing costs and, given the finite supply of available online panelists, helping to maintain online panel health throughout the industry.

We believe that survey engagement measurement tools such as TrueSample SurveyScore are critical for researchers making business decisions based upon their survey results.  By measuring and benchmarking the respondent engagement level of each survey before and after the survey is launched, researchers can improve survey design to increase respondent engagement and enhance data quality.

For more information on the MarketTools respondent engagement study, see the article reprint from Quirk’s Marketing Research Review: Designed to engage: What is the impact of survey design on respondent engagement?  
 

The primary question asked of us at conferences or at other forums is “Are TrueSample-excluded survey respondents really “bad” respondents, or are they just different?”

TrueSample excludes online survey respondents that are “not real”, “not unique” and “not engaged”. The first and second have to do with the respondents being verifiable and not being duplicates. The third has to do with their performance in surveys – do they speed through the survey relative to other respondents, or do they straightline their responses? In all three situations, the online survey respondents are different from others in characteristics that are separate from their survey responses. In other words, they are outliers, but they are classified as such not because of how they answer the survey questions, but because of other characteristics that they exhibit.

Therefore, the obvious question that we are asked is why we would assume their data is not of high quality and therefore discard what could potentially be valuable information.

To answer that question, we need to consider the underlying problem: we start with the belief that there are, in fact, respondents who are intent on gaming the system and therefore provide less-than-truthful responses, thereby compromising online research data quality.

Starting with that assumption, the next step is how to identify them. We are definitely in unsupervised modeling land here. There are no tags that we can train a supervised model with, telling us what a “bad” respondent is. Supervised modeling is out of the question for this type of quality control – there is no cost effective way to identify a set of “bad” online survey respondents for model training.

So we do what we feel is the next best thing: we identify a set of undesirable characteristics, such as not providing verifiable information, like name/address (considering that is the only survey-agnostic information asked on a survey that we can verify) or speeding/straight-lining through an interview.

We feel strongly that online survey respondents that exhibit these undesirable characteristics are more likely to give data of poorer quality. And since our research (see the white paper on “What Impact do Bad Respondents Have on Business Results”) consistently shows that they provide data that is biased compared to the data provided by the individuals that do not exhibit these characteristics, we feel that the decision to exclude online survey respondents is the correct one.

There is a very valid argument made that the percentage of respondents that we call “bad” is more than small in some cases and in certain demographics. We agree that there are “good” respondents in the discarded pile that may have been excluded, for example, because their names and addresses are not verifiable for legitimate reasons, or because they think and move so quickly that they are in fastest few percentiles across a surveys and have been identified as “speeders”.

But do these people make up the majority of excluded respondents? If so, wouldn’t the data of the excluded people be closer to the data of the “good” ones? Is there something about these legitimately unverifiable people that causes their data to be in the same cluster as the gamers? We do realize that we are likely removing some respondents that are good – i.e. we are committing Type I errors. Having recognized this likelihood we are, in fact, continuing to conduct research to reduce these errors.

The questions that are asked of us are valid. If you truly believe that all online survey respondents are above-board, then in your view, such quality control measures are unnecessary. However, if you believe that there is a percentage of respondents that game the system and that provide questionable data, then we at TrueSample believe that we have a defensible quality control methodology. We also recognize that there is always room to improve a method and reduce errors. To this end, we are continuing to conduct research in this area and will continue to share our findings as we learn more.

We’re pleased to announce the rollout of the following new capabilities in TrueSample, designed to provide researchers and clients with more powerful capabilities to ensure the quality of their online market research sample.

As Twitter continues to gain popularity as means for self-promotion, the need for "account verification" becomes stronger and stronger. For any one legitimate account, particularly those of celebrities or other famous people, there may be tens of copy cats. It's difficult, if not impossible for Twitter users to discern which account belongs to the real person they are trying to follow.

As Twitter describes it, "Verification is currently used to establish authenticity of identities on Twitter. The goal of this program is to limit user confusion by making it easier to identify authentic accounts on Twitter." Once an account is verified, a little blue checkmark appears next to the user's Twitter ID - serving as trusted proof that this person is who he or she says she is.  

This need for verification isn't dissimilar to the need that market researchers have to ensure that the respondents taking their surveys are who they say they are. Researchers need a stamp of proof for each respondent's authenticity – proving that he or she is "real" and is qualified to participate in market research studies.

Interestingly, Twitter is very ambiguous about the methods and technologies it employs to perform account verification. It seems reasonable that the company would keep its policies secret to prevent "gaming" of the system. It's possible that they even use different and subjective criteria for specific circumstances and users. After all, if users were to know exactly what criteria Twitter required accounts to meet in order to be verified, they would surely come up with ways to meet those requirements.

In the market research world, such ambiguity is a hard pill to swallow. Researchers, by nature, don't like not knowing exactly what criteria and data points are being used for decision-making.  And they expect to see research on research indicating why such criteria were selected and how it impacts their data. I think this is a completely reasonable expectation.

But there certainly is a valid business case for using a ‘black box” approach to validation of online market research survey respondents. Just as Twitter doesn’t want its users to know what criteria they use for verification, researchers shouldn’t want online survey respondents to know what criteria they use either. A market research data quality solution must be “opaque” to the survey respondents so that they cannot identify ways to skirt the quality checks in order to be considered valid for a survey. Either the quality solution should be completely hidden to the respondent – meaning they don’t know that a solution is in place at all – or it should appear random and inconsistent and maybe even subjective.

This string of thought leads me one step further…. Is there a case for keeping research quality criteria opaque to not only online survey respondents but also to the researchers conducting them? Can you imagine a circumstance where a research supplier might also be aligned to “game” a research quality system in order to finish a project for an end client? Perhaps a supplier would need to skirt the quality checks in order to be able to fill the respondent quotas for a survey? Or perhaps they would work around the quality solution in order to close a project more quickly and cost-effectively? So, that leads me to another question:

If a reputable, third-party research quality solution were to validate survey respondents using a "black box" approach that did not reveal the methods or techniques being used for validation -- would you accept this approach?

I’d love to hear others’ perspectives on this subject, so if you have any thoughts, leave me a comment.
 

I'm always surprised when I hear a research buyer say that they believe their research vendors are meeting data quality standards on their research even though the vendor has provided no evidence to support this claim.  I've even heard buyers say that their vendors are charging for and implementing TrueSample on every research project, but the end client has no evidence of this implementation. In fact, we've confirmed that, despite their claims, some of these vendors were not using TrueSample.

Buyers do not have to fall prey to these false claims. Buyers must hold vendors accountable for quality, and demand proof that online data quality standards and techniques are being implemented on every research project. If your vendors tell you that they are using TrueSample, then you should request that they share the TrueSample.net reports with you for all projects.

Here's an example of a TrueSample report:

TrueSample reports provide a simple and easy way for the vendor to clearly demonstrate that they've improved your research quality.  And it's a simple way for you to feel satisfied that all respondents in your projects meet the standards to deliver the research results you need.

• For more best practices on working with research vendors to ensure the quality of research data, check out the Buyer's Guide to Online Research Quality.

Hate the player or hate the game?  Depending on where their loyalties lie, people may fall into either camp when they consider the issue of how to maintain the data quality of online research – either the respondent is the problem or the survey is the problem. 

Intuitively, one would think that the answer is that it is a little bit of both.  This is what we on the TrueSample team at MarketTools have been saying as well, and we have been approaching the problem from both directions.  To that end, we have made a pair of bookend presentations at CASRO - at the CASRO Panel Conference in New Orleans in February 2010 and the CASRO Online Research Conference in Las Vegas last week (March 3, 2011):  

• How to Influence Good Behavior through Survey Design
• How to Identify and Exclude “Bad” or Unengaged Respondents to Preserve Data Quality
  (See the recap video from CASRO, below.)

But what has been unavailable is real evidence on whether there actually are two types of respondents – the “bad” ones and the ones “driven bad” by the survey.  Until now.  In our recent research on engagement, we looked at the data in a slightly different way that points to this dichotomy in respondent types.  While this is not a smoking gun situation, I would say that it falls under a scenario where the barrel is warm and someone has a guilty look on their face.

In this research, we analyzed over 1600 surveys that spanned several months and product categories, and tracked several hundred thousand respondents over time.  About 20,000 of these respondents had been unengaged at least once (unengaged in the TrueSample definition refers to a respondent that either sped relative to the norm on over 40% of the pages in a survey or straight-lined while speeding on over 25% of the pages).  We looked to see at what point in the survey they started speeding and plotted that against the survey length.  In the analysis, we compared the unengaged respondents against those that sped on at least one page in the survey, but not enough to be marked as unengaged.

If there is only one class of unengaged respondent, i.e. those that are driven bad through survey design, I would expect to see a trend of the first time of speeding relative to the length of the survey – i.e. as the survey gets longer, unengaged respondents on average tend to speed later and later.  But the finding was quite remarkable in displaying the distinction between the two types of respondents. 

As can be seen in the figure below, the respondents that were unengaged in the surveys started to speed right off the bat, and obviously, the length or design of the surveys had no bearing on when the speeding started.  However, the respondents that sped on at least one page but were still engaged overall tended to speed much later in the survey, with an average first time of speeding that varies with survey length.  Interestingly, the first incidence of speeding flattens out at the 8 minute mark – this appears to be the level where impatience starts to set in.  This graph clearly shows the two types of respondents – the “bad” ones that behave badly regardless of the survey, and the ones that are probably driven to do so by the design of the survey.

We will likely encounter both types of respondents in most surveys.  It is apparent, therefore, that we need to attack the problem from both sides – design better surveys through tools such as TrueSample’s SurveyScore to keep respondents engaged, and exclude bad respondents through tools such as TrueSample’s Engagement metrics to preserve data quality. 

Hate the player or hate the game?  Why not just find a way to increase the love?

Check out the video summary of Suresh’s presentation, “How to Identify and Exclude 'Bad' Respondents from Research Studies”, at CASRO on March 3, 2011:

There are two emerging areas and one persistent issue in online market research that keep my brain going overtime these days as I work with the TrueSample team on the product vision for TrueSample. They are:

1.    Real-time sampling and social media sampling
2.    Mobile surveys
3.    Representivity in online sampling

We know that we can eliminate fake, duplicate and unengaged survey respondents from online sample panels or even mobile sample panels, but to take TrueSample to the next level and address the emerging market research data quality concerns that arise with new sampling methods and survey modalities or to solve the ever-nagging issue of representivity, we have to think bigger and more out of the box. There are more nuanced biases and risks to account for and we are building a solution that will get us there.

Just some random thoughts to share with you:

Real-time sampling and social media sampling are here to stay – like it or not. Despite researchers’ concerns about the quality issues that arise from real-time or social media samples, these sampling methods hold much promise, if used wisely. The key is to understand what the actual market research data quality issues are – rather than just dismissing these methods altogether. And once we understand them, we can build technology that automatically addresses the issues in a consistent and repeatable way.

It may be obvious, but – the value of mobile surveys can only be unlocked when we know how to keep mobile respondents engaged.  I’m just not sure this nut has been cracked yet. Sure, there are companies using mobile surveys in innovative ways – like obtaining quick ratings for new movies or having consumers scan barcodes for appealing products in stores. But if anyone is thinking that we can move our existing concept tests or attitude and usage studies to a mobile device, they’ve got another think coming. I say we need to walk before we run… we have to start measuring and quantifying respondent engagement on mobile devices using tools like TrueSample SurveyScore. We have to understand what works and what doesn’t, and we have to build data models that help us to design effective surveys for a mobile audience.

In some cases, traditional identity validation may decrease representivity in online samples.  This may sound controversial coming from someone whose product provides identity validation as one of its features. But common sense says that if you’re using name and mailing address as a way to validate the identity of a survey respondent (like most panel companies are these days), then you may be rejecting survey respondents who are “real” but do not have a constant or long-term address, like 18-24 year olds. This segment, along with other “high velocity” segments, such as Hispanics, need to be verified using new and different data sources.  Luckily, we’ve already identified a solution that is now a part of TrueSample and we’re conducting analysis of its impact on market research data quality as we speak. So stay tuned for more info soon….
 

 The TrueSample team is continually exploring the nuances of survey data quality that impact the results of online market research.  Our goal is to continue to ask questions and explore the possibilities as we guide the industry in assuring the highest possible data quality in quantitative market research.

Today, we’ll look at category and method exclusions – these are often discussed as good avenues for managing survey response quality.  In the context of these discussions, a “category” refers to a type of product or service (for example, skin care products) and a “method” refers to a type of study being conducted (like a concept test).

Many in the Market Research industry are interested in category and method exclusions because there is a belief that, in some cases, presenting the same respondents with several similar surveys in a short period of time may cause them to respond differently than they otherwise would.  This could be due to boredom, an increased ability to game the survey, or other reasons. But the results are the same—an inability to make valid apples-to-apples comparisons across respondents, and a lack of trust in the resulting data.  As if that weren’t bad enough, sending respondents too many surveys that seem too much alike may eventually stop them from participating in market research altogether.

So how do we avoid such problems?  First, we need to track the appropriate categories and methods for each survey.  Next, we need to be able to tie together that survey’s respondents and the date they entered the survey with the categories and methods we’re tracking.  That’s all the information that we need to make the exclusions – we would be able to answer a request like, “Give me a sample of women in their 20s, but exclude anyone who took a survey about makeup in the last 30 days.”

Unfortunately, the solutions that exist today are often quite fragmented and time consuming, requiring researchers to use multiple tools and multiple steps to make category and method exclusions.  In some cases, researchers are able to make these exclusions only with previous respondents to their own surveys.   To overcome these hurdles, companies with a commitment to data quality (like TrueSample) are working on developing technology solutions that will make this effort fast, simple and with less fragmentation across the industry.