While respondent fraud is a serious issue, it's crucial to emphasize that much of the responsibility for ensuring survey data quality actually lies in the hands of researchers and how they design surveys.
With the rapid evolution of anti-fraud technology, we’re catching a significant amount of fraud pre-survey. Because of this, the scale of data quality issues in-field resulting from poorly designed surveys can, on some occasions, dwarf the data quality issues caused by fraud by an order of magnitude.
There are three important areas researchers should focus on when designing surveys to ensure high-quality data:
Tackling Overclaim in Screening Questions
We have conducted extensive research to measure levels of overclaim in screening questions. It's not uncommon to find upwards of 20% of respondents not providing truthful answers at the screening stage of a survey.
It's easy to complain about respondents overclaiming to avoid being screened out. However, a respondent on average is screened out of 6 out of 10 surveys they attempt. Consider this and you may better understand their motivations to overclaim in an attempt to finish a survey.
Addressing overclaim is a solvable issue. It can be mitigated with effectively designed screening questions and reliable detection protocols. Various techniques can be employed, such as:
- Using effective question funnels to mask the survey topic
- Selecting high-low incident options to determine respondent honesty
- Employing well-thought-out preliminary questions to preempt overclaim
For instance, instead of asking if someone owns a car (which might lead to overclaim), ask what they would want to own, and include a car in the response options. This approach effectively identifies those who desire but do not own a car. Find more insights on designing better screening questions here.
Debiasing and Humanizing Your Questions
Another common source of erroneous data is unconsidered biases created in question framing. For example, asking “What is your working status?” can lead to a 20% overclaim on employment, as the question implies that having a job is important.
As researchers, it's important to be cognizant of human nature. We all want to appear in a positive light, yet we’re full of human frailties that may be embarrassing for some.
For example, asking “Did you do any exercise yesterday?” may embarrass people into falsely claiming they did. However, by adding reasons for not exercising, such as illness or injury, respondents may feel more comfortable being truthful, significantly reducing the number of false claims by 50%.
Tackling Respondent Fatigue
More often than we’d like, we see surveys that are lengthy, overly wordy, dry and frustrating for respondents to take. When little thought is put into the design, usability and narrative, we find these poorly designed surveys lead to high dropout rates.
Respondents are often frustrated with repetitive question types, like the agree-disagree Likert scale. Our research-on-research shows that 25% of respondents disengage (nearly automatically) when answering basic Likert scale questions, and there’s a notable 40% drop in click-count when questions are repetitive.
With brand and advertising recall questions, we find that respondents disengage quickly. We witness up to a 30% passive recall of fictitious brands used to test attention levels.
The key to overcoming these challenges lies in innovative survey design. Engaging surveys and varied question formats (that don’t rely on resorting to the platitudes of a liker questions) can significantly enhance the participant experience. For inspiration, try using our empathetic approaches to survey design.
Subscribe for monthly research tips
Our experts are providing monthly research tips on survey design best practices, analyzing survey data and more. Subscribe using the form below.
