INSIGHT on INSIGHT: Controlling Respondent Error
The vast majority of people that choose to be survey respondents or interview participants want to provide useful and accurate information. There are very, very few malicious individuals intentionally trying to provide useless or wrong answers or participating solely for the compensation they receive.
However, there are a variety of ways that respondents can be a source of error. Fortunately, it is generally easy to recognize the root cause of each error and to make simple changes to prevent or control them.
In addition to design errors, consider if your project contains any of the following causes of respondent error:
SOURCES OF RESPONDENT ERRORS
Recency bias: Short-term memory tends to heavily influence our momentary views so research will always be tied to the particular time it is executed. This is why little research about general shopping habits is done during the frenzied spending of the Christmas season. It is also why learning about habits around certain annual events (like Back-To-School) is best done 12 months in advance, while shoppers are actively engaged in or have recently finished experiencing the event.
Claimed behavior differing from recorded evidence: Unless one has photographic memory, they are unlikely to recall exact habits over the past 12 months (the typical historic time frame most research tries to capture). This is what makes POS data from retailers so useful as an accurate and complete record of the truth. However, the limitations of perfect recall does not equate to claimed behaviors being completely useless. The aggregation of recall error across a large sample of respondents and the relative and comparative accuracy of claimed data provides an immense amount of insight that could never be revealed through POS data alone.
Misremembering: It is common knowledge that people transpose memories or insert false information to address gaps in recall. This happens when people associate advertising with the wrong brand or they switch the order that events happened. The desire to satisfy a moderator can also prompt mis-memories in order to provide a satisfactory in-person interview. Research can control this risk by not asking about events in the too-distant past, not asking unreasonably detailed questions or by giving respondents an opt-out answer.
Rationalization: Humans frequently feel compelled to publically credit their intellect even when emotions are driving their decisions. In truth, there are many, many things we do that are impulsive, sub-conscious or emotionally-driven. Rationale or logic can only be used in hindsight to try and make sense of the decision. Huge insight can be discovered when this is recognized and the interaction of logic and emotion is studied.
Minimized burden: We all have a certain amount of laziness that comes out when we can get away with it while not feeling guilty. That may surface in well-intended survey respondents or interview participants when they fulfill their obligation yet provide incomplete answers. In surveys, this happens when respondents stop reading through a list of answers once they have found a few to select (the impact of this can be minimized through randomization). In interviews, this happens when they fail to articulate their answer in full (this is remedied through effective probing or follow-up questions).
Maximized benefit: The term Professional Research Participant has been around for a long time. The growth of online surveys has theoretically made it easier for individuals to make a living wage through constantly taking surveys (through multiple email addresses, aliases, or membership in numerous panels). The concern is that these respondents can create bad data as they bang through surveys as fast as possible, selecting random answers just to get to the end. Through a variety of controls utilized by panels and the reality that taking surveys for a living is a relatively painful job, this should represent a fraction of a percent of respondents.
Personal bias or interest: It is probably safe to assume that a higher share of survey respondents or interview participants have strong opinions and enjoy sharing them with others. It is really tough to get someone to talk about a subject they have little interest in. However, these opinionated people are often also more likely to be active in their social networks, sharing a lot of word-of-mouth information or be early adopters for the categories they care about.
Fatigue: Fatigue affects every activity people do, no matter how enjoyable. The most dedicated survey taker or interviewee still has a limit with their time and attention. While absolute thresholds exist, setting accurate expectations about how much time participation will require, keeping the content as engaging and interesting as possible, and making the participant feel appreciated and rewarded will minimize the risk of them checking out or shutting down before all the questions have been answered.
Lifestyle choice: Due to temporary situations, certain groups can be difficult to recruit, such as mothers with new babies or new small business owners. They have other priorities, limited time, and little incentive to let research participation distract them from greater demands in their life. On the other extreme, individuals that are less active or less involved in other activities may be more likely to participate.
There are a variety of ways to reduce, control or manage respondent error. Recognizing the possible sources is the first step.
Do you see any of these factoring in past insight work you’ve done or saw from others? If so, be prepared for new of different learning to come from future projects that fix the various sources of respondent error.