INSIGHT on INSIGHT: Controlling Design Error
Errors in how insight projects are designed can be devastating to a project as they can compromise the overall integrity of the results. This is true because there are rarely good options (read: fast and cheap solutions) to compensate for or correct errors in design. Design errors can also carry throughout the entire project, compounding the impact they have.
Whether this is your first insight project or you’re more experienced, consider using the following checklist to make sure no catastrophic design errors work their way into your project:
SOURCES OF DESIGN ERRORS
Poor screening criteria: Insights are only relevant to the population they represent. This is why political polls typically report results among ‘likely voters,’ not just registered voters or the entire population. However, smart pollsters will still include broader groups because they know the potential exists to alter results by getting more unlikely voters to show up on Election Day.
Poorly defined time period: Avoid the ambitious expression “the past year” (respondents could interpret this as the time since the last new year or the past 12 months). Use the language past 12 months (P12M) to make sure all respondents are referencing the same time period. To the same end, avoid user-defined terms like recent or soon.
Wrong time period: Many surveys qualify respondents based on recent or planned behaviors (such has having bought a category or planning to buy it soon). While past 12 month behavior (P12M) is most common in my work, categories with longer purchase cycles may need to frame behaviors in the context of the past several years (think cars or TVs) while high frequency categories might be more appropriate to only recruit past 3 month buyers (think milk and eggs). However, each of these categories may want to capture future purchase intent in a different time frame, such as only interviewing people planning to buy a TV in the next six months of asking survey respondents about what they will do “the next time you plan to buy milk.”
Poor question wording: Clients might think there is one obvious way to word a question. However, many questions have numerous permutations that can slightly to significantly alter the context of the question and thus the meaning of the answer. A simple example is the difference between asking “How much do you like the color green?” versus “How much do you dislike the color green?” Another is the difference between asking “Which product do you like most?” versus “Which product would you be most likely to buy?”
Leading questions: Questions should never cause respondents to believe one answer is better or preferred over others. They should also not be phrased in a manner that prematurely directs or constrains the response.
Poorly ordering questions: Every question asked biases the perception of future questions. A survey can’t jump into details related to brand preference or loyalty only to step back and ask how important brand is compared to other attributes. Question order often needs to go broad to narrow, even if the final questions are the most important ones.
Incomplete answer choices: Writing any survey requires sufficient category knowledge to provide complete pick lists. Results can vary from irrelevant to misleading when important options are missing (like overlooking popular colors or scents or flavors, or missing common attitudes or habits).
Forcing preference: There are many situations where forcing preference helps clarify results (like making respondents pick the one most appealing concept). However, there are other situations where forcing preference creates false insight (like asking a mom which child is her favorite).
Using the wrong type or form of a question: While ‘select one’ or ‘select all that apply’ questions with a finite pick list are the most common questions for internet surveys, there are many other variations that may be more appropriate. The question form should be selected based on minimizing respondent burden, maximizing the flexibility of analysis options on the back end, and producing results that are clean and easy to interpret.
Lack of randomization: The order that both questions and answer options are presented will influence their answers. This influence can range from assuming the first options are more important or preferred to respondents not reading through an entire list after selecting one or two answers. It is particularly important to randomize when testing different stimulus, when presenting pick lists that have no logical order and when represent long lists that could cause fatigue.
Unnecessary randomization: While randomization is done to avoid bias, unnecessary randomization can cause confusion or frustration. This obviously applies to questions with sequential answer options (like age or income ranges), but also applies to questions with recognized or alphabetic options (like stores shopped or brands bought). In these situations, respondents prefer to jump to certain answer options (versus scanning the entire list) or will have limited patience to search for their answer option hidden in a scrambled list.
Lack of clear meaning in word choice: The multiple meanings of certain words can confuse respondents or blur results as different respondent imply different meaning as they answer questions. Examples include use of the word ‘good sales’ (does it mean products that are often offered at a good discount or products that appear to sell a lot?) or ‘cheap’ (does it mean inexpensive or low quality?). Selecting words like “good promotions or discounts” or “inexpensive” provides important clarify.
Poor answer options causing blurred data: In the simplest form, this means making sure answers are mutually exclusive and use ranges or points appropriately. Consider how someone wanting to answer 50% would handle a question with answer options “25% to 50%” and “50% to 75%”. In other situations, common or dominant answer options shouldn’t fall at the edge of a range. That is, if most respondents are expected to have an answer around 50%, don’t offer ranges of “25% to 49%” and “50% to 74%”. Rather, offer ranges that include a 40% to 60% option to keep this central group clustered together.
Poor display logic causing false choice: Many surveys contain sequential questioning where the answer to one question determines the proper options or relevance of another question. Not considering this can expose respondents to inappropriate or irrelevant options that complicate their experience. The consequences can range from confusing to dirty data (such as a respondent not indicating awareness of a brand, but claiming to have bought it in the past 12 months).
Overlooking the “none of the above” of “other” options: While good surveys are designed to recognize the vast majority of relevant or appropriate, they are rarely perfect. Many questions should include a ‘none of the above’ for respondents to opt out of all answer options. Many can also benefit from an “other (please describe)” option to capture text entry and possibly discover unexpected perspective, preference or habits that point to new opportunities.
Poorly timed execution: When projects are executed influences who are available as respondents. Weekday respondents to online surveys are more likely to be non-working, older or lower income. Weekday interviews are less likely to attract full-time white-collar workers. Evening and weekend execution timing can be important to reach certain attractive profiles. When tight execution timeframes are necessary, additional screening and controls should be put in place to ensure a representative sample.
There are lots of ways to make design errors. Unfortunately, there aren’t as many ways to fix them without paying a high price in time and money.
Do any of the above errors sound like possible issues in past projects you’ve done? If so, realize how much new insight can be captured once you’re able to field new work designed to produce better data.