INSIGHT on INSIGHT: The Error in Our Ways
Let’s just admit it:
Almost all types of market research are flawed in some way, which leads to insights that can also be imperfect.
However, this imperfection does not make the insights useless nor does it compromise the many practical benefits of having some insight versus no insight.
As long as proper precautions have been take to avoid completely negligent work or completely unsupported conclusions, imperfect insight is not unlike scientists or engineers that rely on imperfect or incomplete theories to guide their understanding of the physical world. They can simultaneously know a theory has limitations yet still confidently rely on it to accurately and consistently predict observable behaviors in the real world.
Insights exist to create predictive models for how certain systems or markets or categories work. They rarely predict how individual people behave on individual occasions. However, they can predict how a larger number of people behavior over a larger number of occasions.
The appropriate analogy would be the fact that statistics can’t predict if one coin toss will end up heads or tails, but it can predict that a large number of coin tosses will have results that concentrate around a 50% heads/50% tails split (assuming an unbiased coin).
Knowing this it important to having the right expectations of insight and utilizing the results from insight appropriately.
Acknowledging the existence of error is also an important step toward controlling and reducing the various sources of that error.
The more one can control the biggest sources of error, the more accurate the insight becomes and the more comfortable people can be making decisions based on that insight.
There are many possible sources for error, but they fit into four basic buckets. To limit the length of this article, I’ve only included a description of the four groups, but click through each to get a more detailed explanation and list of examples.
FOUR SOURCES OF ERROR IN INSIGHTS
Design Error: The many design details that determine how a project is executed can subtly or drastically determine how and what information is collected. This can lead to collecting wrong information or incomplete information.
Respondent Error: Respondents are not machines and therefore are not perfect. They can intentionally and unintentionally share information that varies from the truth. Hillary Clinton or Brian Williams might describe this as misremembering or the fog of time. These errors can be honest mistakes or be motivated by various perceived or actual incentives.
Analysis Error: Anyone that has spent much time with data knows how it can be filtered and manipulated to tell whatever story is needed. However, inexperience, laziness and sloppiness can also unknowingly introduce a number of errors that can lead to results that range from being off by more than rounding error to results that are no more accurate than randomly-generated numbers to inverting results to be the exact opposite of truth.
Interpretation Error: Beyond processing and analyzing data, there is a great deal of interpretation before the information becomes insight. This interpretation relies on having the right experience, perspective, context and recall to recognize and understand the meaning hidden within the data. Lacking the right combination of these skills, insignificant insight may be presented as core conclusions while huge golden nuggets never get mentioned because they aren’t recognized for the value they represent.
All of the errors in the four categories above culminate in making one of two over-arching errors: Concluding an insight exists where it doesn’t or overlooking an insight by not recognizes its existence. You can read more about these here: Type one and type two errors.
AVOID THE WAY OF ERROR
It would be arrogant and ignorant to pretend that most insights are flawless. However, the approach to getting to the insight can be approximately perfect. Understanding how to control for error and recognizing the margin of error that exists around data helps guide reasonable decisions. Acknowledging these limitations when sharing results with external partners or any other audience builds credibility and demonstrates a grounded and realistic perspective.