INSIGHT on INSIGHT: Separating the Signal From the Noise

In a world with huge amounts of data that are growing bigger everyday…

In a world continually creating new ways to capture or create more data…

In a world with new approaches to analyze and interpret data…

Effective insights depend on the ability to sift through a lot of data and recognize the signal among the noise. 

In one sense, this reflects the working definition we use for shopper insight: 

Insight is the ability to recognize meaningful trends, patterns or relationships within a larger set of data or information.

Some insight can be a blinding case of the obvious, immediately recognized both for being true and being able to guide strategies and actions. 

Other times, the ability to recognize insight is not so easy.  In these situations, it is important to avoid the two types of errors that can compromise the utility of a project or cause a client to send resources down the wrong path:  Type One Error and Type Two Error

TYPE ONE ERROR:  THE FALSE POSITIVE

There are times when an insight or conclusion is produced that simply is not true.  This reflects the fact that insight projects are done to produce insights.  There is a natural bias to want or need to find insights within data.  After all, an insight project with little insight would be considered a failure, right?

In quantitative insights, this would represent times when one data point is treated as significantly different than another while there is actually no statistical difference between the two.  This would be like declaring a winner when 58% of respondents gave concept M a Top 2 Box (T2B) purchase intent (PI) and 57% gave concept K a T2B PI.

In qualitative insights, this might be reaching the conclusion that instructions for a product are confusing.  While this might have been mentioned in interviews, it might not be meaningful because it does not project to the broader population or it might mask the truth that very, very few people actually read the instructions.

In the first quantitative example above, the consequence of the error might be less significant as both concepts had similar scores so they could be expected to have similar in-market performance. 

However, further investigation might have revealed that concept M had a 33% Bottom 2 Box (B2B) purchase intent (people indicating they probably or definitely would not buy it) compared to a 7% B2B PI for concept K.  This would strongly suggest moving forward with concept K.

Additional analysis could also reveal that concept M had significantly higher purchase intent among high volume category buyers, possibly indicating it would produce more sales over time even if it had the same number of potential buyers compared to concept K.

In both of these situations, a conclusion (concept M is better) is not accurate.  Yet that conclusion will likely trigger a cascade of decisions that could lead to results that fall far short of expectations.

Other examples of false positives include thinking one is pregnant when they are not or thinking one has identified the cause that triggers an effect when other factors are actually driving the relationship between the two variables.

TYPE TWO ERRORS:  THE FALSE NEGATIVE

Other times, a project produces no insight, or no conclusion is arrived at when one actually did exist, but was overlooked.

In quantitative insights, this would represent a time when one observes a difference between two data point, but that difference is disregarded for some reason.  While that reason might be appropriate (such as due to small base size), it doesn’t change the truth that the difference exists.  This could also represent a situation when one unintentionally overlooks a meaningful difference between data points because the wrong analysis or wrong groups are used in the research.

In qualitative insights, this error might be the result of ignoring comments from one individual that sound too outlandish and are not reflective of other participants.  In reality, that person might be voicing a position others are too self-conscious to admit.  Or that person has greater self-awareness or perspective to have arrived at the realization sooner than others.

In the examples used for type one errors, not looking at B2B PI or not looking closer at groups of greater value (like high volume users) could be considered other forms of type two errors.

In some ways, type two errors are less dangerous because they typically result in the lack of action versus taking the wrong action.

In other ways, type two errors come with huge opportunity costs due to the potential value of the missed insight and non-existent response to it.

Think about how wealthy someone could become if they could recognize patterns in the stock market and invest accordingly.  Or count cards while playing blackjack so they knew when the remaining deck was loaded with face cards.

Few people spend time considering these situations because they have accepted operating with the constant lack of this knowledge.  They have accepted a reality full of type one and type two errors.

AVOID THE EXTREMES

Both the act of unintentionally fabricating insights and ignorantly overlooking others diminish the value a project is able to deliver. 

As you tackle future insight work, pause long enough to challenge the key insights you’re gravitating to.  Make sure sufficient evidence can confirm their existence.

And dig a little deeper to make sure you haven’t overlooked even more powerful insights just because they are less obvious.