INSIGHT on INSIGHT: Advanced Analysis Planning

It may seem premature to discuss or plan the analysis of research results at the beginning of a project.  However, the sequential nature of the insight process makes the necessary:

The right questions need to be asked in the right format…

…to produce data that can be analyzed in a certain way…

…to deliver the results that lead to the desired outcomes…

…that justified doing the project in the first place.

This is why everything begins with setting the right objectives.  Those objectives will determine what is the most desired or best analysis approach, which should then guide how the project gets designed.  If done right, the project design should not dictate (read: “limit”) what analysis options exist on the backend.

DON’T BE CAUGHT LACKING

In practice, lots and lots (probably the majority) of research projects do not spend enough time thinking through the analysis plan.  They plan to get the data and then figure out what can be done with it.  This happens for a variety of reasons:

Lack of experience:  Creating insights is a craft learned over time.  These mistakes are not unlike someone designing and building a house for the first time without considering what direction doors should swing and where light switches should be located until doing the final walk-through.  There is an assumption that things will work out fine…until they don’t.

Lack of time:  The do more with less mentality of many companies doesn’t give managers time to visualize and walk-through complex plans to consider different scenarios.  It is easy to justify cutting out the “luxury” of spending time in our mind palace when 80 hours of work exist for every 40-hour week.

Lack of knowing the unknown:  People tend to get in trouble because of scenarios they didn’t predict, not the ones they did.  Research results don’t always come back as predicted.  They are more likely to produce many shades of gray versus a clean black-and-white story.  Projects get in trouble when the one analysis approach they were designed for produces ambiguous results.

Lack of motivation:  Analysis is often an under-appreciated and lonely endeavor.  Sometimes, days of analysis can result in huge files of data, but absolutely no useful insight.  Other times, there is no accountability for the amount and type of analysis performed.  Insight presentations often consist of the first reasonable insights that were discovered, not the best or most valuable insights.

Lack of appreciation:  The audience for insights rarely cares about the process followed to uncover the insights.  In reality, the less the analysis needs to be explained and the more succinct the insights can be presented, the better.  In a sense, great analysts are like great musicians, athletes or artists…they make their craft look deceptively easy, and convince the people watching that they could have produced the same results.

WHY ADVANCED ANALYSIS PLANNING MATTERS

Advanced analysis planning is important regardless of the scope of a project.  It is just as easy to get caught with a small and narrow set of data that fails to produce the designed insight as it is to get lost and overwhelmed by a large and broad set of data capable of producing far more insight, but also far more meaningless data.

Advanced analysis planning can often reveal design flaws that otherwise might not be revealed until the raw data is in-hand and it is too late.  This includes:

Realizing what unique segments need to be readable:  It is never good to realize you can’t identify or define particular segments the way you want to.

Knowing what head-to-head comparisons are desired:  With complex surveys, it is easy to have different groups skip through different sets of questions that might make comparisons impossible.

Being able to create the desired visualization:  “Select one” questions can mask a very close second or third preference while it is difficult to create accurate pie charts from “select all that apply” multiple choice questions.

Going deep enough with questions to get to the root cause:  There is nothing more frustrating than arriving at an incomplete answer because one more probing or explanatory question wasn’t asked.

This begin with the end in mind approach is a way to reverse engineer the best project design.

How much thought have you put into advanced analysis planning?

Have you thought through the analysis a fraction as much as you have thought through the survey outline or discussion guide?

Or are you just hoping you’re asking the right questions and the right answers will naturally fall out of the resulting data?