MISTAKE #74: You thought you could manipulate data and get away with it

Good salesmanship always involves some amount of creative license.  I get it.  Your job is to close a sale, not provide a balanced and unbiased perspective. 

But if you mislead the buyer, expect shoppers to quickly out you as they refuse to live up to the false expectations you set.

The era of big data has given more companies more information to play with.  And this data has created more ways to intentionally or accidentally do misleading or completely wrong analysis that leads to misrepresentation of reality.

The number of ways data can be misrepresented grows with the types of data available.  And buyers are getting wiser and wiser to the intricacies of how the same data set can produce contradictory conclusions depending on the motivation of the person doing the analysis. 

Depending on your disposition, the following list could be titled:

“15 Watchouts When Working With Data”

 or

“15 Ways to Get Data to Say Whatever You Want”

 (Only you can decide which list you’re most interested in reading)

You constrain the time period:  Is business looking good over the past week?  Past 4 weeks?  Past 13 weeks?  Using time periods is a great way to reveal trends, but don’t crop them in a manner that creates a misleading perception of the current state.

You nationalize geographic skews:  Most products have variations in regional performance.  These variations exist for reasons that can’t necessarily be replicated nationally.  Don’t project potential national numbers based on regional data.

You bias your sample:  When studying your category shoppers, make sure the sample is representative.  Extrapolating conclusions from a self-selected sample of fans or highly engaged shoppers are only giving you a fraction of the perspective you need.  And it is probably an overly positive perspective that doesn't reveal ways to drive incremental sales.

You only report what you want to hear:  We all know how much data gets left on the cutting room floor.  However, being too quick to ignore, cover up or forget unpleasant facts is a huge missed opportunity.  Not sharing and talking about issues or shortcomings ensures they’ll continue to fester and probably be worse the next time they surface.

You ask biased questions:  So, is this one of the best articles about data you’ve ever read or the best article?  Don’t worry.  I don’t need to hear your answer to feel good about myself.  Consider how the wrong questions can produce the wrong data or misguide how analysis is approached.

You are too quick to use experience to explain the why behind data:  While experiences are a core part of building knowledge and expertise, they should not always be quickly embraced as fact.  Experience could produce good hypotheses, but one should be more cautious to jump from the rigorous analysis of data to quickly believing the first explanation that seems to make sense.

You prematurely truncate data:  I often work with data that has too many variables to display for easy consumption.  However, I’ve learned not to succumb to the temptation of ranking the data and dropping off the bottom results.  Instead, I’ve learned to both report this truncated data for easy consumption, and to also spend time looking at all the data (and often focusing on what might be commonalities across the bottom ranking items).  This allows you to tell the big story without prematurely overlooking opportunities that linger further down the list.

You ignore statistical validity:  All data has a margin of error that surrounds it.  Two numbers that are 3% apart might indicate a meaningful difference if the sample size is over 1,000, but are nothing more than noise if that sample size is 100.  Make sure you’re not grasping at small, potentially non-existent differences to build your case, tell your story, or (worse yet) build your selling strategy around.

You’re only looking for the answers you want to find:  Effective analysis of data almost always begins with a clear understanding of what to look for.  You’re sure to find data points that make you feel good about your business, but they probably aren’t good for your business.  Consider the direction that is given to analysts as they seek to uncover the issues or opportunities that will actually lead to the results you want.

You are poor at presenting data:  Expensive software is available to help companies visualize data in new and different ways that make the trend or pattern or outliers more obvious.  Many books have been written about the art of selling with data.  Using default graph settings in Excel is rarely the best way to incorporate data into your story.  But the most advanced software is useless in the hands of someone that doesn't know what story they are trying to tell.

You focus on the average, not the distribution or segments:  Averages are so misleading.  They create a false sense of consistency or the assumption that your efforts can target a ‘center of mass’ to accomplish your goal.  For example, the “average” shopper often masks unique segments that had different needs, act different, look different and represent different opportunities with different potential values.

Your numbers don’t add up:  Data that is not mutually exclusive cannot be displayed in pie charts.  Be careful not to choose the wrong visualization as you try to bring your data to life.  Or as you just trying to fit it on a single page.

Your data is too aggregated:  Another large part of the challenge with big data is knowing how deep to dig.  Large datasets can be sliced and diced in a nearly infinite number of ways.  These deeper or thinner slices don’t always produce anything useful.  But they can give you the confidence you’ve reached the point of diminishing returns and are ready to draw conclusions and act on the next higher level of analysis.

Small sample sizes aren’t acknowledged:  So you’ve got data that shows your category has really unique dynamics at a particular retailer or within a particular store profile.  That might be a great opportunity, but it could just as easily be a little more than noise if that retailer represents just a percentage or two of the market share.  Small sample sizes (or small data sets of any kind, including a limited number of transactions or short time periods) are worth noticing but need to be monitored more than acted on.

You ignore multicollinearity:  Variables often interact or are interlaced, which can cause a data point to be artificially amplified.  For example, a unique behavior may exist among lower incomes, older ages, and people living in rural areas.  Accurately interpreting this data requires the knowledge that rural areas have a much higher share of lower incomes and older ages.  The three data points are largely representing the same set of people.  The same patterns form around the fact that younger ages and minorities have much higher concentrations in urban markets.

Let’s be clear about the lesson here: 

The data (almost) always wins.  Data exists to help you discover reality, not to help you make your own reality.

 Whether it is done accidentally or intentionally, the misuse of data does not change the truth in the data.  Be careful with the stories you tell or the spin you use.  You’re not only setting yourself up to eventually be found out, you’re probably missing out on bigger opportunities. 

If you realize your company might have developed some of the bad habits mentioned above, we’d be happy to help get you back on the right track.  We can bring a new perspective to either validate that you're looking at your data the right way or produce new insight that reveals what you’ve been missing out on.