INSIGHT on INSIGHT: Picking the Right Sample Size

By now, you should have a clear understanding of the core objectives of your research, you should have thought through how the results will be presented, and you should know what qualifying respondents look like.

At this point, it is important to pick the right sample size for your project.  This applies as much to quantitative research as it does to qualitative research.

In a world of unlimited resources (that is, unlimited money and time), it would be safest to study the largest possible sample or even the entire population of the people that would qualify as respondents.  In reality, resources are never unlimited.  And in practice, smaller samples can provide a very safe proxy for the insights of large populations.

These six principles should guide the total number of people you decide to include in your research.

A sample size of 1,000 tends to be a psychological threshold of confidence:  Similar to the fact that more people will buy a product at $9.99 than $10.00, many people evaluating research findings like to see the four digits and know the results represent at least 1,000.  For irrational reasons, some will be more suspicious of or more willing to question results with 900-something respondents.

Consider the smallest slice of data you want to read and build the sample size from the bottom up:  If you plan to look at a sub-segment that represents 10% of you probable respondents, that group will only have about 100 people in it (assuming the total sample was 1,000).  This significantly increases the margin of error (i.e. the known accuracy) of that sub-segment which increases the odds that any differences simply represent random noise in the data.  We generally like to see a minimum base size of 250 for any important sub-segment.  This provides a margin or error less than +/-4.0 at 80% confidence.  For less important sub-groups, we still push for a base size of at least 100.

Consider the unique groups you want to compare:  Twelve one-on-one interviews can provide a lot of perspective, but the confidence in that perspective diminishing quickly if one of the research objectives was to compare attitudes across six sub-segments (thus only giving you two interviews for each group).  In quantitative research, there can be lots of comparisons among groups that are not mutually exclusive (i.e. groups with a lot of overlap such as Walmart shoppers v. Target shoppers).  In these situations, it can be extremely helpful to be able to isolate unique segments (i.e. Walmart-not-Target shoppers v. Target-not-Walmart shoppers) versus trying to tease difference out of groups that share 70% of the same respondents.

Consider the margin of error you are willing to tolerate:  Decisions based on data rely, in part, on the confidence in the accuracy of the data.  Accuracy increases with sample size.  To use an extreme example, a sample of one respondent has a +/- 64.08% margin of error while a sample of 1,000 respondents has a +/-1.98% (both at 80% confidence).  With a sample of one, a numeric score of fifty could actually be below zero or above 100% for the entire population while that same score among a sample of 1,000 is confidently between 48 and 52.  Larger sample sizes reduce the risk of delivering insights and basing decisions on nothing more than noise.

Consider the potential need for secondary analysis after the initial insight is revealed:  New shopper insight inevitably generates new questions or hypotheses.  Well-designed research should be able to address a portion of these through additional analysis of the existing data, saving time and money.  Being too efficient on sample size can force some questions to go unanswered, requiring incremental investment in additional research to answer.

Consider how many legs are included in your research:  Legs of research refer to different branches of questions that different respondents are exposed to.  It often involves having different groups be exposed to different stimulus, such as respondents seeing different package designs or different price points.  Generally speaking, the greater the number of legs a survey has, the fewer people each will be exposed to.  Sometimes large total sample sizes are required to create readable data through different components of the survey.

To a degree, larger sample sizes are always better, but there is certainly a point of diminishing returns.  By considering what is necessary to deliver the research objectives for a project, you should be able to arrive at the optimal number of respondents to stretch your budget as far as possible.