Developing Human Capital: Using Analytics to Plan and Optimize Your Learning and Development Investments

Appendix GHow Many Participants Do You Need in a Study?^a

The number of participants required to achieve a statistically significant answer is a hard value to estimate definitively, but there are ways to calculate decent approximations, if you know (or can assume) some things about the data. We like statistical significance at a fairly high level—preferably 99 percent—so that determines our preference. Statistical significance is the product of three factors:

1. The effect size. The effect size is a statistical way of saying how big of a difference is seen between the test and control groups. Not surprisingly, a 50 percent improvement is a lot easier to spot than a 1 percent difference.
2. Variation. There are several statistical terms for this, such as variance or standard deviation. Those are mathematical ways of measuring just how consistent results are. Human beings typically have a lot of variation in their performance, but it is a question of just how much. For example, the amount of variance in public schools is usually higher than that in private schools.
3. Population size. The more people in a study, the more certain you can be that the effects are real. If only two people are in the test group, an improvement might well be a lucky shot. If 100 are in the group, much more certainty can happen.

There are equations and techniques in statistics where, if you know the level of statistical certainty desired and can estimate or guess about two of the above three values, you can ...