Power Analysis
A power analysis means slightly different things in different settings, but is essentially a way to quantify the power of your data to indicate the alternate hypothesis when the alternate hypothesis is in fact true, as well as reject the null hypothesis when the null hypothesis is in fact false. These two things are actually not the same and figure in separately as I’ll expand on below!
The most common need for a power analysis is the comparison of two continuous variables for a statistically significant mean difference between the groups. The power analysis will incorporate, in this case, four variables or values:
- The alpha (alpha = Type I error risk, the risk we accept the alternate hypothesis when the null hypothesis is true): a value between 0 and 1, usually set at 0.05
- The power, or beta (1/beta = Type II error risk, the risk that we accept the null hypothesis when the null hypothesis is false): a value between o and 1 and usually set at 0.8, 0.9, or 0.95 (meaning the Type II risk is 0.2, o.1, or 0.05 respectively).
- The number of observations in each sample (or “n”; may be the same for each sample or different)
- Effect size: a value indicating the size of the difference we are seeking to detect between the groups, relative to how spread out each group is.
Given any of the three variables/values above, we can determine what the fourth value is based on the test we plan to use. Most commonly, we have an a priori alpha, beta, and effect size (that is we know how much risk we will accept in our statistical tests [alpha =0.05 and beta =0.8 usually], we know how big a difference we care about) and what we want to know is how many samples (or participants) we need in order to meet that criteria.
A power analysis can be done before a study (pre-hoc) to plan the number of participants, or following data collection (post-hoc) when the number of participants is already known to determine how much of an effect size the data could detect.