Substantive significance is concerned with meaning, as in, what do the findings say about population effects themselves?
Researchers typically estimate population effects by examining representative samples. Although researchers may invest considerable effort in minimizing measurement and sampling error and thereby producing more accurate effect size estimates, ultimately the goal is a better understanding of real world effects. This distinction between real world effects and researchers’ sample-based estimates of those effects is critical to understanding the difference between statistical and substantive significance. The statistical significance of any test result is determined by gauging the probability of getting a result at least this large if there was no underlying effect. The outcome of any test is a conditional probability or p value. If the p value falls below a conventionally accepted threshold (say .05), we might judge the result to be statistically significant. The substantive significance of a result, in contrast, has nothing to do with the p value and everything to do with the estimated e
