Do Arbitrary Significance Levels Distort Published Results?
Alan S. Gerber Yale University, New Haven, Connecticut Neil Malhotra Stanford University, Stanford, California, neilm{at}stanford.edu’ + u + ‘@’ + d + ”//–> Despite great attention to the quality of research methods in individual studies, if publication decisions of journals are a function of the statistical significance of research findings, the published literature as a whole may not produce accurate measures of true effects. This article examines the two most prominent sociology journals (the American Sociological Review and the American Journal of Sociology) and another important though less influential journal (The Sociological Quarterly) for evidence of publication bias. The effect of the .05 significance level on the pattern of published findings is examined using a “caliper” test, and the hypothesis of no publication bias can be rejected at approximately the 1 in 10 million level. Findings suggest that some of the results reported in leading sociology journals may be mislea
