I often hear semi-informed people dismissing scientific studies that they don't like on the basis that the study used a "small" sample size. Rarely do they define a cutoff for "small", or articulate a rationale for dismissing studies with "small" samples.
On the rare occasions where I have heard such people try to articulate a reasoning behind their concern over small-n studies, it has roughly been that they had a misconception that small n's are more likely to produce "flukes", i.e., Type I errors.
In reality the type I error rate is set by the researcher as alpha, not determined by n, so I don't really give this concern any credibility.
There IS a legitimate concern I'm aware of with SOME statistical tests which have a formal assumption about the normality of the parent population. Often those tests are said to be robust against violations of that assumption but ONLY for moderately large n (usually some rule of thumb like 30 or more).
But is there anything I'm missing here? Any other angle from which to criticize small-n studies?
Of course there are all kinds of criticisms you can make about NHST and relying on p-values, but if we're taking NHST and the "p < alpha" criterion as givens for the moment, what criticisms are there of small-n studies, ONCE they have already rejected H0 (thus, criticizing them as "under-powered" is kind of irrelevant) and been published?
On the rare occasions where I have heard such people try to articulate a reasoning behind their concern over small-n studies, it has roughly been that they had a misconception that small n's are more likely to produce "flukes", i.e., Type I errors.
In reality the type I error rate is set by the researcher as alpha, not determined by n, so I don't really give this concern any credibility.
There IS a legitimate concern I'm aware of with SOME statistical tests which have a formal assumption about the normality of the parent population. Often those tests are said to be robust against violations of that assumption but ONLY for moderately large n (usually some rule of thumb like 30 or more).
But is there anything I'm missing here? Any other angle from which to criticize small-n studies?
Of course there are all kinds of criticisms you can make about NHST and relying on p-values, but if we're taking NHST and the "p < alpha" criterion as givens for the moment, what criticisms are there of small-n studies, ONCE they have already rejected H0 (thus, criticizing them as "under-powered" is kind of irrelevant) and been published?