I often hear semi-informed people dismissing scientific studies that they don't like on the basis that the study used a "small" sample size. Rarely do they define a cutoff for "small", or articulate a rationale for dismissing studies with "small" samples.
On the rare occasions where I have heard such people try to articulate a reasoning behind their concern over small-n studies, it has roughly been that they had a misconception that small n's are more likely to produce "flukes", i.e., Type I errors.
In reality the type I error rate is set by the researcher as alpha, not determined by n, so I don't really give this concern any credibility.
There IS a legitimate concern I'm aware of with SOME statistical tests which have a formal assumption about the normality of the parent population. Often those tests are said to be robust against violations of that assumption but ONLY for moderately large n (usually some rule of thumb like 30 or more).
But is there anything I'm missing here? Any other angle from which to criticize small-n studies?
Of course there are all kinds of criticisms you can make about NHST and relying on p-values, but if we're taking NHST and the "p < alpha" criterion as givens for the moment, what criticisms are there of small-n studies, ONCE they have already rejected H0 (thus, criticizing them as "under-powered" is kind of irrelevant) and been published?
On the rare occasions where I have heard such people try to articulate a reasoning behind their concern over small-n studies, it has roughly been that they had a misconception that small n's are more likely to produce "flukes", i.e., Type I errors.
In reality the type I error rate is set by the researcher as alpha, not determined by n, so I don't really give this concern any credibility.
There IS a legitimate concern I'm aware of with SOME statistical tests which have a formal assumption about the normality of the parent population. Often those tests are said to be robust against violations of that assumption but ONLY for moderately large n (usually some rule of thumb like 30 or more).
But is there anything I'm missing here? Any other angle from which to criticize small-n studies?
Of course there are all kinds of criticisms you can make about NHST and relying on p-values, but if we're taking NHST and the "p < alpha" criterion as givens for the moment, what criticisms are there of small-n studies, ONCE they have already rejected H0 (thus, criticizing them as "under-powered" is kind of irrelevant) and been published?
And there are significant issues with samples in terms of generalization at least when people are concerned. These include psychometric error, people changing their mind, and in the era of cell phones actually getting a non-biased sample. Change over time is a major problem as is uncertainty on the part of those who respond and non-response rates. Missing data has become a major concern of data analyst as has non-response rates.