Two way anova and Bonferroni

#1
After emphasizing again and again to students the problems with multiple p values and possible ways of dealing with them, one asked me why we didn't use Bonferroni on the two (or three) p values in a standard two way anova table. I don't know. Or should we, but nobody does? Or do some people? kat
 

Miner

TS Contributor
#2
Multiple comparisons of means only comes into play when you have more than two levels per treatment This is a common occurrence for a 1-way ANOVA, but is less common for k-way ANOVAs. k-way ANOVAs are typically used to analyze designed experiments which are most commonly 2^k designs with 2 levels per treatment. General full factorial designs are often prohibitively expensive to run versus 2^k designs, so treatment levels are usually limited to two.
 
#3
This isn't a case of multiple comparisons - it's simply multiple p values. There are three p values in your anova table, one for each factor and one for the interaction, and so there are three chances of making a false positive claim. If you do lots of two way anovas with random data, at least one of the p values in your anova table will be < 0.05 about 15% of the time. Our 95% protection against false claims is severely eroded. So, do we need Bonferroni?
 

Miner

TS Contributor
#4
Interesting discussion. It was very difficult to find anything on this topic as family-wise error rates are always discussed in the context of a 1-way ANOVA. However, I did find one article that supports your concern when an exploratory ANOVA is used without an a priori theory. The article presents several approaches for dealing with this concern. However, I am not aware of any software that offers a way to deal with this.
 
#6
Thanks Miner, this is a very interesting article!

I assume it is relevant for any multiple regression.
I guess this is the reason why an automatic process like the stepwise method for multiple regression is good only as a screening method and not as a way to build a model?
 
#7
I would presume so. The issue raised would seem to be independent of the actual analysis method and focused on whether that method was used for data snooping vs. confirmation of an a priori theory.