Diff p-value when using 1wayANOVA and 2wayANOVA on same set of data

[megsl]

I used the 2way ANOVA to compare 2 treatments (control vs. T1) across 4 groups (A-D). I used the 1way ANOVA to compare 3 treatments (control vs. T1 vs. T2) in group A - only group A had 3 treatments.


But the results for the tests were different: the 2way ANOVA said there was no effect of T1 in group A, but the 1way comparing the 3 treatments said there was an effect between control and T1. Both used Bonferroni multiple comparison's test, and compared the cell mean with each row, same confidence interval. 2way ANOVA p-value was 0.4496, 1way ANOVA p-value was 0.0348.


What's going on, shouldn't the p-value be the same since the data is the same and the test is essentially testing the same thing? Am I missing something that is fundamental?

 

[GretaGarbo]

Maybe the data was not balanced.

What's going on, shouldn't the p-value be the same since the data is the same and the test is essentially testing the same thing? Am I missing something that is fundamental?

No, the data is not the same and the model is not the same.

 

[EdGr]

First of all, was the group by treatment interaction from the 2 way significant? If it was NOT, then you shouldn't even be looking at simple effects (like t1 versus control within each group). It is the presence of the interaction that supports and justifies such comparisons.

Assuming there was one, did the simple effects from the 2-way ANOVA use the pooled error term from all 8 group by treatment combinations (the MS within) as a basis for the error term? If so, might some of the other groups have had much greater variability than group A? That could increase the error term and make it harder to see differences. Similar issue if this was a repeated measures design.

With a 2-group comparison, the program may not apply any Bonferroni correction. With 3 groups, it would correct. So that could affect the p-value as well.

You saw the CI was the same. Do you mean the CI for the difference of means? If those are the same in the two analyses, then the p-values should be the same. But I doubt that is what you meant.

Ed