Applying the Bonferroni Correction to a T-Test considered

[Vthompson]

Hello! I need your help please
I am comparing the mean scores for an exam for one class (A) to four different classes (B,C,D,E), such as AB, AC, AD and AE.
The data for all classes is normal but in class E. I am performing a T-Test for AB, AC and AD only, but using the Mann Whitney test for the comparison of AE being that A is normal and E not normal.
I am applying the Bonferroni Correction to take the family wise error into account.
Therefore new alpha is .05/4 = .0125.
I am using SPSS to run the test for me. The T Test reports the outcome for Levene's Test for Equal Variances.
My question is, do I use the p value .0125 when looking for the results from Levene's test? That is if p>.0125, equal variance are assumed, and if p<.0125, equal variances are not assumed. This will show me how to interpret my p value for the T Test.

Thanks much for your input.

 

[obh]

Hi V,

1. Bonferroni Correction is conservative, as it assumes all groups are totally independent
2. If you run 3 comparisons you should divide by 3, not by 4.
3. What is your sample size for each group?
4. Better to run Welch's t-test (unequal variables)

 

[noetsi]

I think its a fascinating question how often this approach is used. I don't think I have ever seen it reported in an actual journal although I am sure it is sometimes.

 

[hlsmith]

It does seem they have 4 comparisons, so the '4' would be appropriate. I agree that you should just run the unequal variance version of ttests for all groups. This makes it easy and cautious. Lasly, @obh is likely probing to see if your sample is large enough to ignore non-normality. Side note, the normality is in the residuals not the data and with a large enough sample the ttest will likely do fine.

 

[hlsmith]

I think its a fascinating question how often this approach is used. I don't think I have ever seen it reported in an actual journal although I am sure it is sometimes.

I see it regularly and use it regularly.

 

[noetsi]

I see it regularly and use it regularly.

I just have never seen it mentioned in journals and I have read quite a few They always report some nominal alpha level not adjusted, or which they don't say is adjusted.

We probably read different journals.

 

[Vthompson]

Hello O,
Thanks for your reply!
1) Shouldn't I divide by 4 because I am comparing AB, AC, AD and AE? Or are you saying because I am using a T-Test for AB, AC & AD, I should divide by 3? But in this case, when using the Mann Whitney test for comparing AE, do I leave alpha at .05?
2) The sample sizes are: A=21, B=29, C=30, D=30 and E=27.
3) I just ran the Welch's t-test for unequal variables, thank you!

 

[Vthompson]

It does seem they have 4 comparisons, so the '4' would be appropriate. I agree that you should just run the unequal variance version of ttests for all groups. This makes it easy and cautious. Lasly, @obh is likely probing to see if your sample is large enough to ignore non-normality. Side note, the normality is in the residuals not the data and with a large enough sample the ttest will likely do fine.

Thanks for your input!!!

 

[obh]

Hello O,
Thanks for your reply!
1) Shouldn't I divide by 4 because I am comparing AB, AC, AD and AE? Or are you saying because I am using a T-Test for AB, AC & AD, I should divide by 3? But in this case, when using the Mann Whitney test for comparing AE, do I leave alpha at .05?
2) The sample sizes are: A=21, B=29, C=30, D=30 and E=27.
3) I just ran the Welch's t-test for unequal variables, thank you!

Hi V,

Sorry for a moment I thought you plan to run 4 comparisons and run only 3 ...reading again I see I misunderstood.
If you run 4 tests you should divide by 4, doesn't matter what test you run.

You may try to run Holm correction, which is a bit less conservative.

How did you decide E is not normal? (did you run 4 normality tests...?)
If E is not extremely asymmetrical you can run a t-test as well, the rule of thumb is 30, but 27 should be okay.

 

[Vthompson]

Thanks O for all your input.
Yes, I tested all the data sets separately using the Shapiro-Wilks test which found the data set not normal for E. The p value was .004.

 

[obh]

So the non-normal data isn't related to multiple testing, but if the data is reasonably symmetrical than you may use the t-test.

PS, unless you want only to identify a large effect size, the test power might be too low.

 

[Vthompson]

Yes it is related to the multiple testing ( data set A-normal will be compared to data set E-non normal). I will use the Mann-Whitney for that one.

 

[obh]

I meant that you may found incorrect non-normal data due to the multiple SW testing, but since the p-value is very small this is not the case.
If the E data is reasonably symmetrical than you may use the t-test.