non parametric multiple pairwise comparisons - which correction is allowed?

#1
Hello all,

I have a data set from plant experiments with only 4 replicates. data are numbers of mites after different treatments. Because the datasets are small they are not normally distributed, so I use non parametric tests: Wilcoxon. First I did a pairwise comparison, without corrections which is not allowed normally, but I see the expected differences. then I did the same test with Bonferroni correction and don't see any significant differences anymore (alpha =0,1). Thirdly I did the same test again with BH correction and I see the differences again.

So my question is, am I allowed to apply the BH correction for analysing my data? Or Do I have to apply bonferroni correction and thus have to conclmude that there are no statistical differences and I should do more replications in future experiments?

Here my simplified data and the results of my 3 tests.


A B C D E F G H
111 47 0 102 1 0 0 12
88 31 0 152 0 0 0 2
216 35 0 102 0 1 1 75
169 30 1 75 0 3 6 46



Pairwise comparisons using Wilcoxon rank sum test

P value adjustment method: none

A B C D E F G
B 0.029 - - - - - -
C 0.027 0.027 - - - - -
D 0.309 0.029 0.026 - - - -
E 0.027 0.027 1.000 0.026 - - -
F 0.029 0.029 0.505 0.028 0.505 - -
G 0.029 0.029 0.505 0.028 0.505 1.000 -
H 0.029 0.886 0.027 0.041 0.027 0.059 0.059




Pairwise comparisons using Wilcoxon rank sum test

P value adjustment method: bonferroni

A B C D E F G
B 0.80 - - - - - -
C 0.74 0.74 - - - - -
D 1.00 0.82 0.72 - - - -
E 0.74 0.74 1.00 0.72 - - -
F 0.82 0.82 1.00 0.80 1.00 - -
G 0.82 0.82 1.00 0.80 1.00 1.00 -
H 0.80 1.00 0.74 1.00 0.74 1.00 1.00

Pairwise comparisons using Wilcoxon rank sum test

P value adjustment method: BH (Benjamini, Hochberg)

A B C D E F G
B 0.048 - - - - - -
C 0.048 0.048 - - - - -
D 0.413 0.048 0.048 - - - -
E 0.048 0.048 1.000 0.048 - - -
F 0.048 0.048 0.566 0.048 0.566 - -
G 0.048 0.048 0.566 0.048 0.566 1.000 -
H 0.048 0.954 0.048 0.064 0.048 0.083 0.083

Kind Regards
Joachim
 

hlsmith

Less is more. Stay pure. Stay poor.
#2
In a scientifically rigorous world you would have had a protocol drafted prior to conducting the research, stating which approaches you were going to run. Then you would be 100% married to the protocol. So you probably should be committed to the BC, which is considered fairly cautious for false discovery. If you were ignorant (like many) and just selected the first and easiest correction you knew of, then went fishing to restore the possible significance, then it may be up to you due to ignorance. Ideally you would disclose this with your results, if you flip-flopped. Not trying to be cynical with the word "ignorant" just matter of fact.
 
#3
Thats indeed my whole point. I'm trying to improve my statistical data analysis and would like to know what the best tests are for our type of research. Because I see that most people I know just do pairwise comparisons with the Wilcoxon test and don't even know they need to correct it for multiple comparisons. Currently I'm trying to develop this protocol "to get married to". So I'm writing a script that does the tests and makes you use a specific test under specific circumstances. Because I'm not a statistician, I need some help in choosing the right test. currently my protocol is as follows. Please correct me if it doesn't seem right to you.

1) I test for normality with boxplots, histograms and the shapiro wilk test for normality
2) I check for homoscedasticity with the levene test
3) if 1 and 2 are positive I do parametric tests if not non parametric tests ==> for our type of data we usually end up in the second group

4) PARAMETRIC TESTS
4a) if there are only two groups to compare I use the t-test
4b) if there are more groups to compare I use ANOVA to check for differences in means
4c) if there are differences I use the Tukey test for pairwise comparison

5) NON PARAMETRIC TESTS
5a) if there are only 2 groups to compare I use the Wilcoxon rank sum test, is this test the same as the Mann Whitney test?
5b) if there are more groups to compare I use the Kruskal Wallis test to check for differences in means
5c) if there are differences I do pairwise comparison with the Wilcoxon rank sum test, but I know I have to do a correction for the multiple comparison. I also know there are Bonferroni, Benjamini Hochberg corrections etc. But I hove no idea which one I should use. I want to put this decission in my protocol, so we can't choose the correction which will make our data significantly different the way we would like it to be.

All comments are welcome.