Multiple comparisons of dependent (and independent) tests

ÖdM

New Member
#1
Hi! - How does one correct for multiple comparisons after a series of statistical tests on partially overlapping data?

For example, I am currently doing a sensitivity analysis in checking how the volume of a spherical region-of-interest (ROI) influences comparisons of brain activity when comparing participants before and after a training intervention. There are 4 ROIs, and each have 3 different sizes. This is the table of p-values:

4mm 6mm 8mm
ROI1 0.020 0.059 0.029
ROI2 0.047 0.043 0.035
ROI3 0.047 0.197 0.011
ROI4 0.012 0.001 0.015

Obviously, the row-wise results are likely to show similarities given that the data is partially overlapping, while the p-values in each column are more independent. Bonferroni or FDR-correction does not really make sense to me in this situation.
Suggestions would be very welcome.
 
#2
Since a number of people have looked at this question (but not provided an answer), here's an answer I got that might be of interest to you from Dr. Daniel Wilson at Oxford:

1. You can use the HMP, but beware that under certain forms of dependence (not investigated in the original paper) its false positive rate can be elevated. Worst case I have seen so far is from 5% (the target) to 7%. There will be a letter published in PNAS soon on this, with a response from me.

2. You can use a multilevel version of the Simes test, which will be described in my letter soon to be published.

Since HMP and Simes are multilevel tests, it is valid to report the significant p-values at the finest level of resolution within each hierarchy of results.