adjustment for multiple comparisons in meta-analysis

[ludomuzii]

I wrote a meta-analysis on the use of two different drug regimens on the symptoms associated to endometriosis. Four papers were included in the meta-analysis.
The two regimens were evaluated as to the results in four outcomes: recurrence of the disease as seen at ultrasonography during follow-up, and the recurrence of three pain symptoms (three different symptoms evaluated separately: dysmenorrhea, non-cystic pelvic pain, and dyspareunia).

I received comments from the reviewers, and one of them asks the following:

"The authors actually look at four outcomes - the recurrence of endometrioma and recurrence of dysmenorrhea, non-cyclic chronic pelvic pain and dyspareunia. However, no adjustment for multiple comparisons is provided and all outcomes are considered statistically significant at p<0.05. The authors should utilize a procedure to control the overall alpha level."

How should I change the manuscript for resubmission? I do not really understand the comment. Should I use a different alpha level, and not the usual 0.05? Which procedure should be used to control the overall alpha level? Is this really “multiple comparisons”? I think it is only multiple, different outcomes.


Thank you very much

 

[Englund]

Have a look at Bonferroni corrections along with it's many closely related alternatives. The problem with using hypothesis tests when multiple tests are carried out is that there is a greater chance of observing a p-value below the alpha level for at least one test.

 

[Englund]

Given independent tests we have the following, which is why we must correct the original alpha level:

\(P(\text{at least one significant value})\) \( = 1-P(\text{all non-significant}) = 1-(1-\alpha)^k > \alpha\)

 

[hlsmith]

I may side with your last point, in that you are not conducting pairwise comparisons - but 4 different analyses. I typically reserve alpha corrections for multiple tests within an analyses (e.g., say outcomes in induced pregnancies, natural, Caesarian). I don't recall seeing any meta-analyses correcting for multiple outcomes.


A follow-up may be, did any of the original studies correct for this, have you seen any other meta-analyses doing this? Also, a side question, are most of your pvalues fairly lower than 0.05?

 

[ludomuzii]

Have a look at Bonferroni corrections along with it's many closely related alternatives. The problem with using hypothesis tests when multiple tests are carried out is that there is a greater chance of observing a p-value below the alpha level for at least one test.

thank you very much!

 

[ludomuzii]

thank you very much.
no, actually i do not think that any of the original studies corrected for this, neither i can recall of other meta-analysis doing this