Power analysis

[OneXero]

Dear all,

First of all I would like to thank you for helping me with this question.

For my research I investigated 23 patients. I looked at there range of motion of the shoulder and investigated the angle at which a dislocation/relocation occurred during forward flexion (FF) and abduction (ABD). These are my results:

Dislocation FF (°): Mean = 141° (SD=30.0)
Dislocation ABD (°): Mean = 101° (SD=18.9)
P value (Pared student T Test): P < 0.001

Relocation FF (°): Mean= 96° (SD =42.3)
Relocation ABD (°): Mean = 75° (SD = 23.2)
P value (Pared Student T Test): P = 0.124

A statistically significant difference (p<0.001) was found between the angle of dislocation during FF compared to abduction. Mean angle of dislocation during FF was 36 degrees higher compared to abduction, 95% CI = [19;53].
However, the reviewer commented the following: the power for the comparison of differences for flexion and abduction is 0.4 (according to the values presented in the table). Please mention as the data is rather not sufficient to support a statement, that there is no difference.

Can someone explain me how he got this 0.4. And how I can correctly describe this in my research?

Thanks in advance!

 

[hlsmith]

Well first off, a higher p-value means you failed to reject your null hypothesis, not evidence towards supporting it. You likely did not use the proper terminology and are getting dinged and will likely remember this before you submit results again.

They likely conducted a power analysis for a paired t-test, given:

sample size: 23
alpha: 0.05
mean differences;
and SD values.

Of note, conducting a post hoc power analysis is usually faux paus, but can provide some insight. You can see your issue is that huge SD for relocation FF. You should try not to use pvalues and focus on the differences between the two groups' values as a histogram. Doing this allows the reader to visually understand how different the groups were given the sample.

 

[OneXero]

Dear hlsmith,

thank you for your quick response.

If I understand correctly I can say that there is a (statistical significant) difference between the 2 groups for dislocation of FF compared to ABD but that there is no difference for the relocation between both groups (because difference is not significant). Also I will need to mention a low power for the difference of relocation of FF compared to ABD. Or is it because the power in general is not high enough that i cannot discuss the significant difference between dislocation angles of FF and ABD?

The difference between angle of relocation is not of importance to me in this research i just mentioned the results in the table. Therefore I do not understand why the power analysis here is of that great importance?

 

[obh]

Hi OneXero,

Did you calculate the sample size based on the effect size you want to achieve before conducting the research?

If you only calculate the power now you should do it based on the expected effect size not of the actual effect size.
Say what difference did you want to be able to identify with your test? 1°? 10°? 30°?

For dislocation, there is a significant difference since p<0.05 (p<0.001) (assuming you use α=0.05)
The power remark was related to the Relocation ...

"How did he get 0.4?"

To calculate the power you need to know the standard deviation of the differences for each patient. (assuming paired t-test)

You can use http://www.statskingdom.com/30_test_power.html
With one sample t-test this is exactly the same as paired-t you just need to use the differences.

 

[hlsmith]

Don't list a p-value if you don't want someone to call it out. I wouldn't mention low power, but would mention small sample size and high variability. You shouldn't report power after the fact that you conducted the analyses already.

 

[hlsmith]

@obh - good point about conducting the a priori presumed power calculation and not a post hoc analysis!

 

[obh]

I think the same , but you should calculate the priori power (based on the expected effect, which you should do before) , so you could understand the research limitations.

0.124 is bigger than 0.05 but still small. If the priori power of the research is low, there is a not small probability that with a powerful research the results would be significance.

So if the power is small, you can't conclude any "final" conclusion, only show a direction like hlsmith suggested.

 

[hlsmith]

Funny, you got me thinking about this a little more @obh

Likely the calculation was always under-powered, but perhaps the original a priori power calculation had say, power = 0.60 and the actual post hoc was power = 0.70. Where by chance (sampling variability or maybe slightly ill-informed prior knowledge) the OP got a better result than would have been expected. This question opens up all kinds of secondary inquiries about repeated testing, publication bias, how hypotheses may be a random variable of it's own, etc. Fun stuff.

 

[obh]

correct hlsmith that we are dealing with random variables. if for example the priori power is 0.2 which is very law. Still in 20% we will get significant results (if of course there is a significant difference ). Post hoc is highly correlate with the p value.I prefer to use only the priori power and if somebody doing the power calculation after the research, it should be priori like power with the expected effect size.

 

[hlsmith]

Agreed and the post calculation is conditional on result, meaning you got something and that is the reason for the calculation.

 

[OneXero]

Hello all,

Thank you for the very interesting discussion.
Can you please explain how I calculate the a priori power?

 

[obh]

Hi OneXero,

Did you look at the link I mentioned?

First, you need to decide what effect size you want to be able to identify with your test? 1°? 10°? 30°?
The power is a function of the effect size.

Then you calculate what is the probability that the test will reject H0 assuming H1 is correct.

For example, if you want to be able to Identify a change of 10° and Mean= 96° you calculate the case for H1 with Mean = 96+10 =106

For t-test, H1 distribute non-central T.

In the link, you need to choose distribution: t, sample: one-sample

you can use the standard deviation of the differences (differences between the values in the pairs in pair t) from your research, not the standard deviations of each group, and the unstandardized effect size
if you wouldn't know the standard deviation (which is not this case) you would use standardize effect size.

You will get the power and the chart that explain the calculation.

Power calculator:
http://www.statskingdom.com/30_test_power.html

You can also look at the following link for more explanations of power.
http://www.statskingdom.com/doc_test_power.html

If you prefer to use R:
library(pwr)
pwr.t.test(n = 23, d = 0.5 , sig.level = 0.05 ,alternative="two.sided" , type = c("paired"))

sigma - "standard deviation of the differences"
d=expected effect size/sigma

you should choose your preferred tail (alternative) from the following: “two.sided”, “less”, “greater”






.