Power calculations help..

#1
Hello all,

I don't suppose i can ask a question on power calculations?

I've just conducted a RCT involving two student groups one undertaking an educational intervention and another receiving standard clinical teaching. Both groups did a 24 item MCQ knowledge assessment and a 7 statement questionnaire on confidence using 5 point Likert scales.

The power calculations are a bit haphazard as the original statistician bailed out and I'm trying to pick up the pieces in writing this.

The statistical analyses i used were
paired sample t-tests for the 24item MCQs
Wilcoxon signed rank tests for the confidence questionnaires.

Note: I didn't make any inter-group comparisons, only intra-group pre/post.

Using the G-Power calculator I did an A priori calculation assuming power of 0.8 and an effect size of 0.5 and noted I needed a sample size of 34 and 35 for the paired t and wilcoxon signed tests respectively.

I achieved sample sizes of 55 and 53.

I then did some post hoc calculations after calculating the effect sizes I achieved and noted that my paired t-tests had sufficient power of >0.95 but of the 7 statements in my confidence questionnaire (Analysed using Wilcoxon signed rank tests) 2 of the 7 statements had powers of <0.8 (One was 0.65 and another was 0.12)

Am I correct in assuming then that my a priori precalculations guided me towards the appropriate sample sizes and the post hoc calculations confirmed that I had sufficient power or not? (And that i did all the above correctly instead of going on a tangent)

How do I present this? Am presenting this work next month and if they ask me did i attain sufficient power would my answer be: yes for the paired t-tests and 5 of the 7 statements only.

Many thanks!
 

fed1

TS Contributor
#2
Your post hoc calculations are a little awkward. I wouldnt do that part.

Just use your a priori calculations. The effect size you choose for your
power calculations should depend on what you feel would be a meaning full
change. The old rule of thumb is effect sizes of 0.2, 0.5 and 0.8 are samll mediumand large, or something like that. Use the sample sizes you actually had, 50 or whatever.

The point of power calculations is not to find the chance of detecting the difference that truly exhists (we dont know that). It is to find the chance of detecting a difference under a specific alternative (Ha).

:wave:
 
#3
Many thanks for this.

Yes I was aware of the 0.2, 0.5 and 0.8 rule.

Does it mean though that I can say my study had sufficient power across all outcome measures? And my defence of this is the a priori calculations and attaining the sample sizes?

I'm just concerned that one of my outcome measures in the Likert scale based questionnaire shows a non-significant result and according to the post-hoc calculations it says I don't have enough power.
 

fed1

TS Contributor
#5
So a priori you had some sample size and effect size
you based your calcs on

n1, delta1.

for which the proposed experiment had 80% power.

You only got n2 < n1 and a point estimate of delta_hat < del.

Its worth adjusting the power calculations if you did not find
a significant difference (if you did power calcs are moot, obviously)
but you should only change n1 -> n2.

You chose delta1 because it represented a clinically significant difference.
It is a clinically signifcant difference no mattter what delta_hat you realized.

At least thats my take.