How common is it for researchers to omit a power calculation?

#1
Hi,

I'm currently writing my dissertation for my final year at university. The stipulation that I must discuss collectively the topic of statistical power has sent me down somewhat of a rabbit hole.

I started by looking through the 12 studies I have for my systematic review for mentions of power. I also learnt that studies that did not mention power may have instead reported confidence interval, and considered that sufficient. There was also the possibility that the journal set a standard for power, that studies would have to meet before publishing, meaning that it may not have been explicitly stated in the manuscript.

I read through all the studies in my review, and found that only three out of the twelve had mentioned any sort of power calculation. Moreover, the nine studies that didn't discuss power also had no allusion to it, such as discussing effect size or confidence interval. As far as what I can tell online, omitting power calculation is very bad practice.
Can it honestly be the case that nearly all the studies in my review haven't calculated power? or may I be missing something due to my lack of experience reading papers?

Most of the sample sizes were small, from around 20-50 participants, apart from one or two studies that used sample sizes in the 100s.
Considering I have to discuss power, if the above is true my discussion will inevitably centre around the poor decision to not measure power, and how the strength of the research as a whole and my review will be lower because of it. I'm nervous to point this out, as I thought reporting power was just one of those things that was mandatory to include, as how can you determine the usefulness of the result if your sample size and power were too low to be relevant?
Is there anything else I could've missed?

Many thanks!

Also, here are the studies I'm talking about.

1. Babault, N., Païzis, C., Deley, G., Guérin-Deremaux, L., Saniez, M. H., Lefranc-Millot, C., & Allaert, F. A. (2015). Pea proteins oral supplementation promotes muscle thickness gains during resistance training: a double-blind, randomized, Placebo-controlled clinical trial vs. Whey protein. Journal of the International Society of Sports Nutrition, 12(1), 1-9.

2. Candow, D. G., Burke, N. C., Smith-Palmer, T., & Burke, D. G. (2006). Effect of whey and soy protein supplementation combined with resistance training in young adults. International journal of sport nutrition and exercise metabolism, 16(3), 233-244.

3. DeNysschen, C. A., Burton, H. W., Horvath, P. J., Leddy, J. J., & Browne, R. W. (2009). Resistance training with soy vs whey protein supplements in hyperlipidemic males. Journal of the International Society of Sports Nutrition, 6(1), 8.

4. Fouillet, H., Mariotti, F., Gaudichon, C., Bos, C., & Tomé, D. (2002). Peripheral and splanchnic metabolism of dietary nitrogen are differently affected by the protein source in humans as assessed by compartmental modeling. The Journal of nutrition, 132(1), 125-133.

5. Hartman, J. W., Tang, J. E., Wilkinson, S. B., Tarnopolsky, M. A., Lawrence, R. L., Fullerton, A. V., & Phillips, S. M. (2007). Consumption of fat-free fluid milk after resistance exercise promotes greater lean mass accretion than does consumption of soy or carbohydrate in young, novice, male weightlifters. The American journal of clinical nutrition, 86(2), 373-381.

6. Joy, J. M., Lowery, R. P., Wilson, J. M., Purpura, M., De Souza, E. O., Wilson, S. M., ... & Jäger, R. (2013). The effects of 8 weeks of whey or rice protein supplementation on body composition and exercise performance. Nutrition journal, 12(1), 1-7.

7. Kalman, D., Feldman, S., Martinez, M., Krieger, D. R., & Tallon, M. J. (2007). Effect of protein source and resistance training on body composition and sex hormones. Journal of the International Society of Sports Nutrition, 4(1), 1-8.

8. Tang, J. E., Moore, D. R., Kujbida, G. W., Tarnopolsky, M. A., & Phillips, S. M. (2009). Ingestion of whey hydrolysate, casein, or soy protein isolate: effects on mixed muscle protein synthesis at rest and following resistance exercise in young men. Journal of applied physiology.

9. Wilkinson, S. B., Tarnopolsky, M. A., MacDonald, M. J., MacDonald, J. R., Armstrong, D., & Phillips, S. M. (2007). Consumption of fluid skim milk promotes greater muscle protein accretion after resistance exercise than does consumption of an isonitrogenous and isoenergetic soy-protein beverage. The American journal of clinical nutrition, 85(4), 1031-1040. REPORTED POWER

10. Brown, E. C., DiSilvestro, R. A., Babaknia, A., & Devor, S. T. (2004). Soy versus whey protein bars: effects on exercise training impact on lean body mass and antioxidant status. Nutrition Journal, 3(1), 22.

11. Banaszek, A., Townsend, J. R., Bender, D., Vantrease, W. C., Marshall, A. C., & Johnson, K. D. (2019). The effects of whey vs. pea protein on physical adaptations following 8-weeks of high-intensity functional training (HIFT): A pilot study. Sports, 7(1), 12. REPORTED POWER

12. Volek, J. S., Volk, B. M., Gómez, A. L., Kunces, L. J., Kupchak, B. R., Freidenreich, D. J., ... & Quann, E. E. (2013). Whey protein supplementation during resistance training augments lean body mass. Journal of the American College of Nutrition, 32(2), 122-135. REPORTED POWER
 

hlsmith

Less is more. Stay pure. Stay poor.
#2
What would be your rebuttal if i said, if they found an effect beyond chance the study was powered? However it doesnt mean that it may still be a false positive found via a secondary objective or a result of investigator degrees of freedom.
 
#4
Its very common, in fact I prefer it. Periodically someone will write something about how not reporting this is very important to 'quality', then people do it for a while, then stop. This cycle continues with no observable effect on the science.
 
#5
What would be your rebuttal if i said, if they found an effect beyond chance the study was powered? However it doesnt mean that it may still be a false positive found via a secondary objective or a result of investigator degrees of freedom.
As Andrew Gelman demonstrated., a "statisticaly significant" result from a vastly underpowered study (i.e., by chance, an effect was measured in the sample which had a much larger absolute size than the "true" effect in the population, because only then it could take the significance hurdle) has a nearly 50% chance to be in the wrong direction.

With kind regards

Karabiner
 
Last edited:
#8
how do you know if it is underpowered?
For the discussion I was referring to, one can easily construct a plausible scenario like: "with n=30, a population
effect d=0.1 will lead to a statistically significant effect in 10% of tests." I'd guess a 10% power can be
considered vastly underpowered, given that we'd like to achieve 80% by default (the 10% are made
up by me, I'm too lazy right now to do the proper calculations). As far as I can see, in several disciplines
such as psychology, education, biology etc., the common effect sizes can be assumed to be around
d=0.1 to d=0.3 (taking also in to account measurement error).
Such problems are nicely (IMHO) subsumed in ths graph https://media.springernature.com/lw...-5/MediaObjects/10940_2017_9374_Fig1_HTML.gif
from https://link.springer.com/article/10.1007/s10940-017-9374-5

With kind regards

Karabiner
 
#10
I don't know, seems like tail wagging the dog here. You know that a study is underpowered because you can 'easily construct a plausible scenario', then how do you know the plausible scenario holds? I am more familiar with the mantra hlsmith was reciting above, ie
if they found an effect beyond chance the study was powered
, I don't reckon its a coincidence the trop sounds so familiar, but I too am lazy and don't want to go looking up references on the issue.
 
#11
It looks like the exaggerating ratio is extremely large when the power is less than 0.2, say around 2 when the power is 0.2, and asymptote to one only with the power of 0.8.

And the type S error (wrong effect direction for significance result) rate is an asymptote to zero with the power of 0.2.
So if the power is less than 0.2 you may get an incorrect direction of the effect.

But you mention the sample size was 20-50.
I don't know what tests they run, but if you would take for example tow sample t-test (pooled), and n1=20, n2=20

You calculate the test power based on the required effect size.
For a large standardized effect size (0.8), the test power is 0.69
For a medium standardized effect size (0.5), the test power is 0.34

So the direction of the effect will be correct, and the effect size will be almost accurate for large effect size,
but exaggerate effect size for medium effect size.
 
#12
I don't know, seems like tail wagging the dog here. You know that a study is underpowered because you can 'easily construct a plausible scenario', then how do you know the plausible scenario holds? I am more familiar with the mantra hlsmith was reciting above, ie , I don't reckon its a coincidence the trop sounds so familiar, but I too am lazy and don't want to go looking up references on the issue.
Seems to be a misunderstanding here. I was just referrring to the fact that a "statisticaly significant" result in a
vastly undeporwered study (in contrast to a large study) can easily be in the wrong direction. As for knowing
whether a study is markedly underpowered, I guess that this is not too difficult to assess, at least in sciences
with biological entities as subjects. Usually, I do not need a power calculation for knowing what to expect from
an n=20 or n=30 study in psychology or medicine, where mostly we deal with small to medium effects. If
a researcher doing a small study convincingly claims that he expects a large effect, then maybe a power calculation
might be interesting.

With kind regards

Karabiner
 
#13
oh i see. it can sort of be interpreted as 'science is hard if you can't explain much of the variation you are seeing'. Some medicines are very effective, for example many vaccines are >70% effective relative to placebo, at least in a lab. Condoms are also very effective, ive been told.