Post-hoc power analysis for a Welch ANOVA and also a Kruskal-Wallis test.

[ble_21]

I planned to analyse some data with several ANOVAs (I have many different dependent variables), but much of the data violates the ANOVA assumptions, so they require Welch's ANOVA or the non-parametric Kruskal-Wallis test instead.

Using SPSS, I generated effect sizes (partial eta squared) and observed power values for the data which I analysed using an ANOVA.

I calculated estimated omega squared as an effect size for the Welch's ANOVAs using this formula:
est. omega squared = DFbetweensjcts(F-1) / (DFbetweensjcts(F-1)+N)

I calculated eta squared as an effect size for the Kruskal-Wallis test using this formula:
eta squared = (Kruskal-Wallis chi-square value) / (N-1)

The problem is, I am unsure how to calculate observed power for the Welch's ANOVA or Kruskal-Wallis test. Is it possible to do this? If so, how?

Also, I found the formulae to calculate omega squared and eta squared somewhere online, so I'm a little unsure whether these are correct.

Any help would be really appreciated!

 

[ondansetron]

I planned to analyse some data with several ANOVAs (I have many different dependent variables), but much of the data violates the ANOVA assumptions, so they require Welch's ANOVA or the non-parametric Kruskal-Wallis test instead.

Using SPSS, I generated effect sizes (partial eta squared) and observed power values for the data which I analysed using an ANOVA.

I calculated estimated omega squared as an effect size for the Welch's ANOVAs using this formula:
est. omega squared = DFbetweensjcts(F-1) / (DFbetweensjcts(F-1)+N)

I calculated eta squared as an effect size for the Kruskal-Wallis test using this formula:
eta squared = (Kruskal-Wallis chi-square value) / (N-1)

The problem is, I am unsure how to calculate observed power for the Welch's ANOVA or Kruskal-Wallis test. Is it possible to do this? If so, how?

Also, I found the formulae to calculate omega squared and eta squared somewhere online, so I'm a little unsure whether these are correct.

Any help would be really appreciated!

What do you hope to accomplish by conducting a post-hoc power analysis? What information will the "observed" power analysis provide for you?

 

[ble_21]

I would like to calculate power so that I can report whether any non-significant results were under-powered, because if so, then I will know that further research is needed with larger sample sizes. Also, some of the journals I hope to submit my results to ask that power is reported so that it can inform the design of future research.

 

[ondansetron]

I would like to calculate power so that I can report whether any non-significant results were under-powered, because if so, then I will know that further research is needed with larger sample sizes. Also, some of the journals I hope to submit my results to ask that power is reported so that it can inform the design of future research.

I see. Well, the appropriate way to answer your question is an a priori power analysis, meaning that it is done before the data are collected and seen. You specify a set of assumptions (alpha, effect size, etc) and a desired power to yield a sample size estimate to ensure the specified power level according to your assumptions. If you don't meet the sample size requirement and fail to reach significance on a test, you could propose underpowering as a reason.

However, post-hoc power analysis is of little value to answering this question, I believe. In a non-technical way, seeing the data and basing the power calculations off of it is like seeing the answers to a test and concluding you really earned your grade on the exam.

Maybe Dason could weigh in to verify or add theoretical clarification. As far as I know, though, post-hoc power analysis isn't entirely valid for assessing if your study was underpowered. I would probably prefer to state power as a potential reason and say a new study should include an a priori power analysis to assess the reproducibility of the conclusions.

 

[GretaGarbo]

I agree with what Ondansetron said.

Look at this and this and search for "post hoc powere of test".

 

[ondansetron]

Little did I know, I already have The Abuse of Power article downloaded and in a reading list. For the OP, read the article by Hoenig for further background, and use it as a reference to show the reviewers that it's not a well advised plan to utilize a post-hoc power analysis-- that is, assuming they asked for post-hoc and not a priori.

 

[Jake]

I agree with ondansetron and GretaGarbo.

To expand on their comments a little, "post-hoc power" (i.e., doing a power analysis where you just plug in the observed effect size, observed sample size, etc.) produces a "power" estimate that is simply a transformation of the p-value that you already observed. A p-value of p = .05 translates to post-hoc power = 50%. If you rejected the null, then post-hoc power will be >50%. If you failed to reject the null, then post-hoc power will be <50%.*

What this means is that if you already know the p-value for your analysis, then post-hoc power literally adds no new information. It is mathematically impossible to have a non-significant p-value & high post-hoc power; and it is impossible to have a significant p-value and low post-hoc power.

(* Note that this is slightly oversimplified; in truth the post-hoc power value that corresponds to observed p = .05 is 50% for large-sample z tests; for finite samples, this equilibrium power value approaches 50% from above as the sample size grows, but is slightly greater than 50% for small samples. See Russ Lenth's excellent paper on the topic.)

Also, some of the journals I hope to submit my results to ask that power is reported so that it can inform the design of future research.

Yes, but what they want is a prospective power analysis, not a post-hoc power analysis. Since you can't do a truly prospective power analysis (since you can't go back in time), what you can do instead is to present a power analysis for a range of small, medium, and large effect sizes (NOT just for your observed effect size) for your observed sample size, so that readers have an idea of what your a priori chances of detecting common effect sizes were. That would actually be informative, unlike post-hoc power.

 

[ble_21]

Thanks for your comments, everyone. These are all really useful things to know - I didn't realize that observed power is just a transformation of the p-value, but this seems pretty obvious in hindsight, after looking at my data!

I'll take on your advice and present a power analysis for a range of effect sizes for my observed sample size.

Thanks!