Question in interpreting Effects Size - versus p - Very large data sets

JDB

New Member
Good morning, Happy Memorial from the US

My question is how to interpret Effects Size statistics - eta squared - JASP application

Data Set very large >22,000 data observations

Am told in the literature that the p-values will be very small and useless.

I have looked at a dozen publication on Effect Sizes, why to use, why not to use this or that one

So I reran my AVOVA in JASP with eta squared, got some numbers and have NO idea how to interpret.

Here are two printouts. Any help or suggestions would be wonderful.

Stay safe, stay healthy, wear a mask.

obh

Active Member
Hi JDB,

I may read tomorrow the review.

When you have such a huge sample size, even a minor insignificant effect may be statistically significant.
In a small sample size, a small effect size may not be significant.
So yes you should also look at the effect size, always, not only in huge sample sizes.

The Eta square in ANOVA is the sample as the R square in regression, the ratio of the variance explained by the variable.
What is small or large effect is debatable and also related to the specific field, but I saw the following : small: 0.02. medium: 0.13, large 0.26

hlsmith

Less is more. Stay pure. Stay poor.
Yes, so if I follow, your variables seem to explain a fraction of a percent of the variability in the outcome. Which I would imagine is near useless. In addition, in order to look at interaction terms you need to have solid suspicion/theory that they could exist. You can't just look at every combination of variables - otherwise you will find spurious associations and may imagine they are generalizable when they are not.

obh

Active Member
Yes, so if I follow, your variables seem to explain a fraction of a percent of the variability in the outcome. Which I would imagine is near useless. In addition, in order to look at interaction terms you need to have solid suspicion/theory that they could exist. You can't just look at every combination of variables - otherwise you will find spurious associations and may imagine they are generalizable when they are not.
0.13 doesn't seem to be useless, but as you wrote, you can't just take every combination, you need to have theory, as randomly one of the combination may work.

hlsmith

Less is more. Stay pure. Stay poor.
@obh - agreed. I was just referring to the second (super saturated) model. I would image 13% would be useful.

JDB

New Member
thanks for the comments. I am now trying Cohen's d, that that is much more manageable and expandable . As you can surely tell, I was just introduced to effect size 2 weeks ago. I did a lot of calculations before I was told it was available in JASP

JDB

New Member
2 more papers - Tables comparing effects sizes - I found them helpful

Sullivan, G. M., & Feinn, R. (2012). Using effect size—or why the P value is not enough. Journal of graduate medical education, 4(3), 279-282.
free pdf - we like that

Fritz, C. O., Morris, P. E., & Richler, J. J. (2012). Effect size estimates: current use, calculations, and interpretation. Journal of experimental psychology: General, 141(1), 2.
Another free pdf.