What is the difference between statistical significance (based on probability) and...

[Kurisu87]

(Questions wouldn't fit)

What is the difference between statistical significance (based on probability) and an effect size?

I am fairly new to stats and finding a bit of an up hill marathon at the moment. I have been asked to research the question posted above, yet I have exhausted my resources and I am still non the wiser on how to accurately answer the question.. I really do hope you guys/girls can help me out of this slowly deepening hole!

All the best!

 

[Mathis]

Re: What is the difference between statistical significance (based on probability) an

I think a lot of people mix them up.

You can, for example, have a small effect size, say your outcome increases by 0.01 per 1 increase in predictor. Yet with very high significance, say only a probability of 0.001 of finding this effect under the null hypothesis (that the effect is zero). This will depend on the standard error, which in turn depends on your variance and sample size. Then you can say with reasonable certainty the effect is very small.

You can also have a very large effect, but with low significance; a large effect with high significance; a small effect with low significance.

The effect is the parameter estimate, the significance is reported as the p-value.

Hope this helps. It's not easy to explain in layman terms here, and I find a graphical representation often helps interpret the significance.

 

[MrJP]

Re: What is the difference between statistical significance (based on probability) an

Mathis is completely right.

The effect size is the how much your predictor influence your dependent variable. Significance is about the certainty of the effect.
Besides giving a number for effect size, there is a interval (range) expressed as well, for example 0.5 to 1.0, and with 95% percentage certainty the effect size in the complete population is within these numbers. If this range doesn't include 0, it's significant.

So if it ranges from 1.0 to 2.0 it's significant, if it ranges from -1.0 to 10.0 isn't not.

To test 95% percentage certainty is most used in psychology, altough other %'s can be used as well.

 

[hlsmith]

Re: What is the difference between statistical significance (based on probability) an

In addition (to Mathis slight reference), the variability and sample size come into play. For example, if I have a huge dataset, I can find a highly statistical significant difference, but this difference (effect) may actually be miniscule and not relevant or note worthy at all. On the other hand, in a small sample I can find a huge difference (effect), but due to variance and sample size it is not statistically significant at all. The concept of statistical power comes into play.

 

[noetsi]

Re: What is the difference between statistical significance (based on probability) an

I always thought the effect size was the mean difference between level of the DV on the IV. Effect size differences (defined that way) involve a substantive interpretation - is the mean difference enough to matter. Statistical test in contrast are really testing whether the results you find in the sample is likely to actually exist in the population (as compared to being random error). It says nothing at all about the substantive level of the effect. An increasing number of methodologist have argued that hypothesis test be abandoned because they distort the importance of effect - so that people assume that the effect size discovered is signficant because it is "statistically" significant which is not at all the case.