Practical significance

_joey

New Member
#1
The difference between two mean values is 103 units
p-value is 0.00001 at alpha=0.05
95% confidence interval is (50, 150)
difference of 100 units is of practical importance.
Would the mean difference be practically significant for the given difference between two mean values and confidence interval?

Thanks!
 

_joey

New Member
#2
Here you are dealing with margin of equivalence. That margin of practical significance extends from -100 to +100, and your 95% CI lies somewhere within and somewhere outside it. You should state that "the results are not conclusive in terms of practical significance".
i don't know what 'margin of equivalence' is. The question clearly states "difference of 100 units is of practical importance". The difference of the means obtained is 103 units at 0.005 significance level.
 
#3
Here you are dealing with margin of equivalence. That margin of practical significance extends from -100 to +100, and your 95% CI lies somewhere within and somewhere outside it. You should state that "the results are not conclusive in terms of practical significance".

--------------

i don't know what 'margin of equivalence' is. The question clearly states "difference of 100 units is of practical importance". The difference of the means obtained is 103 units at 0.005 significance level.
It means that the difference less than 100 unit is not practically significant. Say, if two people have an 88unit difference, although they may be statistically significant, they are not practically significant as their difference is less than 100.

In such cases, some methods can be applied. The best maybe using the confidence intervals. Check the bounds of CI. Both should stand outside the -100 to 100-unit band of "practical signficance" to certainly conclude whether or not there was any "practical" difference. In your case, only one of them lies outside this range. So, you can't be sure if the difference is practically significance or not, and further studies with better powers are necessary.
 

_joey

New Member
#4
It means that the difference less than 100 unit is not practically significant. Say, if two people have an 88unit difference, although they may be statistically significant, they are not practically significant as their difference is less than 100.
But the difference calculated is _not_ 88 units, it's 103 units.

In such cases, some methods can be applied. The best maybe using the confidence intervals. Check the bounds of CI. Both should stand outside the -100 to 100-unit band of "practical signficance" to certainly conclude whether or not there was any "practical" difference. In your case, only one of them lies outside this range. So, you can't be sure if the difference is practically significance or not, and further studies with better powers are necessary.
The lower bound of CI should be above 100 or below -100 units?
 
#5
even if your difference is 25500^1000 units, but the lower bound of CI stands "within" that mentioned -100 to 100-unit range, you cannot be sure about practical significance! :) I repeat that both bounds should stand outside this range. It means that if the upper bound is greater than +100, the lower bound must be as well OR if the upper bound stays below -100, the lower bound must be lesser than -100 in order to be sure that your difference is practically significant.
 

_joey

New Member
#6
even if your difference is 25500^1000 units, but the lower bound of CI stands "within" that mentioned -100 to 100-unit range, you cannot be sure about practical significance! :) I repeat that both bounds should stand outside this range. It means that if the upper bound is greater than +100, the lower bound must be as well OR if the upper bound stays below -100, the lower bound must be lesser than -100 in order to be sure that your difference is practically significant.
I hear you. Some authority references would be nice to have to substantiate your comments because I haven't found anything on the web in regards to the CI bounds and practical significance.
 

noetsi

No cake for spunky
#7
To me practical signficance would seem to indicate if the mean difference is substantively important, which is distinct from any statistical property such as a p value. That is a decision that involves judgement and the norms of the field not statistics. It is very much context driven.
 

_joey

New Member
#8
To me practical signficance would seem to indicate if the mean difference is substantively important, which is distinct from any statistical property such as a p value. That is a decision that involves judgement and the norms of the field not statistics. It is very much context driven.
That's what I thought too. Since, 100 units is of practical significance then the difference of 103 units would be practically significant too. victorxstc, however, is talking about CI bounds. I think he is referring to size effect.
 

noetsi

No cake for spunky
#9
I would argue, and the statisticians I have read, that elements such as CI bounds has nothing to do with substantive importance. Much of statistics gets at if the results you find could be tied to random error. That is entirely seperate from issue of whether the effect discovered (if real) matters.
 

Dason

Ambassador to the humans
#10
So are you telling me that if the confidence interval had been (-1000, 1206) and the point estimate was 103 you would be completely fine saying that there is a practically significant difference?
 
#11
@Joey

It was my very essential problem about 6 months ago, for which I searched A LOT to be able to find only a couple of resources. I agree that there were very few topics about it, but you will find that my comment is valid if you search more :)

@noetsi

I agree that clinical significance is something subjective. But sometimes, they manage to assume some objective thresholds (based on some research) for defining clinical significance. This way, it can be semi-objective. the case of Joey seems something like that, where he can use numbers as well (again it is subjective, but better now).
 

noetsi

No cake for spunky
#12
If you discover an effect size you believe are real, what does it matter what the CI is? Alternately if the CI (or p value etc) suggests that the result is real but it is tiny what is the impact practically of that? Statistics and substantive value are entirely different. The only thing statistics can tell you is if the results are likely to be random error not if they matter. That is outside the realm of statistics a point increasingly being emphasized in statistical texts.

I agree that clinical significance is something subjective. But sometimes, they manage to assume some objective thresholds (based on some research) for defining clinical significance. This way, it can be semi-objective. the case of Joey seems something like that, where he can use numbers as well (again it is subjective, but better now).
I agree entirely. My point is that objective thresholds are not determined by statistics but by impact and professional judgement.
 
#13
Dason, I mean both must be simultaneously be either above or below that margin. Your good example, not only crosses that margin, but also crosses zero ;) :D so it is not only practically nonsignificant, it is even statistically nonsignificant :)
 

Dason

Ambassador to the humans
#14
Yes but I was just trying to illustrate why it should matter that we take the confidence interval into account.
 
#15
@Joey

It was my very essential problem about 6 months ago, for which I searched A LOT to be able to find only a couple of resources. I agree that there were very few topics about it, but you will find that my comment is valid if you search more :)
Can you recall what those sources were?
 

noetsi

No cake for spunky
#16
If you want a cite for the difference between test of statistical signficance (including implictly CI) and substantive interpretation of effect size I can give this to you from text (not articles).
 
#17
His definition of practical significance is very conservative. I can't find anything, literally nothing about practical significance to be outside of CI.
 
#18
If you discover an effect size you believe are real, what does it matter what the CI is? Alternately if the CI (or p value etc) suggests that the result is real but it is tiny what is the impact practically of that? Statistics and substantive value are entirely different. The only thing statistics can tell you is if the results are likely to be random error not if they matter. That is outside the realm of statistics a point increasingly being emphasized in statistical texts.



I agree entirely. My point is that objective thresholds are not determined by statistics but by impact and professional judgement.
Part of my comment here is also to Joey :) Even if we determine the margin of clinical equivalence, we don't know about the variation in our sample. So a 103-unit difference is as vague as an average without standard deviation. Can somebody tell if a difference is significantly greater than zero or not, if he doesn't know the standard deviation?

Now, in this CI model, they have replaced that "zero" line with a "band" with a non-zero width. We should check if the CI has crossed that band or not. If it dooesn't cross the band (both bound of CI greater or lesser than the band, simultaneously), we can conclude that our average 103 unit difference is for sure greater than the band (so there is practical difference).

If the CI crosses the band, there are two form:

either it totally crosses the band, as Dason suggested (one CI bound below the band, the other beyond it). Here we can be sure that it is not at all practically significant. Or half of it crosses the band, and the other part remains within the band (the problem of Joey). Now, we can't conclude for sure that if it is practically significant or not. Although we can still become quite subjective and discuss the topic, but now this method of CI can't help us.

-------------

I agree that those objective thresholds are not quite statistically determined (although I have seen a couple of papers, in which the threshold was determined using stats). But once the threshold is found based on consensus or Delphi method (or even statistics if possible), we can use that for a semi-objective analysis.
 
#19
Joey I can't recall the resources, but after desperate search within this forum, I managed to find the starting point. So you can start here (also I recall I had started before this post which I am searching to find it)

Calling to Dason and other super robots :D , would be so grateful if you could help me with the problem stated in the abovementioned thread :) if possible though

edit: I think the real starting point for me is on the net (because I haven't found the first question), but later when I grabbed this CI method, I wanted to check if there is any way to use stat tests in such a situation.
 
Last edited:
#20
victorxstc, thanks for your input but I don't think your comments are accurate in regards to practical significance. Thanks for your comments again.