# Practical significance

#### _joey

##### New Member
The difference between two mean values is 103 units
p-value is 0.00001 at alpha=0.05
95% confidence interval is (50, 150)
difference of 100 units is of practical importance.
Would the mean difference be practically significant for the given difference between two mean values and confidence interval?

Thanks!

#### _joey

##### New Member
Here you are dealing with margin of equivalence. That margin of practical significance extends from -100 to +100, and your 95% CI lies somewhere within and somewhere outside it. You should state that "the results are not conclusive in terms of practical significance".
i don't know what 'margin of equivalence' is. The question clearly states "difference of 100 units is of practical importance". The difference of the means obtained is 103 units at 0.005 significance level.

#### victorxstc

##### Pirate
Here you are dealing with margin of equivalence. That margin of practical significance extends from -100 to +100, and your 95% CI lies somewhere within and somewhere outside it. You should state that "the results are not conclusive in terms of practical significance".

--------------

i don't know what 'margin of equivalence' is. The question clearly states "difference of 100 units is of practical importance". The difference of the means obtained is 103 units at 0.005 significance level.
It means that the difference less than 100 unit is not practically significant. Say, if two people have an 88unit difference, although they may be statistically significant, they are not practically significant as their difference is less than 100.

In such cases, some methods can be applied. The best maybe using the confidence intervals. Check the bounds of CI. Both should stand outside the -100 to 100-unit band of "practical signficance" to certainly conclude whether or not there was any "practical" difference. In your case, only one of them lies outside this range. So, you can't be sure if the difference is practically significance or not, and further studies with better powers are necessary.

#### _joey

##### New Member
It means that the difference less than 100 unit is not practically significant. Say, if two people have an 88unit difference, although they may be statistically significant, they are not practically significant as their difference is less than 100.
But the difference calculated is _not_ 88 units, it's 103 units.

In such cases, some methods can be applied. The best maybe using the confidence intervals. Check the bounds of CI. Both should stand outside the -100 to 100-unit band of "practical signficance" to certainly conclude whether or not there was any "practical" difference. In your case, only one of them lies outside this range. So, you can't be sure if the difference is practically significance or not, and further studies with better powers are necessary.
The lower bound of CI should be above 100 or below -100 units?

#### victorxstc

##### Pirate
even if your difference is 25500^1000 units, but the lower bound of CI stands "within" that mentioned -100 to 100-unit range, you cannot be sure about practical significance! I repeat that both bounds should stand outside this range. It means that if the upper bound is greater than +100, the lower bound must be as well OR if the upper bound stays below -100, the lower bound must be lesser than -100 in order to be sure that your difference is practically significant.

#### _joey

##### New Member
even if your difference is 25500^1000 units, but the lower bound of CI stands "within" that mentioned -100 to 100-unit range, you cannot be sure about practical significance! I repeat that both bounds should stand outside this range. It means that if the upper bound is greater than +100, the lower bound must be as well OR if the upper bound stays below -100, the lower bound must be lesser than -100 in order to be sure that your difference is practically significant.
I hear you. Some authority references would be nice to have to substantiate your comments because I haven't found anything on the web in regards to the CI bounds and practical significance.

#### noetsi

##### No cake for spunky
To me practical signficance would seem to indicate if the mean difference is substantively important, which is distinct from any statistical property such as a p value. That is a decision that involves judgement and the norms of the field not statistics. It is very much context driven.

#### _joey

##### New Member
To me practical signficance would seem to indicate if the mean difference is substantively important, which is distinct from any statistical property such as a p value. That is a decision that involves judgement and the norms of the field not statistics. It is very much context driven.
That's what I thought too. Since, 100 units is of practical significance then the difference of 103 units would be practically significant too. victorxstc, however, is talking about CI bounds. I think he is referring to size effect.

#### noetsi

##### No cake for spunky
I would argue, and the statisticians I have read, that elements such as CI bounds has nothing to do with substantive importance. Much of statistics gets at if the results you find could be tied to random error. That is entirely seperate from issue of whether the effect discovered (if real) matters.

#### Dason

So are you telling me that if the confidence interval had been (-1000, 1206) and the point estimate was 103 you would be completely fine saying that there is a practically significant difference?

#### victorxstc

##### Pirate
@Joey

It was my very essential problem about 6 months ago, for which I searched A LOT to be able to find only a couple of resources. I agree that there were very few topics about it, but you will find that my comment is valid if you search more

@noetsi

I agree that clinical significance is something subjective. But sometimes, they manage to assume some objective thresholds (based on some research) for defining clinical significance. This way, it can be semi-objective. the case of Joey seems something like that, where he can use numbers as well (again it is subjective, but better now).

#### noetsi

##### No cake for spunky
If you discover an effect size you believe are real, what does it matter what the CI is? Alternately if the CI (or p value etc) suggests that the result is real but it is tiny what is the impact practically of that? Statistics and substantive value are entirely different. The only thing statistics can tell you is if the results are likely to be random error not if they matter. That is outside the realm of statistics a point increasingly being emphasized in statistical texts.

I agree that clinical significance is something subjective. But sometimes, they manage to assume some objective thresholds (based on some research) for defining clinical significance. This way, it can be semi-objective. the case of Joey seems something like that, where he can use numbers as well (again it is subjective, but better now).
I agree entirely. My point is that objective thresholds are not determined by statistics but by impact and professional judgement.

#### victorxstc

##### Pirate
Dason, I mean both must be simultaneously be either above or below that margin. Your good example, not only crosses that margin, but also crosses zero so it is not only practically nonsignificant, it is even statistically nonsignificant

#### Dason

Yes but I was just trying to illustrate why it should matter that we take the confidence interval into account.

#### _joey

##### New Member
@Joey

It was my very essential problem about 6 months ago, for which I searched A LOT to be able to find only a couple of resources. I agree that there were very few topics about it, but you will find that my comment is valid if you search more
Can you recall what those sources were?

#### noetsi

##### No cake for spunky
If you want a cite for the difference between test of statistical signficance (including implictly CI) and substantive interpretation of effect size I can give this to you from text (not articles).

#### _joey

##### New Member
His definition of practical significance is very conservative. I can't find anything, literally nothing about practical significance to be outside of CI.

#### victorxstc

##### Pirate
If you discover an effect size you believe are real, what does it matter what the CI is? Alternately if the CI (or p value etc) suggests that the result is real but it is tiny what is the impact practically of that? Statistics and substantive value are entirely different. The only thing statistics can tell you is if the results are likely to be random error not if they matter. That is outside the realm of statistics a point increasingly being emphasized in statistical texts.

I agree entirely. My point is that objective thresholds are not determined by statistics but by impact and professional judgement.
Part of my comment here is also to Joey Even if we determine the margin of clinical equivalence, we don't know about the variation in our sample. So a 103-unit difference is as vague as an average without standard deviation. Can somebody tell if a difference is significantly greater than zero or not, if he doesn't know the standard deviation?

Now, in this CI model, they have replaced that "zero" line with a "band" with a non-zero width. We should check if the CI has crossed that band or not. If it dooesn't cross the band (both bound of CI greater or lesser than the band, simultaneously), we can conclude that our average 103 unit difference is for sure greater than the band (so there is practical difference).

If the CI crosses the band, there are two form:

either it totally crosses the band, as Dason suggested (one CI bound below the band, the other beyond it). Here we can be sure that it is not at all practically significant. Or half of it crosses the band, and the other part remains within the band (the problem of Joey). Now, we can't conclude for sure that if it is practically significant or not. Although we can still become quite subjective and discuss the topic, but now this method of CI can't help us.

-------------

I agree that those objective thresholds are not quite statistically determined (although I have seen a couple of papers, in which the threshold was determined using stats). But once the threshold is found based on consensus or Delphi method (or even statistics if possible), we can use that for a semi-objective analysis.

#### victorxstc

##### Pirate
Joey I can't recall the resources, but after desperate search within this forum, I managed to find the starting point. So you can start here (also I recall I had started before this post which I am searching to find it)

Calling to Dason and other super robots , would be so grateful if you could help me with the problem stated in the abovementioned thread if possible though

edit: I think the real starting point for me is on the net (because I haven't found the first question), but later when I grabbed this CI method, I wanted to check if there is any way to use stat tests in such a situation.

Last edited: