# Performance evaluation

#### martinl

##### Member
Dear all,

I am doing a evaluation of performance for a group of clubs, the aim is to check which club has compliant % below the state average.
The table below shows an example. Each row represents 1 club. The last row represents the state, 68% is the state average. My question is: what statistical test should I use to test whether any difference from the state average is significant or not?

martinl

#### noetsi

##### Fortran must die
I think a good starting question is why do you need to do that. Test of significance get at whether the effect size you find is likely to exist in a population when you have a sample - that is it is the true effect size. It does not tell you that the effect size is substantively different than anything. This is a point that constantly gets confused, and why many are increasingly critical of p values and statistical test.

The difference is significant if, for substantive reasons, you think the difference is large enough to matter. No statistical test can tell you that.

#### Archidamus

##### Member
Let me see if I understand your question. You want to know if one of these 7 clubs has a compliance rate under the state average, but you don't know what the state average really is?

You could infer the state's average and variance based off these 7 clubs, then use that to build a confidence intervals. But that would lead to a very biased test, and probably get thrown out of this forum. I must not be understanding what you are trying to do here.

#### noetsi

##### Fortran must die
Archidamus I think the state average is known (it's 68 percent). The question asked is if any compliance is significantly different that this. While you could do tests like test of proportion to answer this, my argument is that this is really a substantive question that (as is often done, but wrongly IMHO) the author wants to address in terms of statistical significance.

#### martinl

##### Member
Archidamus I think the state average is known (it's 68 percent).
Yes, you are right, the state average is known, it is something that can be derived from the data.

The question asked is if any compliance is significantly different that this. While you could do tests like test of proportion to answer this, my argument is that this is really a substantive question that (as is often done, but wrongly IMHO) the author wants to address in terms of statistical significance.
Yes, that's my goal - to check if any compliance is significantly different from the average.
For example, 67.7 % is not significantly different from 68%, while 2% is. My question is: what test should I use?? Can I use Chi squared?

#### hlsmith

##### Less is more. Stay pure. Stay poor.
I would just create a caterpillar plot (proportion of compliance for each club with associated 95% (binomial) confidence intervals all plotted). I would use this to visualize differences. You could also plot the average on the graph as a reference line. however, I would not calculate the average as you did above, I would weight each average by its contribution toward the total n-value, so it would be a weighted average of averages for the plotted reference line.

As noted there is no reason to conduct a statistical test. You should trust our recommendations and quit trying to do a test, in the end it would make you look foolish.

Thanks.

#### noetsi

##### Fortran must die
I think you could do a test of proportions if you want a statistical test. But as I and hlsmith noted it is not really valid to use a statistical test for what you want. Statistical test make sense when you don't know the true effect size. Here you do know it.

#### ondansetron

##### TS Contributor
I’ll try rephrasing what the others have correctly said: statistical tests are for making inferences (drawing conclusions) about something unknown on the basis of a sample that is supposed to represent the unknown. The statistical test incorporates uncertainty from the sample to help you say something about the unknown.
In this problem, there is no unknown; you know the real values, if they aren’t the same, then they’re different.

#### noetsi

##### Fortran must die
"In this problem, there is no unknown; you know the real values, if they aren’t the same, then they’re different."

Amen. This should be emphasized in any statistics course when they get to p values. Its amazing how few get it.