statistically analyse pass rate of test in different locations

[bobsmells]

I'm looking to analyse some data and as it's been a long time since my statistics days I'd like to know which area of statistics my problem lies.

Lets say I have a number of different driving schools, each doing the same test of their student drivers' ability to have a licence.

Now every month or so I might want to analyse if a particular driving school is "letting people through" easily meaning a significant percentage of people are passing, or conversely, is being too strict and a significant percentage of people are failing.

Now I've done some reading and lots of things came flooding back... and I think we're talking about standard deviation and statistical significance. Although what I've read about the latter tends to imply that we need two sets of data to imply some causal relationship, whereas I'm looking at each school against the mean of their pass rate for the month ...or potentially each school against itself over time or something...?

Can someone give me some pointers as to where to start?

 

[hlsmith]

How many schools? And you need to decide if you would or would not compare schools to each other or just self. What is your outcome, just pass failure or would you look at actual scores?

You need to really define the question and variables then the procedures will fall into place.

 

[bobsmells]

Excellent questions. There will be about 10 schools, and they might do anywhere up to ~100 tests per month, but probably not any more than that.

We're only talking pass and fail, no numerical scores.

Despite the fact that some schools will do more actual tests in a month, I'd say we definitely want to compare the percentage of passes in a particular school against other schools, because we're trying to analyse whether some centres are not stringent enough in their testing etc. So then on one hand we should probably take a single month's results and compare school vs other schools....but then as a separate test it would be good to analyse over time whether the same school is consistently leading in terms of pass percentage. So those are two different tests, I understand that. If you could give me some pointers on both I'd appreciate it.

 

[Karabiner]

In any distribution there will necessarily be cases which are
categorized as "extreme", but do nevertheless belong to the
distribution. Thinking variation is (or should be) the core of
statistical considerations. In other words, if you identify one or
two "extreme" driving schools, you will probably not be able to
distinguish with certainity if they belong to the population of
"normal" driving schools (and just are situated at the end of the distribution),
or if indeed they belong to a different population of e.g. "letting people
through"-driving schools. There are some outlier tests out there,
though these might lead to the wrong impression that by "objective"
result of a test a certain school can officially be labeled as e.g. "letting
people through".

For a start, personally I'd perform some exploratory data analysis,
for example boxplots of the pass rates of the driving schools,
repeated over time. If there are one or two schools which consistently
show distinctive rates, one could have a closer look for the reasons.

Just my 2pence

K.

 

[bobsmells]

Yep, I understand that we wouldn't want to highlight a particular outlier score based on one extreme score, or even a few, because as you say that's part of a normal distribution.

But if a particular school's scores over time were usually/consistently/more-than-a-few-times significantly different to others, i.e. beyond a certain threshold where it's statistically likely that there is some non-random reason for such consistently extreme scores...then that's what I'm trying to find. I can hack together my own layman's algorithm for determining causality, e.g. if a school has the highest pass rate by more than 5% to the 2nd place in 3 months or more in a calendar year then maybe that's worth calling out...but that's a complete guess and I'm sure there's a way to analyse it statistically to see if that's not really an issue or if it is.

I hope I'm explaining myself properly. If you'd like me to dummy up some monthly scores over 12 months or whatever to use as a sample, I could do so. Any ideas appreciated.

 

[Karabiner]

Why not explorative data analysis, as described above?
Also, there are outlier tests for small samples
(e.g. http://www.chem.uoa.gr/applets/AppletQtest/Appl_Qtest2.html ,
a search for outlier tests for small samples will certainly
find more approaches/tests).

With kind regards

K.

 

[Miner]

You might consider ANOM for proportions. This tests each level against the group mean.