Hi, my name is Dave. I am a recent graduate trying to pass a licensing-exam with the state of California. I think I passed the test (my score=73%), but they use a criterion-referencing method where results are compared to a criterion (their passing score= 75%) rather than to the mean. The idea is that level of difficulty of tests vary, and by comparing a score to the criterion, it does not matter what the group does. But I think the group performance should be consistent with the results, ie, in the 2nd to last iteration the test was deemed more difficult, the pass-point was lower at 118 of 175 correct and (66% passed). Last iteration was deemed easier, the pass-point was set higher at 132 of 175 and (48% passed). So the "harder" test had a higher pass rate and the "easier" test a lower pass rate. These are common sense results, but I think the criterion is wrong.

I don't know much about statistics, but I think if the test really does vary in difficulty, approximately the same % of test-takes should pass each time, if the pass-point is moved reliably. I don't think this is the case.

I plugged the students passing percentage of the last 20 iterations of the test into online calculators for standard deviation and z-score. Each of these 20 results represents a group with a population average of about 750 test-takers. The test-takers are all recent graduates of a Masters program, taking a test that is offered every six months. So this is not a random population.

Mean 56.55

standard deviation 6.56526

Variance (std dev) 43.10263

Population std dev 6.39902

Variance (pop std dev) 40.9475

Value 48

z-score -1.30230943

Do these look like reliable results? Mean 56, standard deviation 6.5 zscore -1.3?

These are the last 20 results in case I calculated incorrectly:

48,59,61,61,53,61,53,66,58,57,68,66,47,49,50,65,53,54,51,51

I don't know much about statistics, but I think if the test really does vary in difficulty, approximately the same % of test-takes should pass each time, if the pass-point is moved reliably. I don't think this is the case.

I plugged the students passing percentage of the last 20 iterations of the test into online calculators for standard deviation and z-score. Each of these 20 results represents a group with a population average of about 750 test-takers. The test-takers are all recent graduates of a Masters program, taking a test that is offered every six months. So this is not a random population.

Mean 56.55

standard deviation 6.56526

Variance (std dev) 43.10263

Population std dev 6.39902

Variance (pop std dev) 40.9475

Value 48

z-score -1.30230943

Do these look like reliable results? Mean 56, standard deviation 6.5 zscore -1.3?

These are the last 20 results in case I calculated incorrectly:

48,59,61,61,53,61,53,66,58,57,68,66,47,49,50,65,53,54,51,51

Last edited: