rating the accuracy of predictions

[HanPom]

I have a set of predictions made by teachers for student exam results, and the outcomes for the students. I would like to use the accuracy of these predictions to weight/rank the predictions made for the new cohort of students. The number of students taking the different subjects also varies making the accuracy of the weighting variable. Can anyone suggest a robust methodology for doing this?

Thanks!

 

[ondansetron]

How are the exam results given? Are they points out of 100 or letter grades? Does every student have only a single score?

are these exam results for a single test, a single course? What is the typical class size? What is meant by weight/rank predictions (simply whose prediction was the best by some metric)?

more information is useful.

 

[Archidamus]

You want to use peoples guesses to build a model? Why not just use the best guesser for all future guesses?

 

[hlsmith]

More information is needed for sure, but you could look at plotted observed vs expected values to see where the teachers are performing better on the spectrum of scores. Then go back and tease out student characteristics to see if something is telling (subgroups). Though, you would want to make sure you are not discriminating against students based on apparent or latent characteristics. I would look to the literature on analytic studies addressing educational remediation.

 

[HanPom]

More information (!):

• each subject measures what they think the students will get 5 times through the year which we convert into statistics for the subject cohort (e.g. 54% are predicted A).
• The students then sit their exam and we record the percentages achieving each grade.

I would like to use the accuracy (or not) of the predictions of the different subject departments to weight the current predictions (e.g. department x is predicting 25% will get A, but last year they over-predicted by 13% at this point in the year, so I can have x confidence about their prediction). It is this last bit that I don't really know how to do - I did think about putting confidence limits on the predictions, but what statistical technique would one use to create these?

Many thanks for the responses already.