rating the accuracy of predictions

H

HanPom

New Member
#1
#1
I have a set of predictions made by teachers for student exam results, and the outcomes for the students. I would like to use the accuracy of these predictions to weight/rank the predictions made for the new cohort of students. The number of students taking the different subjects also varies making the accuracy of the weighting variable. Can anyone suggest a robust methodology for doing this?

Thanks!
 
ondansetron

ondansetron

TS Contributor
#2
#2
How are the exam results given? Are they points out of 100 or letter grades? Does every student have only a single score?

are these exam results for a single test, a single course? What is the typical class size? What is meant by weight/rank predictions (simply whose prediction was the best by some metric)?

more information is useful.
 
hlsmith

hlsmith

Not a robit
#4
#4
More information is needed for sure, but you could look at plotted observed vs expected values to see where the teachers are performing better on the spectrum of scores. Then go back and tease out student characteristics to see if something is telling (subgroups). Though, you would want to make sure you are not discriminating against students based on apparent or latent characteristics. I would look to the literature on analytic studies addressing educational remediation.
 
H

HanPom

New Member
#5
#5
More information (!):
  • each subject measures what they think the students will get 5 times through the year which we convert into statistics for the subject cohort (e.g. 54% are predicted A).
  • The students then sit their exam and we record the percentages achieving each grade.
I would like to use the accuracy (or not) of the predictions of the different subject departments to weight the current predictions (e.g. department x is predicting 25% will get A, but last year they over-predicted by 13% at this point in the year, so I can have x confidence about their prediction). It is this last bit that I don't really know how to do - I did think about putting confidence limits on the predictions, but what statistical technique would one use to create these?

Many thanks for the responses already.
 
You must log in or register to reply here.