I am preparing an experiment in which I would like to test a heuristic. This heuristic predicts how well certain facts are known. The values returned by the heuristic are between 0 and 1.

I was thinking of testing this heuristic by asking people if they know a particular fact and compare the percentage of people that know it with the output of the heuristic. Participants can answer True, False, Don't Know.

When my heuristic predicts that a fact is more likely to be known (value closer to 1) higher percentage of people will say that the fact is True.

Example to clarify this:

F1: Isaac Newton was a physicist.

F2: Isaac Newton was a warden of the Royal Mint

Heuristic(F1) = 0.9

Heuristic(F2) = 0.1

Presenting the above facts to 10 people will yield the following results:

_________________________

|Fact | True | False | Don't Know|

|-----------------------------------|

| F1 | 8 | 0 | 2 |

| F2 | 1 | 2 | 7 |

-----------------------------------

Looking only at the True responses indicates that 80% people know F1 and 10% people know F2.

To measure how 'good' my heuristic is, I will look at Spearman correlation between the scores of the heuristic and the percentages of people who knew the fact.

I would like to ask if this seems as plausible way of evaluating it and if I didn't miss any obvious errors from statistical point of view. I looked for similar threads in this forum but I didn't have much luck at that.

Thanks,

Roman Kutlak