Leniency of assessors

[fraatsr]

Hi,

I would like to research whether the assessors in our company are equally lenient/stringent in their judgement. After an assessment, they can give a candidate three different verdicts, positive, negative or conditionally positive. For the research, I made a frequency table with the number of verdicts per consultant:



Which test should I use to analyse a) if there is an overall significant effect of the consultants (they differ in leniency)? and b) which consultant is significantly more lenient/stringent compared to the average?

I was thinking about a chi square, but I'm not sure.

Thanks,
Rob

 

[fed2]

you need the ratings per consultant I would think.

 

[Karabiner]

Which test should I use to analyse a) if there is an overall significant effect of the consultants (they differ in leniency)? and b) which consultant is significantly more lenient/stringent compared to the average?

Well, basically you have an ordinal scaled dependent measure (Pos, cPos, Neg). If the observations were strictly
independent (i.e. no candidate was assessed by more than 1 assessor), then you could perform a Kruskal-Wallis
H-test as global test, and Mann-Whitney U-tests for pairwise comparisons. Depending on the software used, you
would probably have do construct a different table to perform the analysis.

With kind regards

Karabiner

 

[fraatsr]

Thanks for the replies, the observations are indeed independent.

Why is it an ordinal scale? There is a true zero, e.g. if the consultant never gives a positive advice, and distances between numbers are equal.

If you look at this example:
https://www.spss-tutorials.com/chi-square-independence-test/

If I would rearrange the data so the first column would be the candidate, the second column the consultant and the third column the advice, wouldn't I be able to answer my first question?

 

[Karabiner]

For example, assessor AAA rated 122 cases on the three point scale positive/conditional positive/negative.
I cannot see any true zero in this rating scale, and I assume that the three levels can be ordered with respect
to "positivity/liniency" (or "negativity/stingency").

If you look at this example:
https://www.spss-tutorials.com/chi-square-independence-test/
If I would rearrange the data so the first column would be the candidate, the second column the consultant and the third column the advice, wouldn't I be able to answer my first question?

So that you will have 430 lines (or so), each with these three entries?

How to proceed probaly depends on whether the observations were dependent or
were strictly independent (i.e. no candidate was assessed by more than 1 assessor).

With kind regards

Karabiner

 

[fraatsr]

of course, you're correct about the measurement level. I was thinking about the positive advice as being a leniency scale from 0-100%, but the advices are not mutually exclusive.

So that you will have 430 lines (or so), each with these three entries?
Correct, that's actually how the file originally looked, but I used a pivot table in excel to rearrange the data.

How to proceed probaly depends on whether the observations were dependent or
were strictly independent (i.e. no candidate was assessed by more than 1 assessor).
The advices are strictly independent, each candidate has one assessor.