# Simple question. Type of data?

#### Burnsie_UK

##### New Member
So, there is a test that you can score 1, 2 or 3 one.

Each number is a fault category.

1=fail
2=pass with errors
3=pass no error

Therefore 3 is better than one. However, I have had to define such.

I still believe that this is ordinal data.

Am I being stupid here?

#### Miner

##### TS Contributor
No. Given your definition, I would agree that it is ordinal. From your phrasing, does someone disagree with you? What is their reasoning?

#### Dason

Yeah... entirely ordinal. Just because somebody attaches numbers to something doesn't make it continuous.

#### Burnsie_UK

##### New Member
Thanks for replies both.

So, I am trying to compare the scores given by two groups.

One group is an expert group, and the other is a novice group.

I am trying to see if the scores both groups give relate to each other.

The scores are going to be ordinal, in that 1= fail, 2=pass with error, 3=perfect.

These individuals scores make up a whole score for a whole test for one person.

Looking at some papers, some have used ICC, others have used kappa.

I have found a paper that uses an unweighted kappa for ordinal data (and a stats paper saying weighted kappa is for ordinal).

They used ICC for inter rater reliability who the whole composite score (7-21), and unweighted kappa for the individual scores (1-3s).

Confused.com

#### Miner

##### TS Contributor
• Kappa is treating ordinal data as if it were nominal data. This results in the loss of information.
• Weighted kappa takes the ordinal nature of the data into consideration, but results could be influenced by the weighting used
• Kendall's is often used with ordinal data
• ICC is used for continuous data or when there are enough ordinal levels to consider the data to be continuous (3 would not meet this criteria, but 7-21 would). There is risk involved in making this assumption.

#### Burnsie_UK

##### New Member
• Kappa is treating ordinal data as if it were nominal data. This results in the loss of information.
• Weighted kappa takes the ordinal nature of the data into consideration, but results could be influenced by the weighting used
• Kendall's is often used with ordinal data
• ICC is used for continuous data or when there are enough ordinal levels to consider the data to be continuous (3 would not meet this criteria, but 7-21 would). There is risk involved in making this assumption.
Thanks

Kendall's Tau??

#### Burnsie_UK

##### New Member
Great.

So comparing rater groups for the individual screens (1-3 scoring system) using Kendall's

And comparing the composite scores between groups, ICC?

I feel I should also look at intra rater scores, but this would involve a further testing protocol as currently I'm trying to do "live" testing in the real world, and thus test-retest is pretty impossible!.... But if I was to do so....?

#### noetsi

##### Fortran must die
Why are you doing test/restest. ANOVA will test if there are changes from one group to another group in time, but I don't think that is what you are doing. I am not sure what you are doing

#### Burnsie_UK

##### New Member
Well, thinking out loud here, if rater A scores participant A a score of 2 on Monday

We then compare all raters (in two groups) from Monday on all participants. If there is agreement between the groups, then great. But surely with only 3 possible outcomes, the chance of accidental agreement is high.

Thus, test then all again on Wednesday and Friday and see if rater A gives the same score for the same screw as they did the other days?