I have two observers who watched a series of videos looking for gestures. They recorded the onset-time of any gesture they observed. An agreement was scored when the two observers recorded a gesture onset within 1-second of the other observer. A disagreement was scored when the one observer recorded a gesture onset, but the other did not.
The agreement matrix looks like this:
******| Present | Absent |
----------------------------
Present | 290 | 373 |
-----------------------------
Absent | 67 | 0 |
----------------------------
Clearly the observers have differing opinions about what constitutes a gesture!
Percentage agreement is calculated easily enough: 290/(290 + 373 + 67) = 39% (Miserable agreement, yes, but that's the point of this portion of the paper.)
But, I'm trying to include kappa statistics in addition to percentage agreements in this paper. This has worked fine for other tables, and I'm learning a good bit about the statistic, but this table has that structural zero in the bottom right corner and it's giving me grief. It is logically impossible for the value to be anything other than a zero, because a record only exists in the cases where one observer or the other (or both) observed a gesture. Am I right to assume that the traditional kappa statistic would be invalid? It doesn't seem right to calculate an expected value for a cell that logically must be zero.
Granted, I could consider every single second in the video record where neither observer recorded as gesture as an agreement on "no gesture present," and enter this data into the lower right cell. However, this would artificially inflate agreement, though, as gestures were relatively rare. [Additionally, it's probably not a sound assumption, as some gestures take more than a single second to execute, so observations over subsequent seconds are not particularly independent of one another.]
Is there a way to calculate a kappa (or kappa-like) statistic for a 2x2 matrix with a single structural zero?
The agreement matrix looks like this:
******| Present | Absent |
----------------------------
Present | 290 | 373 |
-----------------------------
Absent | 67 | 0 |
----------------------------
Clearly the observers have differing opinions about what constitutes a gesture!
Percentage agreement is calculated easily enough: 290/(290 + 373 + 67) = 39% (Miserable agreement, yes, but that's the point of this portion of the paper.)
But, I'm trying to include kappa statistics in addition to percentage agreements in this paper. This has worked fine for other tables, and I'm learning a good bit about the statistic, but this table has that structural zero in the bottom right corner and it's giving me grief. It is logically impossible for the value to be anything other than a zero, because a record only exists in the cases where one observer or the other (or both) observed a gesture. Am I right to assume that the traditional kappa statistic would be invalid? It doesn't seem right to calculate an expected value for a cell that logically must be zero.
Granted, I could consider every single second in the video record where neither observer recorded as gesture as an agreement on "no gesture present," and enter this data into the lower right cell. However, this would artificially inflate agreement, though, as gestures were relatively rare. [Additionally, it's probably not a sound assumption, as some gestures take more than a single second to execute, so observations over subsequent seconds are not particularly independent of one another.]
Is there a way to calculate a kappa (or kappa-like) statistic for a 2x2 matrix with a single structural zero?
Last edited:
