Intra- and Interrater reliability of imaging findings

[Hugo]

Hi everyone,

I'm having some difficulty deciding how best to analyze the data from a project. I had 4 radiologist read CT scans looking for specific signs (mostly giving True/False answers) on two occasions separated by a washout period. The last two variables included a 'Questionable' category.

In calculating the intrarater reliability, I assume Kappa would suffice. However, with the interrater, I'm not sure. Someone suggested that Krippendorff's alpha could be used, and might be better than the multiple Kappa. I'm also not sure whether I should treat it as 8 observers, or rather 4 observers in which case I might have to combine the two readings (perhaps 2x0's = 0, 1x0 and 1x1 = 1, 2x1's =2) and then do a weighted analysis?

Thanks so much!

(I'm using SPSS 17)

 

[Hugo]

Okay, so I got STATA, which could do weighted Kappa nicely, and sorted out the intra- and inter-rater reliability (for the most part). I also used DAG_stat.

However, the 95% CI's from DAG_stat, STATA and Statistica (via a friend) dont match:

DAG_stat: Cohen’s Kappa = 0.68 (95% CI: 0.45 - 0.91)

STATA:
Kappa [95% Conf. Interval] Z Prob>Z
-------------------------------------------------
0.6813 0.3881 0.8482 4.73 0.0000

Statistica: Kappa=0.68 (0.43-0.89)

I have absolutely no clue why?? I gave it the same data (2=True):


...........|..........V3B2
.. V3B1 | ..........1 .........2 | .....Total
-----------+----------------------+----------
.........1 |........10..........5 | .......15
.........2 |......... 1........30 | ........31
-----------+----------------------+----------
....Total |........11........35 |........46