Kappa for nonsquare tables

[ondansetron]

I have two raters who could have assigned any of 16 ratings to a subject. There are some categories that were not used, so the resulting table is not square and SAS will not give me the Kappa (Cohen's) for it.

I'm having trouble figuring out how to do this quickly and efficiently to get the Kappa statistic with some zero rows or columns. SAS notes some option about weighting with ZEROES but it seems I would need to basically reformat the data into the summarized counts of each possible outcome.

Anyone able to offer advice to tackle the problem? If I do need to summarize the counts for each category, what is the quickest way to do this for a 16x16 table?

 

[hlsmith]

I had this exact same issue earlier this year, if I can remember which study it was for - I will look up what I did.

 

[ondansetron]

great, thanks! Variables are nominal ratings, by the way.

 

[hlsmith]

hmm, just too busy to look it up right now, this was my SAS code, not sure if it is applicable to you.

if this still doesn't work with uneven structure, I must have done it in R with another procedure:

proc freq data=epa2_rev;



table EPAn_S_1ID_sets * EPAn_SD_1ID_sets / agree (wt=fc) alpha=0.01;

exact agree;

run;

 

[ondansetron]

Yeah I didn't work, but STATA automatically handles it regardless of square or not, so I'm gonna squeak by with that for now. Thanks!

I had spent a while looking for it and couldn't come up with a way that worked for me.

 

[spunky]

Yeah I didn't work, but STATA automatically handles it regardless of square or not, so I'm gonna squeak by with that for now. Thanks!.

Do you know that STATA does that it lets you handle it? Is it still fitting everything as if it were a square matrix and applying some sort of 'continuity correction'? I remember some software just throw in a 0.5 if there are 0s anywhere.

 

[hlsmith]

I must have done it in R which does whatever stata is doing, will check tomorrow.

 

[ondansetron]

Do you know that STATA does that it lets you handle it? Is it still fitting everything as if it were a square matrix and applying some sort of 'continuity correction'? I remember some software just throw in a 0.5 if there are 0s anywhere.

Actually, I don't think I looked at how it's handled. That would've been smart. I'll try to read into it. Do you have any SAS recommendations?

 

[spunky]

Do you have any SAS recommendations?

I've never touched SAS in my life ¯\_(ツ)_/¯

 

[ondansetron]