inter-rater reliability

Hi, I have done a test where 2 different raters had to rate 1000 texts independently according to the following options: - Happy - Sad - Angry - Confused - Could not tell Now, I am trying to determine their agreement (i.e reliability). However, I am facing a dilemma -specially when the do...

Hi, For my research, I need to calculate the inter rater reliability. There are 4 observers, who each observe the same objects and score it for a total of 7 variables. The variables are of ordinal level, but not each variable has the same amount of outcome variables. Which inter rater...

Hi everyone, I am trying to figure out what would be the best statistic to use to quantify the amount of agreement that exists between subjects who were asked to press a button whenever they felt a certain emotion while listening to a short (2min) piece of music. Plotted as time series, the...

Hi - I'm looking for a measure for the level of agreement between a number of raters who score performance of something on 1-4 scoring range. So, for example, we have 10 raters, who have each independently assessed something: 1. 4 2. 4 3. 3 4. 3 5. 2 6. 3 7. 4 8...

Hi everyone! I need help with a research assignment. I'm new to IBM SPSS statistics, and actually statistics in general, so i'm pretty overwhelmed. My coworkers and I created a new observation scale to improve the concise transfer of information between nurses and other psychiatric staff...

I've done a study for my PhD comparing different methods/conditions that might affect inter-rater agreement when rating the creativity of other people's drawings. I had 600+ drawings that needed to be rated, so it wasn't feasible to have the same few people rate all the drawings. So 24 raters...

Hello, I'm tasked with analyzing how reliable raters are in rating a single piece of work along multiple dimensions. I'm trying to answer questions like: 1) Which rubric dimensions (some or all) have unacceptable variance across the raters? How do I define the cutoff for unacceptable...

Inter-rater Reliability measure for ordinal data with multiple raters I am wanting to obtain a measure of IRR for a diagnostics tool on the NHS. 6 professionals assessed an individual on multiple health domains such as Behaviour Cognition Psychology Communication Mobility Each...

I am trying to calculate inter-rater reliability scores for 10 survey questions-- most of which are binary (yes/no). The agreemeant level bettwen the two raters is 70-90% on nearly every question, however the Kappa score is often very poor (0.2- 0.4). Can this be right? Secondly, can you...