Converting Likert Scale to Binary and Correlating

#1
Hi there,

Apologies if a thread like this has been done previously, I couldn't find one.

I'm looking for a bit of feedback on a research project I'm designing. It is outside my usual scope, and I don't have current access to a statistician; thought I'd reach out here.

I apologize that I can't be completely forthcoming with details about the project; it involves PHI and a sensitive subject. I can answer questions to try and clear things up in terms of my description.

Purpose:

Overall, I want to be able to correlate a Likert scale to a Yes/No response. So, saying 'a score of ** on the Likert scale corresponds to No'. I want to do this to try and validate the scoring system I've devised, and to turn a subjective measure into an objective one.. if that makes sense.

Ask a number (having trouble coming up with appropriate n) of professionals in a field to rate 100 photos based on 8 factors, each on a scale from 1-10, with 1 being Very Poor and 10 being Very Good. There will be 100 total photos, of varying quality. These photos will not be able to be shown for publication purposes -- again due to PHI/sensitive nature. To solve this problem and communicate my findings effectively, I plan on inserting several (having trouble coming up with appropriate n) photos into the pool to serve as controls. These photos will be constructed in a way to be representative of 'Good' and 'Bad' photos - i.e. ones that would clearly translate to 'Yes' or 'No'. To perform intrarater reliability, each rater would evaluate each photo again on the Likert scale 1 week later.

Following this analysis, I would use the same 100 (+control) photo set to perform a separate analysis. Different professionals (but with the same training) (again, unsure of n) would evaluate the photos, but instead performing the Likert scale, they would simply answer the question 'Would this photo be acceptable for xyz?' and choose 'Yes' or 'No'. Again, they would perform the same set 1 week later.

Analysis:

Descriptive statistics and ICC for initial Likert scale. Unsure of the rest - how to correlate Likert to Yes/No is the trouble.

Questions:

1. How can I perform an appropriate power calculation to determine the number of photo raters needed for each analysis?
2. Is there a way to estimate the number of control photos required?
3. The main issue: how can I best correlate the Likert scale to the Yes/No?
4. Is this entire design plausible? I haven't been able to find any similar studies in the literature - is there a better way to go about this?

Apologies if this is too vague/poorly fleshed out. I'm wracking my brain with this project trying to get it off the ground. I know a pro statistician would be best but I just don't have the access right now. Any help is appreciated - even if it's partial or just general advice. I'm happy to help clarify things as much as possible.

Thanks.
 

j58

Active Member
#2
Regarding your question 4: Using one group of raters for the Likert ratings and a different group of raters for the binary choice is statistically very inefficient, and will force you to compute the "correlation" between the average Likert rating (across raters) with the proportion of "yeses" for the photos. Ordinarily, to assess the correlation between two measures, each rater would rate each object using both measures. The "correlation" can then be computed from the raw data without averaging, permitting control of between-rater variation. It would be interesting to know just how much statistical power you would lose by using separate groups of raters. My guess is that it would be monumental: I would not be surprised if you had to increase the number of raters by a factor of 10 to compensate (which really means a factor of 20, since you'd need two groups). So, if there is any way you can have one group of raters perform both ratings, you should.

Regarding your question 3: Regardless of how you resolve question 4, relating a Likert score to a binary choice is conceptually straightforward. Using an approprate type of regression, regress the binary choice (DV) against the Likert rating (IV). The estimated regression equation would then relate a Likert score to the probability of "yes" on the binary choice.
 
Last edited:
#3
Thank you! Very helpful.

I suspected that having different reviewer pools would cause some power loss. I'm not really sure how I got it in my head that I needed two reviewer pools for each analysis, thanks for steering me away from this.

I'm likely going to go with your suggestion -- do you have any recommendations about how to go about performing a power calculation to determine my target number of reviewers?

Thanks again!
 

j58

Active Member
#4
Power depends on the statistical analysis technique that you plan to use, so you need to decide that first. I would probably use mixed logistic regression, with random terms for picture and rater, and a fixed term for Likert scale rating. I just googled "power for mixed logistic regression" and the results included several hits that look promising.