Which Test of Significane to use when comapring two interfaces using likert statements?

[stef0s]

Hello guys,

i googled so much but i can't find an answer to my question. I hope you can help me out. I have the following problem:

In a user study the same participants had to use two different user interfaces for entering their preferences. Both interfaces were designed from scratch. There was no baseline or similar. After the participants used each of the UIs they had to rate some 5-point-likert statements (e.g., This interface is attractive, or This interface seems more suitable), where the worst possible feedback was "strongly disagree" and the best possible feedback was "strongly agree".

Now i want to compare the statements (This interface is attractive) for both interfaces and see if there is a statistically significant difference.

Which test can i use? Can i use the Mann-Whitney-Test to see if there is a difference? Or the fisher's exact test?

Thank you so much!

 

[GretaGarbo]

Can i use the Mann-Whitney-Test to see if there is a difference? Or the fisher's exact test?

Yes, why not.

I would use a t-test.

 

[stef0s]

Yes, why not.

I would use a t-test.

Hey Greta,

thanks for your fast response. This was my first thought too, to use a t-test. However, the majority argued that Likert scale data is ordinal and that you cant use a t-test on ordinal data. And that's where I got irritated. It seems like it's not clear on what to use.

 

[GretaGarbo]

However, the majority argued that Likert scale data is ordinal and that you cant use a t-test on ordinal data.

Well, people have different attitudes towards that. (On this site too.)

If the Lickert item is coded: 1, 2. 3, 4, 5 I think that it is OK to estimate means and do t-test. Others do not think so.

But if the item had been coded 1, 2, 3, 100, 1000, then I would not think that it would be OK with means and t-tests. (But the scale would still be ordinal. Nothing had been faked in anyway.)

I have a pragmatic attitude towards this.

Many universities accept students based on sums of scales that are of this ordinal type. If they can, then I can.

- - -

But if you run both a t-test and a Mann-Whitney, they will likely give the same result. But some people believe that Mann-Whitney tests if the medians are equal. It does not! But if you estimate medians on Lickert items you will get the most boring result. It will be 3, 3, 3,... and so on.

Means on the other hand, will vary. Not so boring, and more informative.

 

[stef0s]

Well, people have different attitudes towards that. (On this site too.)

If the Lickert item is coded: 1, 2. 3, 4, 5 I think that it is OK to estimate means and do t-test. Others do not think so.

But if the item had been coded 1, 2, 3, 100, 1000, then I would not think that it would be OK with means and t-tests. (But the scale would still be ordinal. Nothing had been faked in anyway.)

I have a pragmatic attitude towards this.

Many universities accept students based on sums of scales that are of this ordinal type. If they can, then I can.

- - -

But if you run both a t-test and a Mann-Whitney, they will likely give the same result. But some people believe that Mann-Whitney tests if the medians are equal. It does not! But if you estimate medians on Lickert items you will get the most boring result. It will be 3, 3, 3,... and so on.

Means on the other hand, will vary. Not so boring, and more informative.

Hey Greta,

I would like to ask you one more thing. Am I correct assuming that I need to use an dependent t-test, because the same participants used both interfaces? Or should i use the two sample t-test assuming unequal variances?

Thank you so much!

 

[GretaGarbo]

Am I correct assuming that I need to use an dependent t-test,

Well, yes. plot them (in an x,y plot) and if the correlation is significant, then do a pairwise t-test.

Normally a two-sample-t-test is called when one group of persons have evaluated one product and an other group have evaluated an other group.

A Welsh t-test is when there are different variances.

 

[obh]

Well, people have different attitudes towards that. (On this site too.)

If the Lickert item is coded: 1, 2. 3, 4, 5 I think that it is OK to estimate means and do t-test. Others do not think so.

But if the item had been coded 1, 2, 3, 100, 1000, then I would not think that it would be OK with means and t-tests. (But the scale would still be ordinal. Nothing had been faked in anyway.)

I have a pragmatic attitude towards this.

Many universities accept students based on sums of scales that are of this ordinal type. If they can, then I can.

- - -

But if you run both a t-test and a Mann-Whitney, they will likely give the same result. But some people believe that Mann-Whitney tests if the medians are equal. It does not! But if you estimate medians on Lickert items you will get the most boring result. It will be 3, 3, 3,... and so on.

Means on the other hand, will vary. Not so boring, and more informative.

Hi Greta

I like the pragmatic attitude probably t won't be too bad.

One of the t -test assumptions is the normality, but reasonable symmetrical data is considered to be good enough.
likert with only 5 values probably won't be symmetrical ...(for example if the center is around 4 you can go only 1 value to the right (5) and 3 to the left (1,2,3)

So maybe Mann-Whitney is a little better? (we agree that it doesn't compare the medians but the entire distribution)

 

[obh]

Well, yes. plot them (in an x,y plot) and if the correlation is significant, then do a pairwise t-test.

Normally a two-sample-t-test is called when one group of persons have evaluated one product and an other group have evaluated an other group.

A Welsh t-test is when there are different variances.

Another question Greta, , isn't it sufficient that the same people did both tests to do pair t-test?
Do we really need to do the correlation check?
I believe it will reduce the variance of the sample.
Actually, what can we lose if we do the pair-t test?