Error bars don't overlap but aren't significantly diff. - interpretations/reasons?

[Emilemmo]

Hello all, I'm struggling to understand/interpret my graph and kruskal wallis test.

Basically, I've created a bar graph which looks pretty good. There are 5 bars, each with error bars, and none of the error bars overlap. All of my 5 groups look significantly different. HOWEVER, when I do a Kruskal Wallis test on the 5 groups, it says that only 2 of 5 are significantly different.

Visually, the error bars aren't even close to each other, so I can't understand why they are not significantly different. Not sure how to interpret this. Does it have something to do with an overall/familywise error that is applied? I have small sample sizes, could this be it? Why does my graph look significant when my data isn't?

Any advice about what is going on would be greatly appreciated.

Many thanks in advance

 

[Karabiner]

Re: Error bars don't overlap but aren't significantly diff. - interpretations/reasons

Basically, I've created a bar graph which looks pretty good. There are 5 bars, each with error bars

When creating error bars you have to take into account that you perform several analyses/comparisons at the same time.

HOWEVER, when I do a Kruskal Wallis test on the 5 groups, it says that only 2 of 5 are significantly different.

Kruskal-Wallis cannot "tell" you that. Kruskal-Wallis only "tells" you, globally, whether the 5 groups may be the same or not. But not which of the groups differ from each other. Do you refer to post-hoc tests?

With kind regards

K.

 

[hlsmith]

Re: Error bars don't overlap but aren't significantly diff. - interpretations/reasons

Also if you correct your post hoc tests for false discovery then did you modify your confidence bands on your plots.

 

[Emilemmo]

Re: Error bars don't overlap but aren't significantly diff. - interpretations/reasons

Thank you for your replies

Yes - I should have been clearer: I used pairwise comparisons on SPSS after the Kruskal Wallis to compare the different groups. SPSS gives you a lovely output with the significance level for each pair. Some of the pairwise comparisons were not significant, which is why I am confused.

Also, I should have mentioned that my error bars are 2 standard errors, which was suggested as the best by someone who is a lot more knowledgeable than me!

 

[Emilemmo]

Re: Error bars don't overlap but aren't significantly diff. - interpretations/reasons

Also if you correct your post hoc tests for false discovery then did you modify your confidence bands on your plots.

I'm really sorry but I don't quite understand what you mean. The post hoc tests are done automatically in SPSS when you select 'pairwise comparisons' and I usually just report those results. I didn't know that you might have to correct the post hoc tests for false discovery, I wonder if SPSS does this (with the familywise error), or is it something that I need to do? Many thanks.

 

[hlsmith]

Re: Error bars don't overlap but aren't significantly diff. - interpretations/reasons

Just because the KW is significant doesn't mean all pairwise comparisons will be significant. When you make post hock pairwise comparisons you need to adjust the alpha level used to prevent false discovery (multiplicity) . What are your plots presenting? If you are running wilcoxon paireise comparisons there is a reason your not using the means or ttests so you shouldn't think graph means and tested medians should be equivalent and say if you did do this with attests and plotted means you would correct the alpha in attests and plots making the confidence intervals even wider.

**** autocorrect!