Critiquing a Network Meta-analysis: Interpreting inconsistency

Hi all. I'm a non-statistician trying to critique a network meta-analysis on a group of migraine medications called triptans (PDF link). The statistics is a bit over my head so I'm hoping you kind folk can help.
Not sure what happened with my last post but I'll try again..

I'm struggling with how they're assessing inconsistency: "We also plotted the posterior mean deviance of the individual data points in the inconsistency model against their posterior mean deviance in the consistency model to identify any loops where inconsistency is present."

I'm not entirely sure what this means. What is a posterior mean deviance? They report inconsistency for 2 of their outcomes. A plot for one of their outcomes is here:

Correct me if I'm wrong. Does this mean that the circled outliers indicate that the direct and indirect estimates for these data points do not agree and therefore there is some invalidity in the pooled estimate? What would it mean if the outliers fell above the diagonal line?

I've found some references on this topic but they're a bit technical that I'm having a tough time understanding.

Appreciate any help.



Ambassador to the humans
Your original post was automatically placed in the moderation queue waiting approval by a moderator. Posts by new users that contain attachments/images sometimes get placed in that queue and it can take a day for a mod to notice. I deleted the original since this one didn't seem to get caught by the filter.


Omega Contributor
I am not overly familiar with network meta-analyses, since it is a newer analytic. Given I am fairly ignorant take my interpretation to the above content with extreme reservations. In addition they appear to use a Bayesian approach.

Well per the article reference #20,

"Inconsistency can be thought of as a conflict between “direct” evidence on a comparison between treatments B and C and “indirect” evidence gained from AC and AB trials. Like heterogeneity, inconsistency is caused by effect modifiers and specifically by an imbalance in the distribution of effect modifiers in the direct and indirect evidence. Defining inconsistency as a property of loops of evidence, the relation between inconsistency and heterogeneity and the difficulties created by multiarm trials are described. We set out an approach to assessing consistency in 3-treatment triangular networks and in larger circuit structures, its extension to certain special structures in which independent tests for inconsistencies can be created, and describe methods suitable for more complex networks. "

So you are looking for data to fall on or near the 45 degree line. So think of it like a correlation of two variables, but this is two variable defining the same relationship, so you want a correlated relationship. The circled values appear to have a near 1 value per inconsistent model and higher values for consistent model. So if the later is direct evidence, I would assume the inconsistency model must have other factors deluding the effect.

I would imagine it is typical for values to stochastically land on either side of the line, but stay close unless the inconsistency based estimates have additional noise or neglecting to control for variability.

Interesting stuff.

P.S., I would imagine you should be able to look up these outlier (studies) and see if they appear different in some regard via design.