Cox/ Binomial Regression: How to reconcile Non-significant model with significant IVs

[illmatic_statistic]

Hello

I'm been performing some survival and regression analyses corresponding to an event of interest being death or a certain surgical procedure, namely a shunt revision for pediatric neuro-surgical patients. I know how to properly execute the procedures w/i SPSS and am familiar w/ the contextual definitions of most of the terms relevant to these tests but have across a few puzzling scenarios.

Scenario A:

The chi square statistic of the -2 log likelihood (or the omnibus test of the chi square in binomial) is not significant, but one of the values of the Wald statistic corresponding to some of the coefficients of the IVs within the model are significant, in addition to corresponding Exp (B) values greater than 1 w/ a CI that does not include 1 (which is sensible since I expected these particular IVs to increase the odds of the event of interest). When I remove the non-significant IVs co-variates and run the test again with the significant IV as the only one within the model, the overall model chi square statistic (Omnibus or -2log) is again insignificant but now the Wald statistic corresponding to the coefficient of the only formerly significant IV loses significance.

i. Questions: I understand that the initial omnibus or -2log likelihood test that takes place at the beginning of a Cox or Binomial Regression analysis assesses the predictive value of the overall model. If this value is not significant, is the model to be discarded? Essentially are these test stopping points that signal to the researcher that all of the information provided subsequently w/i the analysis is irrelevant? Can/should the Exp B values pertaining to significant Wald statistics be reported although the overall model is not significant? In the literature corresponding to my field I usually only see the odds ratios of individual IVs reported, but the overall significance of the model is not referred to explicitly, or perhaps assumed to be significant.

Scenario B:

Binomial logistic models containing one IV each provide insignificant chi square statistics in the Omnibus test of model coefficients or significant but w/ correspondingly very low Nagelkerke R Square values (<0.10). Then when I lump many of the numerical and categorical IVs together as co-factors for a single binomial regression analysis, the omnibus test of the overall model returns a highly significant chi squared statistic, the Negelkerke r squared value is much higher (0.3-0.5), overall predicated classification is high (60-80%), but only one or two of the Wald statistics corresponding to the coefficients of the IVs are significant, with some of those that were significant in the univariate models losing significance completely.

Questions: How do I interpret these findings?

Thanks very much in advance to anyone who has the patience to read this long post.

 

[hlsmith]

Re: Cox/ Binomial Regression: How to reconcile Non-significant model with significant

You have a long post, so at this time I will opt to pick and chose some parts to answer.


I believe the -2 log likelihood is used to compare nested models. In your case, you are probably looking at the saturated model (all covariates) and the empty model (no covariates), so an insignificant value would mean no significant prediction above the model with not predictors. Side point, seems like you may have too much extra stuff still in your model, given scenario B description. What we select as significant is arbitrary, but you should have some cut-off or reason to keep variables in the model. I would trim some of the fat and then revisit your likelihood question or use it to help you develop the model.000000

Scenario B: same as above, you need to get down to a parsimonious model, if you are keep non-significant variables in, you should have a reason.


-Stillmatic

 

[illmatic_statistic]

Re: Cox/ Binomial Regression: How to reconcile Non-significant model with significant

Thanks for the reply hlsmith.

I understand what you're saying when you recommend the parsimonious model. I would prefer a parsimonious model with fewer irrelevant covariates, but w/ regard to Scenario A, it's the non-parsimonious model that provides significant values for the wald statistic of the individual IVs (although the overall model is still only marginally or often non-significant). The parsimonious one, w/ regard to Scenario A, provides non-significant Walds, so it's essentially worthless as I currently understand the binomial regression.

W/ regard to Scenario B, I'm having the same problem: the non-parsimonious model is the one that can actually provide better predictions, but out of all the co-variates only a small fraction have significance w/i the model. When I start to parse down the co-variates to make the model more parsimonious, the model's overall predictive capacity (judged in terms of the Nagelkerke, classification percentages, and Omnibus test) all go down. So I'm wondering how to reconcile these events.

If you're saying that it's up to me determine the cutoff value for the significance of each individual IV coefficient, then that point would allow me to approach this whole problem differently.