Can I compare regression models with same # predictors and same sample?

[CWRUgradstudent]

Hi there,

As a part of my dissertation I want to compare a focal, new, variable against other well-known predictors of my outcome, so I plan to run a hierarchical regression in SPSS with the focal variable entered on the last step. The problem is that my focal variable is highly correlated (r=.84 based on my previous data) with one of the well-known predictors of my outcome variable, yet conceptually my new variable is distinct and theoretically important. Because of the high correlation it seems unwise to include both variables in the same regression model, but still, I'd like to be able to compare the predictive power of the two variables.

Thus, I plan to run the regression with my focal variable first (excluding the well-known predictor), then run a separate regression on the same sample, with my focal variable extracted and replaced by that other well-known/highly correlated predictor, but I'm unsure how to statistically compare these two models (or, rather, the predictive power of the two highly correlated predictor variables, given that the models are otherwise the same)

My question is how might I statistically compare these two models? That is, how can I determine whether or not there is a significant difference in either the overall effect size between the two models, or in predictor strength? Would it make sense to obtain the partial correlation coefficients and use those in a Steiger's Z test? Is there a better way?

Thanks in advance for any help!

 

[rogojel]

hi,
I did not try this, but seems to be a good idea:

do a regression on the two variables that are correlated , build a model of the second one in terms of the first one and use one of the variables an the residuals of the predicted one in the model.

 

[noetsi]

If you have two regression models you can always use BIC or AIC to show which is better. I am not sure whether that is what you are trying to do.

 

[CWRUgradstudent]

Thanks for the reply, Noetsi!

It's my understanding that BIC and AIC are good for taking into account the complexity of the model (i.g., by considering N and df). In my case, the complexity is exactly the same, as it's just one variable being swapped out for another and thus I believe I can just look at the difference in the R^2 for the same purpose here. But is it possible that AIC/BIC will allow me to make a statement about statistical significance that I cannot make based on the change in R^2?

 

[CWRUgradstudent]

Thanks for the reply, rogojel!

Is there a name for this technique that I might look up in order to find more information?

 

[Dason]

hi,
I did not try this, but seems to be a good idea:

do a regression on the two variables that are correlated , build a model of the second one in terms of the first one and use one of the variables an the residuals of the predicted one in the model.

If you include the original variable and the residuals of the new variable then ultimately you haven't actually changed the model - the predictions will still be exactly the same. The test for the slope on your new variable will be the same as well actually. By using the residuals you remove the correlation between the variables that end up in the model but it doesn't fundamentally change the model.

 

[rogojel]

hi,
I have two variables that are correlated. if I include one fully and sort of the indepent part of the second, that should be a better model then just having one variable, right?
Now that I think of, it looks like first calculating the principal components and doing the regression with them, only in two dimensions.

regards
rogojel

 

[noetsi]

Thanks for the reply, Noetsi!

It's my understanding that BIC and AIC are good for taking into account the complexity of the model (i.g., by considering N and df). In my case, the complexity is exactly the same, as it's just one variable being swapped out for another and thus I believe I can just look at the difference in the R^2 for the same purpose here. But is it possible that AIC/BIC will allow me to make a statement about statistical significance that I cannot make based on the change in R^2?

My understanding is that AIC/BIC shows which is a better model generally (and that it is preferred to R square or adjusted r square). I have not heard that it simply addresses the relative complexity of the model. Note that if you are comparing the two models and you use different observations to calculate the two models R square will not be a valid way to assess the differences.