Comparing across two models


Fortran must die
This came from a logistic regression

"We tested whether the effect of a factor was the same, stronger or weaker after 2000 by comparing the log odd coefficient of a factor in the 97-98 cohort with that in the 2002-2003 cohort. For example the effect of unemployment rate was .2 in the 2002-2003 cohort versus .15 in the 97-98 one"

I thought it was considered invalid to compare effects across samples, because the variation of the samples was likely different. Or is it just not allowed to compare significance test this way.


Not a robit
Can you provide a source for this?

It is definitely wrong to compare coefficients from two different models (different predictors). Yes samples would be different given maturation, etc. Are they the exact same people but the future versions of themselves?


Fortran must die
It is an internal state document so I don't think you could access it. It was published by the Division of Vocational Rehabilitation in Washington State in February 2007. The title is Examining Washington State's Vocational Rehabilitation Rates: Why the Decline.

Many of the people would not be in both samples - they looked at populations several years apart.
It is possible they ignored this issue because they felt they had the true population although they speak of a sample. It is also possible they had the same predictors in both models. What varies is that they had two samples at different points in time.

If they had the same predictors, but two different samples can you compare effect sizes the way they do?

It was prepared by two PHD so I am reluctant to say they are wrong (thus my question here) despite being taught this is invalid. I have not been able to find in the document I have access to (a portion of the total report) their methods - or at least I think I have (what I found is very limited).