SEM: poor fit, large sample, a lot of DF with significant coefficients.

#1
I am running an analysis of the influence of gender on a number of performance outcomes. My model is also moderated by the gender of the evaluator.
The model has really poor fit
(Df=241, N=851
I am looking at RMSEA (default model .309) and Hoelter (.05, Default model =12; .01 Default model = 13)
which I suspect can be explained by the overall little influence of gender on performance outcomes and the huge importance of other factors that are not included in the model. Nevertheless, I am interested in the influence of socio- economic independent variables (like gender, education level, etc.) and I do not have alternative theoretical models to test.

A lot of coefficients are highly significant despite the overall poor fit. Talking into account the overall focus of the study does it make sense to report those significant coefficients despite the poor model fit? Or should I stick to running simple linear regressions for each performance outcome with gender as independent variable?

What is your opinion?

Also please let me know, if you have any questions.
 

Lazar

Phineas Packard
#2
Lack of fit is due to parameters constrained not estimated parameters. Thus whether gender is a big or small predictor is not the problem. Have a look at the modification indices and check you have not specified your models (a parameter is fixed that you thought was free). In any case the mod indices will let you know where the misfit is coming from.
 

noetsi

No cake for spunky
#3
One proviso there. Not all SEM software will show the MI (or so I have been told - Mplus that I worked with did).

Also remember that the most common test of model fit is based on chi square which goes up (and thus tends to reject the null more often) with sample size. This is especially a problem over a thousand cases I believe.
 

Lazar

Phineas Packard
#6
One proviso there. Not all SEM software will show the MI (or so I have been told - Mplus that I worked with did).

Also remember that the most common test of model fit is based on chi square which goes up (and thus tends to reject the null more often) with sample size. This is especially a problem over a thousand cases I believe.
you need to make a distinction here between fit statistics and fit indices. Both are based on the chi-square statistics which is essentially the result from a discrepancy function multiplied by N (you sample size). Because of this as sample size increases even small discrepancies between model implied and observed covariances will be significant. Fit indices like the RMSEA adjust for sample size and so are unaffected by the concerns noted with fit statistics*

*RMSEA is upwardly biased for small samples and simple models
 

Lazar

Phineas Packard
#7
Thank you Lazar! Did not know mod indices!
Beware you do not blindly make changes based upon mod indices. It is generally considered bad practice to modify theoretical models based on post-hoc exploration of them. It is worth considering when a) you get funny results and think you might have made some mistakes in specifying the model. b) when fit is bad considering the source of misfit and seeing how that fits with your existing theory.