- Thread starter noetsi
- Start date

An assumption that is us usually made in ML regression is that the variance of the residual errors is the same in all groups. This can be assessed by computing a one way analysis of variance of the groups on the absolute values of the residuals which is the equivalent of the Levene test

Is this the correct way to generate the empty model to calculate that?

proc mixed data= work.test2 covtest noclprint;

class unitid_pri;

model DV = /solution;

random intercept /subject= unitid_pri;

run;

I calculated the ICC by Estimate /(estimate + residuals) as I found on line. Is this correct?

My ICC is about 3.8 percent. Is that enough to see multilevel models as useful, or is it suggesting that group really has a limited effect?

The Wald test for Random effects, the p value that tells you if a random effect is significant, is commonly seen as wrong. It is recommended that instead you add one random effect, look at the change in the model deviation, and do a chi square test (deviation model 1 -deviation of model 2) with a df of one using a chi square test.

Does anyone know how to do this in SAS?

This is one discussion of the approach

Consider another example in which a model with a random effect for the slope (i.e, is the slope is allowed to vary) is compared to a model without the random effect for the slope (i.e., the variance

of the slope is constrained). This example would appear to be testing a single parameter, but, infact, the two models differ by two parameters. The first model will include an estimate of the slope

variance, τ 2 1, but also an estimate of the covariance between the slope and the intercept, τ10, by default. The covariance cannot be estimated when the slope is constrained to be non-varying,

however. One would ordinarily expect that the difference between the two models would be compared to the chi-square distribution with df = 2, because two parameters differed between the

models being compared. But because variance tests should use a one-tailed test and covariance tests are two-tailed tests, a more complicated significance criterion is needed. Snijders and Bosker

(2012, p. 99) recommend using a "mixture distribution" (or "chi-bar distribution") by comparing the chi-square difference obtained from subtracting D0 – D1 to a combination of two critical values. For

α = .05, the critical values are: one slope 2 χ mix = 5.14, two slopes 2 χ mix = 7.05, and three slopes 2 χ mix = 8.76.

http://web.pdx.edu/~newsomj/mlrclass/ho_significance.pdf

The way I interpret this is if you were testing one random effect you would run it first with the fixed but not random effect for a slope (that is for the slope fixed). Then run it with the random and fixed effect for that predictor and determine how the deviance differed. Then, using this deviance, and a df of 2, you would run a chi square test. If the result was greater than 5.14 you would conclude the random effect was significant at the .05 level.

Or are you comparing the empty model to the model with a random and fixed effect for that predictor specified and using those two models to get the difference in the deviation (everything else would be the same I assume).

Does anyone know how to do this in SAS?

This is one discussion of the approach

Consider another example in which a model with a random effect for the slope (i.e, is the slope is allowed to vary) is compared to a model without the random effect for the slope (i.e., the variance

of the slope is constrained). This example would appear to be testing a single parameter, but, infact, the two models differ by two parameters. The first model will include an estimate of the slope

variance, τ 2 1, but also an estimate of the covariance between the slope and the intercept, τ10, by default. The covariance cannot be estimated when the slope is constrained to be non-varying,

however. One would ordinarily expect that the difference between the two models would be compared to the chi-square distribution with df = 2, because two parameters differed between the

models being compared. But because variance tests should use a one-tailed test and covariance tests are two-tailed tests, a more complicated significance criterion is needed. Snijders and Bosker

(2012, p. 99) recommend using a "mixture distribution" (or "chi-bar distribution") by comparing the chi-square difference obtained from subtracting D0 – D1 to a combination of two critical values. For

α = .05, the critical values are: one slope 2 χ mix = 5.14, two slopes 2 χ mix = 7.05, and three slopes 2 χ mix = 8.76.

http://web.pdx.edu/~newsomj/mlrclass/ho_significance.pdf

The way I interpret this is if you were testing one random effect you would run it first with the fixed but not random effect for a slope (that is for the slope fixed). Then run it with the random and fixed effect for that predictor and determine how the deviance differed. Then, using this deviance, and a df of 2, you would run a chi square test. If the result was greater than 5.14 you would conclude the random effect was significant at the .05 level.

Or are you comparing the empty model to the model with a random and fixed effect for that predictor specified and using those two models to get the difference in the deviation (everything else would be the same I assume).

Last edited:

You use the empty model to predict the interclass correlation. Is this the correct SAS model to do that? I am not sure if you specify the intercept as random or not.

proc mixed data= work.test4 covtest noclprint;

class unitid_pri ;

model weeklyearnings_clo = /solution;

random intercept /subject= unitid_pri;

run;

While I am at it the link below has a macro that performs the LR deviance test with mixed p values (both strongly recommended especially with random effects). One thing that is unclear to me is if you have to run the

%include '\\cdc\private\mixture method pvalue macro1.sas'; macro always or whether this unique to the data the author is using. He never mentions this macro at all.

The link http://analytics.ncsu.edu/sesug/2009/SC013.Bardenheier.pdf

proc mixed data= work.test4 covtest noclprint;

class unitid_pri ;

model weeklyearnings_clo = /solution;

random intercept /subject= unitid_pri;

run;

While I am at it the link below has a macro that performs the LR deviance test with mixed p values (both strongly recommended especially with random effects). One thing that is unclear to me is if you have to run the

%include '\\cdc\private\mixture method pvalue macro1.sas'; macro always or whether this unique to the data the author is using. He never mentions this macro at all.

The link http://analytics.ncsu.edu/sesug/2009/SC013.Bardenheier.pdf

Last edited:

proc mixed data=work.test4 method=ml covtest empirical

noclprint ;

class unitid_pri female ;

model dv=female/ ddfm=contain s ;

random intercept /subject=unitid_pri type=ar(1) s ;

parms / ols;

ods output FitStatistics=fm1 SolutionF=SFfm1 ;

Regardless of any fixed effect I test, there is only one variable in the model, I always get that the Hessian matrix is not positive definitive (the model converges, but this issue remains).

I have about 77 groups and 6000 cases. The parms/ols; statement sets a starting value which is one way recommended to deal with a non-positive Hessian matrix (but which made no difference).

NOTE: Convergence criteria met but final Hessian is not positive definite.

Its female status predicting the DV (its looking at groups, but not repeated measures). The intercept is random, the only fixed effect is female.

According to some authors the results are invalid if you get this comment.

If that doesn't work, read the following, if you haven't already. I remember coming across this back in the day. In particular the example about variability across classrooms and using repeated instead of random option. Though I would first mess with the variance/covariance structure. Even though Jake said he would probably not mess with the convergence criteria, I would perhaps tweak it a little to let the model run a little longer.

Lastly per the classroom example, you may need to also accept there isn't much variation explained by the group variable that isn't picked up in a simple model. Also what is your group variable? is it a geographic location, if so, locations may be getting separated but they are tangential and actually more similar than believed.

Link:

http://www.theanalysisfactor.com/wacky-hessian-matrix/

Lastly per the classroom example, you may need to also accept there isn't much variation explained by the group variable that isn't picked up in a simple model. Also what is your group variable? is it a geographic location, if so, locations may be getting separated but they are tangential and actually more similar than believed.

Link:

http://www.theanalysisfactor.com/wacky-hessian-matrix/

Last edited by a moderator: