Multilevel analysis

noetsi

Fortran must die
#1
This is a thread on the wonderful world of multilevel analysis which is called many different things by various authors.

This is the type of comment that raises real doubts in my mind about statistical analysis. It comes from a multilevel graduate class.

Observations are pooled (i.e., groups are
combined and group membership ignored) to
estimate model coefficients. For example, data
from students within schools would be analyzed
at the student level (ignoring schools) even if
school features are of interest.

Between-group variation is ignored.
Estimates are OK when variation between
groups is negligible (e.g., school means are
comparable).

When between-group variation is not negligible,
larger groups will dominate the analysis.


SE will be invalid as well I believe.

Given that almost everything is nested inside of something, and between group variation will not be negligible, this would seem to invalidate much or all of linear and logistic regression. Similar limits come up all the time for other methods.

So if these type of problems are in fact common, how much of the analysis done is actually valid?
 

hlsmith

Not a robit
#3
There is a quote, which I don't know who to attribute to, but it goes: "once you know multi-level modeling, every model becomes (looks like) a multi-level model."


Yeah your above quote seems a little dubious, but to the author's defense, there are tests to determine if random effects are needed or not (amount of variability and I^2 tests in meta-analyses). Though, analysts need to remember that those tests need to be well powered.
 

noetsi

Fortran must die
#5
There is a quote, which I don't know who to attribute to, but it goes: "once you know multi-level modeling, every model becomes (looks like) a multi-level model."


Yeah your above quote seems a little dubious, but to the author's defense, there are tests to determine if random effects are needed or not (amount of variability and I^2 tests in meta-analyses). Though, analysts need to remember that those tests need to be well powered.
What I have seen stressed is the ICC (interclass correlation) there are rules of thumb how high this number has to be to make ML useful. I don't think power is an issue with this although I have not seen that addressed. My data involves thousands of cases effectively whole populations.
 

hlsmith

Not a robit
#6
Yeah, I wasn't referring to your data per se but speaking from my own experience. And yes there are some tests associated with these ICC values.
 

noetsi

Fortran must die
#7
Raudenbush and Bryk raise the issue that diagnostics for multilevel analysis is often different than regular linear regression. Does anyone know a good source for how you do tests of the assumptions of ML (multilevel)? I don't mean the theory, I mean what you actually do in practice including ways to address assumptions that are violated. For example R and B don't address the use of White's SE at all leaving me unclear if those work in multilevel models.

They also raise questions, but provide no useful guidelines IMHO, on how many level 2 units you have to have to estimate each slope at level 2 (well slope plus intercept) when the slopes and intercepts have collinearity. Has anyone seen a suggestion on what the minimum number is? In honesty I am not sure what you do if you don't have enough level 2 units to estimate all the parameters that should be random [so require a level 2 analysis].

What makes this worse for me is that I commonly analyze populations not samples and remain uncertain if I should even test the regression assumptions other than linearity....:p

While I am asking, the authors mentions the use of empirical bayes residuals [we did not use empirical bayes elements in my class at all]. Are these preferable to OLS residuals to analyze the failure of assumptions? I know nothing of these at all....
 
Last edited:

hlsmith

Not a robit
#8
Side note on something I had to figure out when delving into MLM, the more groups the better. By groups I mean clusters. I believe it comes into play via degrees of freedom in the modeling. So ideally you want more clusters to have more power, and I am not talking about levels, but groups AKA clusters in say level 2. I am sorry in that I don't have a great reference for this or any theoretic rationale.
 

noetsi

Fortran must die
#9
I am not sure what clusters are. I am going to run individual results inside our units, primarily to determine how well the units perform since assignment to them is anything but random, but the units have a clear impact on personal results. To me that suggest multilevel models although today I read in an expert in the field...

If, on the other hand, the statistical inference aims only at the particular set of units j included in the data set at hand, then a fixed effects model is appropriate.
https://pdfs.semanticscholar.org/f78d/7e8d9e612263644e653785b55df3a0533049.pdf

I am not really interested in national effects, just our units so I wonder if I should even use multilevel analysis in that case. However, the author does suggest this as well

This is the case, e.g., when one wishes to test the effect of an explanatory variable that is defined at level two, i.e., it is a function of the level-two units only. Then testing this variable has to be based on some way of comparing the variation accounted for by this variable to the total residual variation between level-two units, and it is hard to see how this could be done meaningfully without assuming that the level-two units are a sample from a population.
I have factors that likely work on the units directly and only indirectly on the customers. Since I am not going to model indirect effects as with SEM perhaps I could model this through multilevel analysis?

I have about 50 units, which would be the level 2 in my model.
 
Last edited:

noetsi

Fortran must die
#10
Residuals at level one which are unconfounded by the higher-level residuals can be obtained, as remarked by Hilden-Minton [27], as the OLS residuals calculated separately
within each level-two cluster. These are just the same as the estimated residuals in the OLS analysis of the fixed effects model, where all level-two (or
higher-level, if there are any higher levels) residuals are treated as fixed rather than random.
I am really confused what this means. Does it mean that you have to run separate residual analysis for each upper level group (each J)? I commonly have 50 groups and running fifty separate residual analysis is not a reasonable thing to do...
 

noetsi

Fortran must die
#12
For clarification, what I was calling clusters are your units.
I guessed that :)

When you run multilevel and check residuals, do you check the residuals separately for each group/cluster? So if you have 10 groups you conduct ten residual analysis one for each group (ignoring that you run the analysis for first level then do the same thing for each parameter for each group at level 2 in a 2 level analysis)?

That seems prohibitively time consuming.
 

noetsi

Fortran must die
#13
We have units that I believe have independent and/or moderating effect on our results. Units that provide service that is. I want to test if they have an impact and if so how and what drives that impact at the same time I want to determine what drives success in our units generally.

Multilevel approaches is how I decided to pursue this. But I have wondered recently if that is the best way to do it.
 
#14
Moderating means interaction to me. If you have a potential interaction you typically add an interaction term. Can you bette describe your context.


Thanks.
 

noetsi

Fortran must die
#15
I am not sure that is what I mean. What I mean is that factors like vendor services [nearly all services are provided by outside contractors] we provide impact results. But those services are provided by units who [in practice if not policy] who also impact customers. This could be independent of variables like vendor services or it could interact with them as you suggest.

So our units, which counsel customers and direct them to vendor services influence results and factors like services influence results. Possibly they interact and possibly not, but they both influence results [or that is my theory].
 

noetsi

Fortran must die
#16
This continues to confuse me. Some discussions of multilevel models make the distinction that random effects are important effectively when you don not have all the levels that exist [including ones that might exist in the future]. If you have all possible levels than you have a fixed effect and multilevel analysis is not important.

But other discussions of multilevel analysis focus on the impact of individual (level 1 variables) inside groups. So that if individual variables impact is influenced by groups then you have a random effect. Regardless of whether you have all the levels or not of a given variable. These seem incompatible usages. I often have population data. But it is likely the group, like area or unit, still influence individual variables. So should I use multilevel analysis or not in that case?

Maybe I am confusing multilevel analysis with the nature of random effects.
 

noetsi

Fortran must die
#17
I thought the whole point of ML (multilevel analysis) is that the impact of the lower level variable on the DV varied at various levels of the upper level variable. But recent discussions I have seen say this is only true if there is an interaction effect between first and 2nd level and not otherwise.

Here is an example of what I mean.

This is what the author describes. "In a country where individuals rate their health more poorly on average, having less education has more negative effects relative to countries where individuals rate their health better on average...." that seems like an interaction to me. But the authors consider it a random effect a covariance.

https://www.childhealthdata.org/docs/nsch-docs/carle_2010_mlm_mch_epi-pdf.pdf?sfvrsn=1

I did not see a slide number.
 
Last edited:

noetsi

Fortran must die
#18
Maybe I should start with something simpler. What do random coefficients really tell you. That is the variances and covariance's that are random effects. You interpret a slope as they change in Y for a change in X. How do you practically not in theory, interpret the covariance's and variances that are random effects.
 

hlsmith

Not a robit
#19
I may muddy this up since I haven't ran a MLM for quite awhile, but multiple logistic regression is fixed effects. Say you have another variable which is a level up (e.g., school, county, etc.) where the observations are nested in them and differ between the upper level variable groups. So now you calculate the effects within and between the group levels and you account for more variability. In addition, MLM allows you to control for random intercepts or not if necessary.


Sorry this is vague, but it is based on my recall.
 

noetsi

Fortran must die
#20
You were my great hope in this hlsmith since spunky does not usually comment on ML threads :p
What you say is true I think. What confuses me is what exact the random terms are telling you, how you interpret their coefficients. It appears that to determine the effects of a first level on the DV inside a group variable you use interaction analysis rather than random effects. That said none of the sources I have found tell you how you do this. They say you specify an interaction term which tells you if this exist. But not how you know how the level two variables, or the groups which are not the same thing as the level 2 variables, actually influence the impact of the level 1 variables. Nothing like simple effects comes up.

I want to know how groups moderate level one predictors (how they influence their slopes) but nothing I have found so far deals with this. Random effects tell you to some extent how much they influence the level 1 predictors, but now, in what direction they do this.

The explanatory variable for differences in the intercept [level 2 predictors I think] need not be the same as those for the effectiveness of the treatment. For example, a good predictor of the intercept might be a girl’s height and a good predictor of the slope might be the type of treatment that a girl receives.
This is a problem I have with the combined equation. If the level 2 predictors indirectly influence the dependent variable through the level 1 predictors, but have no direct impact, they should not show up as significant in the combined equation. So they will be dropped despite being important. I do not see anything about estimating indirect effects in the ML literature.