Choosing an appropriate test: MANOVA? ANCOVA? Something else?

[Eaners]

I am performing a "dry run" statistical analysis on my dissertation data pretest, and I'm looking for a little guidance regarding the specific analyses I need to perform.

I have three primary variables:

• Environment. Categorical, 1 of 3 conditions. Independent, assigned by experimenter randomly.
• Self-Monitoring Score. Continuous variable, 0-18. Independent, calculated using responses to Snyder's self-monitoring scale.
• Uses & grats. These are the dependent variables. 59 5-point Likert-type items that have been put through factor analysis resulting in 12 factors with eigenvalues > 1.

From this, I have three research questions, paraphrased below:

1. Are there statistically significant differences in uses & grats based on environment?
2. Are there statistically significant differences in uses & grats based on self-monitoring score?
3. Are there any discernible interactions between environment, self-monitoring score, and uses & grats?

The first question I have a handle on; I ran a one-way ANOVA using the factor analysis scores and the three scenarios, and that worked like a charm.

The other two questions are where I'm getting a little stuck. For #2, you can't do a one-way ANOVA using a continuous independent variable, so would it make sense to break subjects up into quartiles and just run the ANOVA as if the continuous variable was four groups? Or is there a better way?

As for #3, I'm not sure what test at all that would be. Any recommendations?

 

[spunky]

hello again. i'm just gonna be nit-picky because i like you and not because i'm trying to be mean. otherwise i'd just let your thread sink into the depths of oblivion.

Uses & grats. These are the dependent variables. 59 5-point Likert-type items that have been put through factor analysis resulting in 12 factors with eigenvalues > 1.

bad rule for extracting factors. it is well-known that it extracts more factors than it should and its susceptible to things like the number of variables in the dataset. so like say you have a scale that has 100 items and one that has 10, but they all only have a 1-factor structure. the eigenvalue>1 rule would select more factors from the first scale for no other reason that it has more items and a bigger covariance/correlation matrix.

the most accurate methods to indicate the number of factors to extract are computer-based, monte carlo simulation methods. particularly one that is called Parallel Analysis. this is what is recommended by the literature.

for a very easy-to-use web-based app to do parallel analysis on your data click here:

http://ires.ku.edu/~smishra/parallelengine.htm

for a more in-depth discussion (but still accesible to the non-initiated in the secret mysteries and dark arts of psychometric analysis) click here:

http://pareonline.net/pdf/v12n2.pdf

The first question I have a handle on; I ran a one-way ANOVA using the factor analysis scores and the three scenarios, and that worked like a charm.

bad move again. it is well-known that using factor scores as if they were observed scores in analyses meant for observed data (so multiple regression, ANOVA, etc.) results in biased parameter estimates and spurious signifiance. a more in-depth discussion would be:

Tucker, L. (1971). Relations of factor score estimates to their use. Psychometrika, 36, 427–436.

Shevlin, M., Miles, J. N. V., & Bunting, B. P. (1997). Summated rating scales: A Monte Carlo investigation
of the effects of reliability and collinearity in regression models. Personality and Individual Differences,
23, 665–676

the main problem, however, stems from the fact that factor-scores are not invariant to rotations. if you find a factor analysis solution that fits your data decently and extract factor scores to do some multiple regression/ANOVA analysis type i can just as easily use the same factor solution, rotate it until i get a different set of scores and do an analysis that might contradict yours, using the same data and the same factors. this is referred to as the 'factor indeterminacy problem' (which is actually a collection of problems) but the easier one to grasp is this one, which starts by the fact that there are an infinite number of factor rotations that fit the data equally well.

if you want to play in latent-variable land (so factors form factor analysis) you would need to do Structural Equation Modelling (SEM).

The other two questions are where I'm getting a little stuck. For #2, you can't do a one-way ANOVA using a continuous independent variable, so would it make sense to break subjects up into quartiles and just run the ANOVA as if the continuous variable was four groups? Or is there a better way?

As for #3, I'm not sure what test at all that would be. Any recommendations?

i'm not sure if you know this or not (some people do, some people don't). ANOVA *is* regression. it just happens to be a very constrained type of regression where all the predictors are categorical. there is nothing that would prevent you from doing a similar analysis if you just conveniently code your predictors (you know... dummy coding, effects coding, all that stuff) to handle categorical/nominal data, leave the continuous data as it is and fit it like a linear model. i won't go into the details as for why discretizing a continuous variable is not a good idea... but, trust me on this one, it is not.


then again this is solely contingent on the statistical expertise of the people who will review your work. i'm pretty sure you can do outrageously wrong things in your field (which i believe is media/communication studies, right?) and nobody would bat an eyelash

 

[Eaners]

Hi! Thanks for the reply! I really do appreciate it. While you are probably right that my department and committee won't be super in-depth w/r/t my methods, I'm more interested in understanding the right way to do things.

bad rule for extracting factors. it is well-known that it extracts more factors than it should and its susceptible to things like the number of variables in the dataset. ...the most accurate methods to indicate the number of factors to extract are computer-based, monte carlo simulation methods. particularly one that is called Parallel Analysis. this is what is recommended by the literature.

It took me about three minutes to find a half-dozen papers, all from either my field or psych, saying exactly this--and that just because SPSS and other packages tend to default to the Eigen < 1 rule, that's what everyone uses. I do not like that reason.

How do you feel about Velicer's MAP test? It's recommended by my psychometrics prof, and I found a SPSS program that runs it; it suggested 7 variables rather than the 12 that my initial FA was giving me. However, when I went in and actually looked at the loadings and the way the variables were hanging together, 7 looked like too few from a logical standpoint, so I omitted 3 with the lowest communalities (all were <.45) and then threw out three more that were clearly confusing, and suddenly I was at 10 factors (again, still using the Eigen <1 rule). I'm thinking that if I tweak the variables that were hanging together in some of the weaker and more ambiguous factors, I may actually end up with 7 in the end.

bad move again. it is well-known that using factor scores as if they were observed scores in analyses meant for observed data (so multiple regression, ANOVA, etc.) results in biased parameter estimates and spurious signifiance... if you want to play in latent-variable land (so factors form factor analysis) you would need to do Structural Equation Modelling (SEM).

Fair enough. In this case, though, would the method that I mentioned be "good enough" (whatever that means)? I'm looking to only establish that there are statistically significant differences between my conditions; however, I can (and will) include your concerns in my discussion section, though.

i'm not sure if you know this or not (some people do, some people don't). ANOVA *is* regression. it just happens to be a very constrained type of regression where all the predictors are categorical. there is nothing that would prevent you from doing a similar analysis if you just conveniently code your predictors (you know... dummy coding, effects coding, all that stuff) to handle categorical/nominal data, leave the continuous data as it is and fit it like a linear model. i won't go into the details as for why discretizing a continuous variable is not a good idea... but, trust me on this one, it is not.

For the continuous scale that I'm using (Snyder's 18-point self-monitoring scale), there is quite a bit of precedent for breaking the score up into "high" and "low" based on a cut point, since Snyder himself routinely referred to people as either "high" or "low" self-monitors; I'm basically struggling with whether it would make more sense to leave the scores as-is and use them on a continuous scale, or if I should just go with what Snyder et al. have done and group the scores.

then again this is solely contingent on the statistical expertise of the people who will review your work. i'm pretty sure you can do outrageously wrong things in your field (which i believe is media/communication studies, right?) and nobody would bat an eyelash

The more I learn about statistics, the more shocked I am by what people can get away with in basically any field... but you might be right!

 

[spunky]

How do you feel about Velicer's MAP test? It's recommended by my psychometrics prof, and I found a SPSS program that runs it; it suggested 7 variables rather than the 12 that my initial FA was giving me. However, when I went in and actually looked at the loadings and the way the variables were hanging together, 7 looked like too few from a logical standpoint, so I omitted 3 with the lowest communalities (all were <.45) and then threw out three more that were clearly confusing, and suddenly I was at 10 factors (again, still using the Eigen <1 rule). I'm thinking that if I tweak the variables that were hanging together in some of the weaker and more ambiguous factors, I may actually end up with 7 in the end.

well, i think Velicer's MAP test is definitely an improvement over the eigenvalues>1. but parallel analysis is still the best one. actually, Velicer himself in Zwick & Velicer (1986) recommends parallel analysis over his own MAP test. so my guess is that if the creator of the test himself is telling you "don't use my stuff, use this other thing because it is better" then... it's probably a good idea to listen to him.

please keep in mind that all these "factor extraction" methods are nothing beyond rules of thumb. theory is the only true substantive pointer on which you can rely to do these things correctly. so i would try and find a compromise between what the statistical methods and the theory say. remember, these are all pieces of the puzzle but it is up to you to assemble them correctly.

Fair enough. In this case, though, would the method that I mentioned be "good enough" (whatever that means)? I'm looking to only establish that there are statistically significant differences between my conditions; however, I can (and will) include your concerns in my discussion section, though.

define "good enough" please i am not aware of any studies who aim to show what the "least worse" solution is specifically within this context. all i know is that there is one good answer, and that is to use Structural Equation Modelling. you mentioned that this was dissertation-related stuff so in the interest of time i would probably leave it as it is and just keep in mind that what i'm finding could potentially be nothing more than a statistical artifact of the way i'm analyzing the data and go back to it once i'm more proficient in more advanced techinques.

by the way, and me being curious where... which method of factor extraction are you using? since you initially used the eigenvalue>1 rule i'm tempted to guess you're using Principal Component Analysis? and did you rotate your solution? or did you leave those dialogues alone and let SPSS 'do its magic'?

For the continuous scale that I'm using (Snyder's 18-point self-monitoring scale), there is quite a bit of precedent for breaking the score up into "high" and "low" based on a cut point, since Snyder himself routinely referred to people as either "high" or "low" self-monitors; I'm basically struggling with whether it would make more sense to leave the scores as-is and use them on a continuous scale, or if I should just go with what Snyder et al. have done and group the scores.

well, back in the day there was also a precendence for doctors to recommend smoking and saying the world is flat. it didn't quite make it correct, did it? are you aware of how this cut-point was obtained? do you know of validity studies that substantiate this cut-point? if you do then sure, you can cite those and divide the group into "high" and "low" groups. if there are not (and my guess is that there aren't) and Snyder et.al. just felt to use a score for kicks and giggles (which is what people usually do) i'm just gonna call BS on the whole thing from a data-analytic perspective.

i'm still not getting why you can't just fit it all in a regression model that would look like:

Uses & Grats = bo + b1(Self Monitoring) + b2(Environment) where 'Environment' is properly effects-coded so you end up with a b2 coefficient that behaves like you'd do in an ANOVA. but i guess that's up to you.