1-way vs 2-way repeated measures ANOVA with missing data

[blooop]

I have performed an experiment where I tested how long it took users to reach various points on table and their perception of how difficult the task was to perform under various conditions. The user used either a handheld robot arm, or a handheld wand to reach the required points. In addition, the user was given information about how to reach the points by one of four feedback methods: a handheld LCD screen, a see-through augmented reality headset, a virtual reality headset, or gesturing actions performed by the robotic arm.

Gesturing feedback with the wand was not technically possible because it did not have motors in it.

I used a repeated measures design where 21 participants were tested once on each of the 7 modes. Note that we cannot test the (Wand - Gesturing) condition and it is left blank

╔═══════╦═════════════════╦═════════════════╦═══════════════════╦═══════════╗
║       ║ Handheld Display║ Virtual Reality ║ Augmented Reality ║ Gesturing ║
╠═══════╬═════════════════╬═════════════════╬═══════════════════╬═══════════╣
║ Robot ║ []              ║ []              ║ []                ║ []        ║
║ Wand  ║ []              ║ []              ║ []                ║           ║
╚═══════╩═════════════════╩═════════════════╩═══════════════════╩═══════════╝

I measured the results using 1-way repeated measures anova and used the Bonferroni correction on the posthoc analysis to produce a table of pairwise comparisons.

I submitted this as a paper but one of the reviewers said I should use 2-way repeated measures ANOVA instead of 1-way repeated measures.

In some ways I can see why they are suggesting 2-way ANOVA because I have two independant variables, device (robot,wand) and feedback (HHD,VR,AR,Gesturing). However, I was not able to record the last condition (wand-Gesturing) and my statistics software (SPSS) will not allow me to perform 2-way ANOVA without the missing data.

I have done a 2-way repeated measures ANOVA on the first 3 columns of the table excluding the gesturing condition, but this is unsatisfactory because I want to compare the Robot-Gesturing to the other Wand feedback methods.

I have spoken to people in my office who suggested testing for a difference between the Robot and Wand condition first, then testing for differences in feedback methods for the Robot condition and then for the wand condition independently. This has the same problem where I can't compare the Gesturing feedback to any of the Wand conditions.

What I would like to do is say that I am not considering the device (Robot/wand) as an independent variable, but instead consider each of the 7 modes as independent. In reality there is a correlation in performance between the Robot/wand so I don't think this is valid. However, is there a correction I perform that takes this correlation into account so that I can use 1-way repeated measures ANOVA and still perform pairwise comparisons between all the different conditions?

Thanks for you attention.

Appendix:

The paper is over the attachment limit so this is a dropbox link: https://www.dropbox.com/s/0xmebzara9dacrq/FeedbackPostReview.pdf?dl=0

An extract from the reviewer's comments:
"This study has 2 independent variables: 1) the device and 2) the feedback mode (this is called operation mode in Section III, which I believe is poorly named). Hence, the experimental design is 2-way. However, the analyses conducted in the paper utilize a 1-way ANOVA. This is technically incorrect. The authors need to report 2-way ANOVA results, which would clearly and reparately indicate whether there is a significant effect of feedback method and/or the device choice."

 

[mmercker]

Hi, as far as I know ANOVA usually can't handle missing data. Instead, you can formulate the problem as a multilevel model where participant ID is the random factor and device and feedback mode are the two fixed effect predictors. Multilevel regression models are usually more flexible regarding the data and e.g. cn handle unbalanced desing or missing values

 

[blooop]

Thanks for your reply.

I've done some research on how to do Multilevel models with SPSS but the closest thing I could find was a linear Mixed model. After running it on the data I have a clearer idea of what I want the statistics to describe.

This is a box and whisker plot of the data

Questions:
1. Overall, is there a difference in performance between the arm and the wand?
2. When using the robot arm, is there a difference between the feedback methods and if so, which is the best?
3. When using the wand, is there is a difference between the feedback methods and if so, which is the best?
4. How does the robot arm gesturing performance compare to all of the Wand feedback methods?

My existing work with 1-way ANOVA answers all those questions by calculating all the pairwise comparisons between all 7 conditions. However the analysis is not valid because it doesn't take into account the Device (robot arm/wand) as an independent variable.

Does multilevel regression models answer those questions? From what I have research on multilevel regression models with SPSS, I don't know if it can produce that output.

 

[mmercker]

Hi, linear mixed mode is OK, this is a multilevel model. They have various names: Mixed Models = multilevel model = hierarchical model=...

Do you have a numerical otput of the mixed model which cou could present? Unfortunately I am not used to SPSS. So did you use two predictor variables (Arm/Wand)and eachvariable with different levels? Usually each level is compared to a "baseline level" of the corresponding variable. If you want to test specific research questions which are not directly coded in your regression parameters, you can use either specific contrasts which represent your hypothesis (but this is a little bit sophisticated) or multiple comparisons, as you did for your one way ANOVA. But I do not know how to do this in SPSS.

 

[blooop]

Thanks, that's good to know.

I set the independent variable as completion time and the two factors as Device and Feedback.

When doing the 1-way ANOVA the dialogue asked for number of levels, but I was not asked to provide them here.

I then defined Device, Feedback, and Device*Feedback as fixed effects. I did not define any random effects.

I chose to display the marginal means for Device, Feedback, and Device*Feedback. I also chose to compare the main effects using the Bonferroni correction.

These are the commands the setup wizard generated

MIXED Time BY Device Feedback 
  /CRITERIA=CIN(95) MXITER(100) MXSTEP(10) SCORING(1) SINGULAR(0.000000000001) HCONVERGE(0, ABSOLUTE) LCONVERGE(0, ABSOLUTE) PCONVERGE(0.000001, ABSOLUTE) 
  /FIXED=Device Feedback Device*Feedback | SSTYPE(3) 
  /METHOD=REML 
  /PRINT=DESCRIPTIVES  TESTCOV 
  /EMMEANS=TABLES(Device) COMPARE ADJ(BONFERRONI) 
  /EMMEANS=TABLES(Feedback) COMPARE ADJ(BONFERRONI)

I've attached a pdf of the output. It gives a pairwise comparison between the feedback modes (aggregating the results of both Robot/wand). It also gives a pairwise comparison between for the device (aggregating feedback modes). However it does not produce the pairwise comparisons for each of the 7 conditions which is what I would like to produce.