Repeated measures ANOVA with factors not fully crossed at the subject level

#1
Greetings everyone, thanks in advance for any advice. I was asked to analyze a dataset from a study that used the a 2 (Treatment: A vs. B) x 2 (Treatment Type: Auditory vs. Visual) within-subjects design. The DV is the score on a final assessment.

I would normally expect to run a repeated-measures ANOVA in R (using aov wherein SCORE~TREATMENT*TYPE+Error(SUBJECT/(TREATMENT*TYPE))). However, there is a peculiarity of the design that I think may be causing problems. Namely that the factors are not fully crossed at the subject level. The issue appears to be that each subject experienced either of the following:

Treatment A is Visual AND Treatment B is Auditory
OR
Treatment A is Auditory AND Treatment B is Visual

That is, each subject experienced BOTH levels of Treatment (A and B), but only 1 level of Treatment Type for each Treatment, and always a different level of Treatment Type for each Intervention (Auditory for one and Visual for the other, or vice versa).

Ostensibly, the study is focused on the effects of Treatment, Treatment Type, and whether they interact.

In general this is a "within-subjects" design but I keep getting the "error model is singular" message when running a repeated-measures ANOVA using aov, and the sample sizes (for Treatment A-Auditory, Treatment A-Visual, Treatment B-Auditory, Treatment B-Visual) are equal. Maybe I'm off-base here and should focus on a different aspect. Thanks in advance for any advice!
 
#4
Thanks for the reply!

So you have data before, after (A+B)? or Before, after A, after B? if so what is the order?
Each participant was in one of the following between-subjects conditions.

Treatment A (Visual) then Treatment B (Auditory) -------> Final Assessment

Treatment A (Auditory) then Treatment B (Visual) -------> Final Assessment

Treatment B (Visual) then Treatment A (Auditory) -------> Final Assessment

Treatment B (Auditory) then Treatment A (Visual) -------> Final Assessment

The order of treatments is as described above. Note: the order of treatments was in the design for counterbalancing purposes, but is not of interest in the analyses. The goal is to determine any main effects of Treatment (A vs. B), Treatment Type (Auditory vs. Visual), and whether they interact. The design is a bit odd though as mentioned.

Can you please share a simple example of your data?
Sure! I've attached a screenshot below. Each participant has two rows of data. Each row includes one TREATMENT, one TYPE, and the SCORE on the final assessment of skills affected by that treatment.

Data.png

Thanks for any insights!
 

obh

Active Member
#5
Hi DW,

I still don't understand...
Can you please supply a bit more data about the content? maybe it will help me to understand.

Whats are treatments A and B?
What is the core? what is the final assessment?

So do you mean the order doesn't change the results? say the results of A->B will be the same as B->A
"Treatment A (Visual) then Treatment B (Auditory) " will be the same as "Treatment B (Auditory) then Treatment A (Visual) " ?
 
#6
Whats are treatments A and B?
The study is comparing ways to improve memory. There are two Treatment styles being compared.

Treatment A is an Active Memory Training Technique, whereas Treatment B is a Passive Memory Training Technique.

Both A and B can be administered in Visual or Auditory type.

Participants learned a list of words using A, and a second list of words using B.

They learned these words in either of the following orders:

Treatment A (Visual) then Treatment B (Auditory) -------> Final Assessment
Treatment A (Auditory) then Treatment B (Visual) -------> Final Assessment
Treatment B (Visual) then Treatment A (Auditory) -------> Final Assessment
Treatment B (Auditory) then Treatment A (Visual) -------> Final Assessment


What is the core? what is the final assessment?
I'm not sure what core means? As for final assessment, it is a recall test on both lists of words. On the final assessment, which is the outcome of interest, each participant has a score for the list of words that they learned using A, and another score for the list that they learned using B.

So do you mean the order doesn't change the results? say the results of A->B will be the same as B->A
"Treatment A (Visual) then Treatment B (Auditory) " will be the same as "Treatment B (Auditory) then Treatment A (Visual) " ?
The order of treatment is not important. Rather, the treatment orders were counterbalanced such that no treatment was always first or second throughout the entire sample.

The critical question is whether scores on words learned using A are better than scores on words learned using B, or vice versa (Main Effect of Treatment); whether scores are higher with Visual versus Auditory type of administration, or vice versa (Main Effect of Format); and whether there is an interaction. However, the design is incomplete insofar as there is no participant that went through Treatments A and B (both Visual), Treatments A and B (both Auditory).

Thanks for being willing to sort through my explanations!
 

obh

Active Member
#7
typo (s)core ...
Was the final assessment another test? or just a combination of the two test ???

Why TREATMENT*TYPE?

Per my understanding, in repeated measures ANOVA, all participants should go through all the steps.
Maybe you should do instead of repeated measures ANOVA the two factors ANOVA (yes I know you then ignore the dependency)
 
#8
typo (s)core ...
Was the final assessment another test? or just a combination of the two test ???
A combination. That is, a combination of the recall test on the list that was learned via A, and the recall test on the list that was learned using B.

Why TREATMENT*TYPE?

Per my understanding, in repeated measures ANOVA, all participants should go through all the steps.
Maybe you should do instead of repeated measures ANOVA the two factors ANOVA (yes I know you then ignore the dependency)
Thanks for the suggestion. To clarify, in R, running something like aov=SCORE~TREATMENT*TYPE, and that's it (ignore the dependency)...?
 

obh

Active Member
#11
Just to ensure you only used the score, not a final score as a combination of two observations. So every combination of treatment/type/score is one observation. Exactly as you showed in the excel table
 
#12
Just to ensure you only used the score, not a final score as a combination of two observations. So every combination of treatment/type/score is one observation. Exactly as you showed in the excel table
Yes, exactly as in the Excel table, not a combination of those observations. Thank you for your comments and advice on this!