Is it ok to run some planned comparisons and some post hoc?

[Esme]

Hello - I am a phd student in psychology and I would be very grateful for some advice about contrasts.

I have conducted a naming experiment for which response time was the outcome measure. There were 2 groups (treatment v control) and 3 testing conditions (A, B and C). I ran a mixed 2 x 3 repeated measures Anova with Group as the between subjects factor and Condition as the within subjects factor. There was a significant main effect of Condition, F(2, 86) = 70.99, MSE = 477,817, p <.001, η² partial = .62, as well as an interaction between Condition and Group, F(2, 86) = 15.31, MSE = [SPSS didn’t give... another problem for another time], p <.001, η² partial= .26.

I wish now to run comparisons and, whilst trying to work out whether it makes more sense to run planned comparisons or post hoc tests, I wondered whether it would be acceptable to run both: planned contrasts based on my predictions (below), and then post hoc tests on the remaining, untested contrasts - or if I should just pick one type.

Predictions:
Treatment group: Condition B > both A & C
Control group: no difference between B & A
(I was sort of agnostic about how C (a 'neutral' condition) would compare to the other conditions, except with respect to the prediction that B would be slower than both A and C for the treatment group.)

Other comparisons of interest, based on results
- both groups: C < A & B

Result Plot:

Many thanks in advance!

 

[noetsi]

I was always told that if you looked at the results its too late to do a planned contrast although that is a philisophical position not all agree with I assume. Formally you are supposed to generate the planned contrast before you see the data and base those contrast on theory not the empirical results you found.

Planned contrast have more power specifically because you are testing a narrow range of levels not all the possible levels of your model. Which is why you are supposed to have theory for them.

I think it would be justified to run both if you actually have a theory (or are testing someone elses theory) for each planned contrast rather than simply looking at the results and then creating post hoc a planned contrast tied to what you found.

 

[Esme]

Thanks very much, noetsi - indeed the predictions I listed are actual predictions that were outlined my research proposal (before I began data collection). In addition, this type of experiment has been carried out (with subjects from the same population I am testing) several times, and the effect is well-documented. So I am not worried about justifying treating those few contrasts as planned - just wondering whether following these up with post hoc tests would violate a rule, or render the results inaccurate somehow. As you can probably tell, I have neither a strong background in stats nor an easy grasp of the subject. I have consulted all manner of books, websites, you tube videos and papers in order to try to understand the tests I need to run, and I inevitably end up with more questions than I had to begin with. Thank you so much for your reply.

I'm sure this is really dumb, but - does the fact that my groups have unequal n's impact the contrasts at all?

 

[noetsi]

As some one who is not a natural statistician and who has spent countless hours reading books and links in stats(and classes) to correct this I feel your pain. Actually I feel exactly the way you do based on your comments. Take that into consideration when you listen to my answers - I am not the expert others on this board are in stats. And remember the experts themselves commonly disagree on fine points of analysis/stats.

I have never heard a rule against running post hoc tests and linear contrast although I think it is common to do one or the other. You would have to justify why, in the absense of theory, you are pursuing the ad hoc test which some regard as essentially a fishing expedition (and also in danger of generating results tied to a specific sample that might not reoccur in another sample). But that would likely be true regardless of whether you run the contrast or not. And based on your answer above I don't think you would run into this.

While there are some areas of ANOVA that have problems with unequal n's I have never seen it mentioned for contrasts. If different contrast you were testing had different sample sizes than this would I think influence comparisons between contrast because of power issues.