Pre-Test/Post-Test (Near) Dichotomous Data

I am helping a student who is designing the following study and want to ask for feedback on the statistical testing. An outdated approach is common (according to her) of nurses administering 3 medications (“multi-agent approach” – MA approach) when 1 newer medication (“single-agent approach” – SA approach) would be beneficial. The student wants to review charts of the 25 nurses in the unit where she works to see what proportion of the SA approach currently occurs. She estimates examining 100 total cases although it won’t be exactly 4 per nurse. She will then do an educational seminar describing the benefits of the SA approach and examine charts over the next 3 months to see if the proportion of SA cases increases. Note that the student suspects there will be almost no variability within nurse across trials. Although different nurses differ from each other in terms of using the SA/MA approach but the same nurse almost always does the same thing each time according to her.

There are two ways I’m thinking about doing the data analysis: In the first analysis, each patient is a data point. So there are 100 dichotomous data points (SA or MA) pre-seminar and 100 dichotomous post-seminar. If we do this approach, I think the data do not meet the independence requirement of a chi squared test (the same set of nurses are involved pre/post). So I think it would be a McNemar’s Test. We would match Nurse 1/Case 1/Pre-seminar with Nurse 1/Case 1/Post-seminar then Nurse 1/Case 2/Pre with Nurse 1/Case 2/Post, etc. Would this be an acceptable use of McNemar’s Test? I know it’s not the typical use of McNemar’s because there are multiple observations from each nurse in the pre and post periods.
In the second approach, each nurse is the data point. We record for each of the 25 nurses the proportion of SA trials pre- and post-seminar. Because the student believes most of the proportions will be 0 or 1 (the same nurse typically does the same thing) or close to that (like 0.2 or 0.8), I was thinking of dichotomizing the variable. My understanding is that with 25 nurses, that is not sufficient to do a McNemar’s Test. We would look at the number of instances where nurses switched from MA in the Pre to SA in the Post compared to the reverse switch and do a binomial test. Would dichotomizing the proportions and doing a binomial test be OK?

I am thinking we could report both of the tests above. Would it be acceptable to do this? Even if yes, are there better ways to analyze the data?


Super Moderator
You could do a multilevel model here, with patients nested within nurses nested within time periods (pre / post). You would just need to specify this as a binomial logistic model rather than a normal model given the dichotomous DV.

That being said, spending a great deal of time on sophisticated statistical analysis here does seem like putting the cart before the horse: The fact that there is no control condition here means that the design is a "pre-experiment" (not even a "quasi-experiment"), and even if a change in behaviour is detectable there'll be no way to say that it was caused by the intervention. If I had a student propose this design I would probably be going back and saying that they need to re-think the design (although that depends on the level the student is at of course - maybe acceptable for an undergrad research project).


Not a robit
Another option is a difference in differences model where the outcome is a count variable (poisson), where you would control for multi-levels as @CowboyBear mentioned.

But as noted this isn't a novice model.