Can you compare score differences across time using unpaired data?

#1
An anonymous school climate survey was administered to students in 10 schools in 2017. We have student-level data from that survey, which includes each student's overall satisfaction score (scale of 0 to 100) and the school in which the student was enrolled.

Five of the schools then implemented a behavioral intervention program for the next three years and then another anonymous school climate survey was administered to students in all 10 schools in 2020. We want to know if student satisfaction scores improved more in the five schools implementing the behavioral intervention program than in the five "control" schools. Unfortunately, because the surveys were anonymous and no identifying information was collected for the students (we recognize opportunities for improvements in the design moving forward), we do not have the ability to pair the "pre" (2017) and "post" (2020) data.

Is there a statistical test could we use to determine whether improvements in student satisfaction scores were greater for the "treatment" vs. "control" schools? Any guidance would be greatly appreciated! Thank you!
 

katxt

Active Member
#2
Schools are nested within treatments so any testing has to at the school level.
You could try this -
Find the mean score for each school before and after intervention.
Find the difference (after-before) score for each school.
Do a t test of the 5 intervention school differences against the 5 control schools differences.
A 5 vs 5 t test will only catch a fairly large difference but there is little to be done about that. Increasing the number of students will have a minimal effect. To get more power you need more schools.
 
Last edited: