Weighted paired-sample t-test?

#1
We have school-level data for 200 schools, 100 of which participated in a PD (professional development) program. For each school, we have an average test score, as well as the corresponding standard deviation, variance, and count of records associated with that average score. I have come across some websites explaining how to perform a weighted t-test, which seems like it could be useful in analyzing clustered/aggregated data such as this, since the count of records associated with the average test scores varies by school.

It seems like that could be useful to compare the average test scores for the 100 "control" schools compared to the 100 PD schools, but I'm wondering if it's possible to expand that further though to apply the same weighted approach but within a paired-samples? Specifically, for each school we actually have two sets of test scores (along with two sets of corresponding standard deviations, variances, and counts), one for before the PD program was implemented and one for after the PD program. Instead of just using the weighted t-test on the "post" scores, we'd like to be able to take into account the "pre" scores as well. Ideally, we'd like to evaluate the change in scores from "pre" to "post" (comparing "control" and PD schools), keeping in mind the weighted approach. One thing to note is that the "pre" and "post" record counts are usually different for the schools though. Does anyone have any thoughts about how we might tackle this? Thank you in advance for your help!
 

fed2

Active Member
#2
Hmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm.

I think your right about the weights in your first paragraph, i see similar experiments with weights of animals and I set the weight proportional to n. Careful though as different software expects different weights! The idea is to get a weight inverse to the variance. Averages have variance inverse to the sample size, so thats one way to look at why this is valid.

I think this also would hold in the pre-post situation in paragraph 2 if the students sampled at baseline and post were actually same students (and hence same n), under the same assumptions. The same logic would be applied to the paried differences, rather than just the post scores. If the n's are different at pre-post, this is probably not in general valid, since the variance could have a different relationship to 'n'.

You would probably have to enter both the pre and post measures into repeated measures model, along with the weights.

good luck,