Compare change in test/control population means

I would like to do a retrospective observational study on some data. There was some functionality added to customer software and the question is does the added funtionality increase positive member outcomes? I have been building out my datasets with the idea of comparing the treatment group (customers exposed to the new functionality) versus a control group (not exposed to the new functionality). I have contructed the control group to do a a1:1 match using variables that correlate to psoitve customer outcomes (i.e. Gender, Age, and other characteristices).

I am analyzing a continuous outcome variable wich is a success percentage (between 0 and 100%). The higher the customer success percentage the better. I am new to this and learning as I go.

I believe this type of study is considered a cohort study and whatever variables I use to match and create a control group are going to be controlled for. Assuming this much is right:

1.) Does this seem like the right approach to accurately determine if the added functionality is leading to better customer outcomes?
2.) Is doing a t-test the right way do see if there is a significant difference between the test and control group sucess percentage?


Active Member
The analysis of matched pairs must allow for the within-pair correlation. A t-test assumes two sample independent groups. So I'm goin' 'no on 2.), i havn't thought about matching in a bit though.
Thanks for the reply.

In regards to within-pair correlation my understanding is that would be a factor if I was comparing the the treatment group to itself, using a before and after measuement? I am comparing a control group which is independent of the treatment group.

I don't feel confident that customers getting exposed to the treatment are representitive of the whole population. Is creating a control group that "looks" like my treatment population a good approach for isolating the effect of the treatment?

If a T-test is not the right way to measure if the treatment is having an impact, any suggestion on a better way to measure the effect?


Active Member
mcnemars test if binary, or i think paired t-test would be analogous for normal data. if matching is 1:1. See Lachin Biostatistical Methods for discussion of correlation induced (makes brain pop out ears a bit). Yes this is a good way, but the gain is primarily in the efficiency/reduced sample size needed.