Measuring Success of Treatment

jman

New Member
#1
Hi,

I have a fairly basic statistics background so please be patient.

I set up an experiment where I have four groups of customers (randomly selected) and each group was given a different treatment (treatment #1, #2, #3 and control = no treatment). I am now trying to determine if any of the treatment was successful relative to Control in changing the spending pattern of my customers and if so, by how much. Ultimately, I would like to estimate if it is profitable for me to switch over and treat all my customers with the winning treatment so getting a good estimate on actual increase in spend is important. Each group has 10,000 customers.

Before applying treatments, I ran t-tests on the spending (and a few other key variables) for each group in the month prior to treatment and concluded that they are not statistically different (all t-stats < 1.96).

It has been a few months since my test and I am beginning to analyze the results. I ran t-tests on the mean spending during the test period for each group relative to the Control and concluded that spending for treatment #2 to be statistically significantly higher from the control group. But since the pre-treatment spend for this group is also higher than the Control group I have some concerns.

Here are my questions:

First, how should I estimate the actual increase in spend that is due to treatment #2 and how much of what I observed can be attributed to its higher pre-treatment spend relative to Control? (For example, Group 2's pre-treatment spend per month was $2020 and during treatment spend was $2055 vs. Control's pre-treatment spend was $2000 and during treatment spend was $2030).

Second, I tried using ANCOVA to isolate out the effects of the pre-treatment spend but when I plotted the relationship between pre-treatment spend with during treatment spend, the R^2 for the regression is quite weak ( < .40). Which brings up another question as to how low would the R^2 have to be for me to say I should not be using ANCOVA (if R^2 is 0, then there is no correlation between dependent variable and the covariate, thus ANCOVA would be the same as t-test in this case - am I right?)

Any help would be appreciated.