Determining the winner between two pricing strategies

#1
I'd like to conduct an A/B split test of pricing strategies on two web pages, and I'd like to know when to stop the test and declare the leader as the winner, at a specified level of confidence.

The winner is the offer that returns the most revenue per visitor.

Example: Offer A gets 1000 visitors and 25 buyers at $100 each. Offer B gets 900 visitors and 60 buyers at $30 each. Offer A is the winner because $2.50 per visitor is more than $2.00 per visitor.

Obviously, I want to end the test as quickly as possible, declare the winner, and move on to the next test. But how do I calculate my confidence level that the leader is the winner?

If possible, express your solution in terms of Excel's statistical functions.

Thanks.

Charley
 
#2
I believe a simple two group mean comparison t-test would work here. A tutorial for excel can be found here: http://www.stattutorials.com/EXCEL/EXCEL_TTEST1.html

That tutorial also includes a discussion of how you would structure your data for the test.

Presumably, you would run the t-test at specified points in time until the proportion of "winning" t-tests per side hits some predetermined thresh hold.

I don't know a lot about A/B testing but I would caution you that you would not want to start comparing means until you were sure that both ads had been seen by a representative audience. Starting a test on Tuesday morning and then comparing mean expenditures at 2pm could lead to spurious results if the audience between 7AM Tuesday and 2PM Tuesday is systematically different from the audience at other times.
 
Last edited: