Bayesian AB Testing - Minimum Sample Size

[samg86]

I've used python to analyse data from AB tests using Bayesian analysis, and for all tests I assume no prior knowledge and so set alpha = beta = 1.

However I'm finding some odd results at low data volumes, which I thought was my code, but I'm also seeing here: http://developers.lyst.com/bayesian-calculator/

This leads me to believe I don't understand the maths properly

If we take an AB test with the following parameters:

A trials: 100
A successes: 0

B trials: 10
B successes: 0

There is a 90% chance that B is better according to the analysis, however I don't understand how this can be the case with no successes recorded yet? The true success rate could be 0.00001% and this analysis should still be insignificant at this point surely?

How can I adjust parameters to ensure that there is no assumption on the success rate (or at least that I can control this assumption)?

 

[Dason]

If I take 10000000 free throws and miss all of them and then you take 1 and miss it... who do you think is better? I mean we don't have evidence that you're better but most people aren't so bad that that they can take 10000000 free throws and miss all of them so we might be fairly confident saying there is a XX% chance that you're better than me even though you've only taken a single shot.

With a bayesian analysis you have a prior distribution over the parameters of interest. If you're using a beta distribution as the prior for your success probability and use alpha = beta = 1 that means you're saying that any success probability is equally likely before we collect data. Once we collect some data if we find that we go a lot of trials without a success then our distribution on what we think the true success probability is will favor very low values. If you don't have much data then you don't have much ability to change the prior belief that any success probability was possible. So there is still a chance that there is a reasonably not tiny success probability. That's why you're seeing what you're seeing. If you don't want this to happen and don't want to make any prior assumptions then you might want to ask yourself why you're using a bayesian analysis. Either that or choose a more reasonable prior.