Event A occurring after 1500 event 4.

#1
Hello, I recently had an unlikely series of events occur. I’m wondering what the probability (or other relevant value) is, and also how I might calculate it.

I have performed a task about 1500 times in the past with no failures. Recently I made a single change and had 4 failures out of 22 attempts.

This change should not be able to produce this particular failure, and I’m left to believe it’s random chance. I’m wondering what sort of numberical values I can associate with this, and what they mean.

Thanks!
 
#2
I work a lot with systems at work and from experience small changes can have unforeseeable effects.

Lets call the two situations system A and B, before and after the changes respectively. We need to make the assumption that the probability of success is the same every trial for each system. This means that there is no tangible effects of wear and tear on parts, etc. Next determine the risk you are willing to take in making a wrong inference on this examination. Many people use 5%, or alpha 0.05. Then, conduct a 2 sample proportion test (https://newonlinecourses.science.psu.edu/stat414/node/268/). If the probability, or p-value, is less than your alpha than you can conclude that there is a significant difference between the probabilities of success in scenarios A and B.

Without even running this, I can tell you that the difference is significant for an alpha of 0.05.