In my case, I perform an accuracy audits where, for example, my required accuracy is 99.90% and my required confidence level is 90.00%. In my case, the calculations are binomial; that is, a single test is either right or it's wrong, then I move to the next. There is no resampling. I use Excel to calculate minimum sample sizes, confidence, and allowed errors. When an audit reveals too many errors for a sample size and confidence to both exceed the required minimums, I need to calculate penalties based on the number of errors exceeding the minimum required accuracy. This is fine when the measured accuracy is below, for example, 99.90%. But I'm stumped as to how to proceed when the measured accuracy passes the requirement, but the required confidence fails. For example:
I know that if I have a sample size of 3889, and I only have 1 error, then the formula "=1-BINOM.DIST(1,3889,1-0.999,TRUE)" will tell me my confidence level is 90.01%. My measured accuracy would be "=1-(1/3889)" or 99.97%. This meets both my measured accuracy and confidence level requirements.
But if I have 2 errors then my confidence level is 75.53% (failing), even though the measured accuracy is 99.95% (passing). So my accuracy is okay, but it's not at the required confidence level. The audit fails.
Summarizing the first paragraph with the example: My issue is that I need to assign penalties for failing to meet the required accuracy and confidence, but the penalties are based only on the number of errors beyond the required accuracy. With a sample set of 3889 I have to get 4 errors before my measured accuracy drops below 99.90%, even though the required confidence fails with both 2 and 3 errors. I'm stumped on how to assign penalties in a case where the measured accuracy passes the requirement, but the required confidence fails.
Is it possible, and would it be correct, to be able to calculate backwards to determine 'n' by saying something like, "With 3889 samples at 74.53% confidence, then it's expected that 'n' errors would have occurred if the confidence were at 90.00%." If so, how would this be done? Would there be a margin of error that could be calculated? If so, how? I'm stumped on how to move forward with this problem.
Note: I have rudimentary understanding of statistics, but not near enough to feel confident that any answer I come up with on my own would be appropriate. I need guidance, both for a solution and to understand the solution well enough to explain it to others. I also posted this question on an Excel message board before I found this forum. I have not received any answers. I will keep an eye out everywhere so the efforts of those that may help aren't wasted.
I'd appreciate any experienced thoughts on this matter.
Andrew
I know that if I have a sample size of 3889, and I only have 1 error, then the formula "=1-BINOM.DIST(1,3889,1-0.999,TRUE)" will tell me my confidence level is 90.01%. My measured accuracy would be "=1-(1/3889)" or 99.97%. This meets both my measured accuracy and confidence level requirements.
But if I have 2 errors then my confidence level is 75.53% (failing), even though the measured accuracy is 99.95% (passing). So my accuracy is okay, but it's not at the required confidence level. The audit fails.
Summarizing the first paragraph with the example: My issue is that I need to assign penalties for failing to meet the required accuracy and confidence, but the penalties are based only on the number of errors beyond the required accuracy. With a sample set of 3889 I have to get 4 errors before my measured accuracy drops below 99.90%, even though the required confidence fails with both 2 and 3 errors. I'm stumped on how to assign penalties in a case where the measured accuracy passes the requirement, but the required confidence fails.
Is it possible, and would it be correct, to be able to calculate backwards to determine 'n' by saying something like, "With 3889 samples at 74.53% confidence, then it's expected that 'n' errors would have occurred if the confidence were at 90.00%." If so, how would this be done? Would there be a margin of error that could be calculated? If so, how? I'm stumped on how to move forward with this problem.
Note: I have rudimentary understanding of statistics, but not near enough to feel confident that any answer I come up with on my own would be appropriate. I need guidance, both for a solution and to understand the solution well enough to explain it to others. I also posted this question on an Excel message board before I found this forum. I have not received any answers. I will keep an eye out everywhere so the efforts of those that may help aren't wasted.
I'd appreciate any experienced thoughts on this matter.
Andrew