Designing a Photon Detector Test for High Confidence


I have a photon detector that I would like to test. Don't worry about the physics just the statistics. I think for this it is enough to say that every 20 minutes I can detect the direction of an incoming photon. The photon will come into the detector from an angle from 0 to 120 degrees. I can design the test that there will be a number of sources in known locations or a number of sources in unknown locations. I want to test whether this detector works and with a high confidence. From web reading, I see statistically significant means a p-value is less than 0.05 or the confidence interval does not overlap 0 on a graph of correlation on the x-axis and probability on the y-axis. I believe I understand confidence intervals better right now.

I can either do two approaches that I will call discreet numbers or azimuths. An example of discreet numbers would be: say I put one source "out there" and three fake sources. I then use the detector and looking at the different possible sources I say "I just got a photon from source 1." I then look at what is the real source and it came from source 1. How many tests do I need to do to get a certain confidence (either calculated by p-value or confidence interval)?

Azimuths: I put one photon source out there but I have no idea where it is. I then get an azimuth from 0 to 120. I then look and see what the real azimuth should be. How do I calculate the accuracy in this case and the confidence of what the accuracy would be?

Thanks! -xerxes73
Okay, I believe I have figured out the answer but I would appreciate if someone told me whether this was right or not.

In the case of the situation that I described as discrete, I believe I was able to find a similar problem in manufacturing where a certain part has to only have certain percent of defects. I found a binomial calculator on this website:

Let's say I have four possible targets. In order to know if I this photon detector is working better than random, I need to have a certain confidence that I am exceeding random which is 0.25, so I choose a p of 0.25. I am shooting for "statistically significant" so I choose a confidence of 0.95. I want to know how many trials I need to do if I don't have any failures. I type r=0 and then press calculate. This tool tells me 11. So I would need to do 12 to know I am better than random. If I start doing measurements and I get a failure, I then need to go to 19. And if I get two failures before 19, I need to keep going to 24. I would greatly appreciate if someone who knows what they are doing, checks my thinking on this.

Now in the case of azimuth. We get into a weird situation of is it working or not being separate from the azimuth accuracy. A failure in the detector is binary in a way. There should not be more credit for guessing 20 degrees off as opposed to 50 degrees off, when the accuracy of the detector's azimuth measurement is only a few degrees in error, i.e. if it is saying more than a few degrees then it totally did not work at all. So it seems to me that I need to characterize the azimuth accuracy at the same time that I characterize, separately, whether the detector works or not. I think this is all iterative. I will start with if the azimuth measured by the detector is within +/- 3 degrees of the actual azimuth then I will call it a success. I will calculate the probability of success and failure just like in the discrete case above. Then I will look at the distribution of azimuths of all the pass trials compared to the actual azimuth and it should be a normal distribution, although the normal distribution may have some sort of bias. If that normal distribution with a bias doesn't look right then I haven't selected the +/- 3 degrees correctly. It may be too big or too small. I will then iterate around +/- 3 degrees until I get data that looks right. The pass fail versus detector accuracy will trade back and forth in this way. Again, someone who knows what they are doing, please comment.

Thanks! -xerxes73
Alrighty, why no replies? Problem ill-defined? Too hard? Too easy? I would think this would be an interesting problem for folks because it doesn't seem to me to be too complex and it is applied. I would appreciate some insight from an expert on this..... -xerxes73