KS statistics, KS test, multiple empirical distributions

#1
I try to use KS statistics and KS test for fault detection.

Aparatus is generating 6 empirical distributions for two different variables, i.e.; 12 empirical distributions in total.

Each of 6 distributions (for a variable) should be similar, otherwise it should indicate a problem.

I could use KS statistics and compare it with a treshold that is determined from analysing KS statistics (distances) among each of these 6 distributions (in total 6x5 "distances"), that I can aquire them from numerous experiments and compare it against 99.9 percentile treshold. For each variable I would have different treshold (because each set of distances would have close but different percentiles).

Actaually, by collecting KS deviations (distances?) among distributions from numerous experiments I also have empirical distribution for these distances.

Also, I could use KS test, that compares KS distance with a specific criterion (used in KS test), that depends on a number of samples from which empirical distributions isetermined (all distributions for each variable are determined from the same number of samples, about 1000+ samples). The null hypothesis is rejected at a specific level alpha (e.g. 0.001) if KS distances is greater than that criterion. Now, for both variables I would have the same treshold.

The second solution seems right to me, and more "generic", whilst the first one have separate treshold for each variable based on observed KS distances from experiments, and two tresholds seem attractive to me.

What would you recomend?
 
Last edited: