System test scripts - need a statistical model

#1
My company develops and runs system test scripts, containing multiple steps each, on various ERP systems like Oracle. I have been asked to develop a statistical model that allows expectations to be determined from running a sample of the scripts. These scripts are run to test the system for all possible user actions whenever an update is made to the system or an upgrade is applied.

Each script is unique:
  • The number of steps is determined by the script requirements, though fixed for that script once determined. In other words, different scripts have a differing number of steps.
  • No step has a determined likelihood of failure. The scripts have varying likelihood of failure as well.
  • Failure reasons are categorized into nine "conditions."
  • The scripts are placed into three types, based on their criticality.
  • If more than one failure reason occurs on a single script, all reasons are captured.
  • If a step fails, it might prevent the rest of the script from being run, if the data from that step is used in subsequent steps... but, in some cases a portion of or all of the steps remaining might be able to be run anyway. The person running the script must determine that subjectively. In any case, if a single step fails, then the script itself is determined to have failed.

What the project manager wants is to be able to run, let's say, 100 scripts and then determine an expectation regarding running all 6000. But, this structure has so many variables that I have no idea what to tell him.

Any help is GREATLY APPREACIATED !!

Tom