# software speed

#### MikeMulligan

##### New Member
Not homework BTW ... I want to measure the processing time (performance) of some software under a set of fixed conditions. I have a means to log the start and stop of the processing. The only possible factors are some uncontrollable (noise) factors, e.g., the CPU and memory usage. Then there is random variation.

I know a little about typical design of experiments to determine the effect of factors. However, in this case, should I just see if there is any correlation of the uncontrollable (noise) factors on the results. Im not sure yet how to go about that --- determining if the noise factors affect the results over just random variation.

Its not clear to me how to determine how many runs I need to make for statistically valid results.

If someone could just point me in the right direction, I would appreciate it.

Thanks, Stanley

#### hlsmith

##### Less is more. Stay pure. Stay poor.
Is there a certain software you are interested in? Is there a certain function / procedure you are interested in? Does a certain sample size requirement need to be met?

Provide a few more details please.

#### MikeMulligan

##### New Member
Is there a certain software you are interested in? Is there a certain function / procedure you are interested in? Does a certain sample size requirement need to be met?

Provide a few more details please.
The software receives an input message at a specified rate, does some processing, and publishes an output message. The input and output messages are time stamped and logged. I need to measure the time from message input to message output, and to understand the variability in this measured time.

#### hlsmith

##### Less is more. Stay pure. Stay poor.
Some software has a temporary memory that gets built up if not purges or restarted, does your software have this?

So it will be the exact same input and function over and over again?

#### MikeMulligan

##### New Member
It is very possible that there might be some sort of "warm up" effect like this, and it could be complicated. The longer term behavior is of more interest. One thing I should do is determine if there is such an effect.

In general, I suppose I should be looking at time series analysis for each run for that, and then use some method to look at data across runs.

I am still not sure how to determine how many runs I need, even after I have some data to analyze.

#### MikeMulligan

##### New Member
So it will be the exact same input and function over and over again?
Yes, that is correct. Input messages will be sent at a specified rate.

#### hlsmith

##### Less is more. Stay pure. Stay poor.
Well as long as nothing is changing and the process is stable, perhaps some kind of convergence can be established based on some many iterations without substantive change.

#### MikeMulligan

##### New Member
I can use a one-sided t-test for this. Power curve can be used to estimate number or runs, as I have a good guess at the sample standard deviation.