Need help in analysing model results

#1
I have a question regarding testing of two sets of data. The data are three measured flow rates and corresponding three simulated flow rates for three storms. The three storms are independent and are not related in any way. Can we test if there is any significant difference between the two sets of data (measured and simulated).

Alternatively, if there any test to check if the model was successful in simulating the flow rates for the three storms?

The table of results has been attached.

Thanks in advance for all your help!

Thanks TheEcologist for your help.

Deb
 

TheEcologist

Global Moderator
#2
I have a question regarding testing of two sets of data. The data are three measured flow rates and corresponding three simulated flow rates for three storms. The three storms are independent and are not related in any way. Can we test if there is any significant difference between the two sets of data (measured and simulated).

Alternatively, if there any test to check if the model was successful in simulating the flow rates for the three storms?

The table of results has been attached.

Thanks in advance for all your help!

Thanks TheEcologist for your help.

Deb
Hi Deb,

I'm glad you chose to start a thread this, I'll start off with my advice in our PM.

me said:
Regarding your problem:
You can use normal inferential/descriptive statistics to test your simulation results. People often do this, especially when your model has stochastic components you often have no choice. I cant give you any more detailed advice as you haven’t supplied a description of how your (simulated) data looks like (nor what kind of statistic you are looking at; mean, median, trend ect).

Lastly Model validation (checking if your model is “successful”) again often implies using inferential/descriptive statistics. You can think of correlating actual data to simulated data, but again the choice of what to do boils down to how your data looks like.
Now the problem with your data is that you have too little points to try any inferential stats (especially a correlation would have perfect here). However you can’t change the amount of storms you’ve observed (or can you?).

One option you can do is run more simulations,
however another popular measure that you can use with only a few points (2 or more) is Keyfitz’s Delta (1968). It’s an (accepted) method (especially in population modeling) of quantifying the distance between two vectors:

Delta(observerd,simulated) = ½ * sum of all: abs(observed - simulated)

It's actually a standard measure of the distance between probability vectors, when used on prob. vectors the maximum distance is 1 and the minimum 0.
Now you will know best if it is suited for you.

The ref is: Keyfitz, N. 1968, Introduction to the mathematics of population. Addison-Wesley, Reading, Massachusetts, USA.
 
#3
any additional tests?

Thank you very much, TheEcologist for your input. I will do the Keyfitz's delta test for my model. Are there any other tests that might be applicable to my situation, like Tukey's test or F test?? Thanks again.
 

TheEcologist

Global Moderator
#4
Thank you very much, TheEcologist for your input. I will do the Keyfitz's delta test for my model. Are there any other tests that might be applicable to my situation, like Tukey's test or F test?? Thanks again.
Well it depends,

Are the only statistics you have the ones you supplied in this post or are they calculated from a set of data per storm/simulation (e.g. are the 3 pairs of datapoints means or any other summary statistic)?
 
#5
they are not summary statistic

TheEcologist, thanks again for your reply. These flow rates were simulated by the model for the individual storms, and they are not summary statistic like means. I know, many of the statistical tests will not be applicable here. I just wanted to make sure.
 

TheEcologist

Global Moderator
#6
TheEcologist, thanks again for your reply. These flow rates were simulated by the model for the individual storms, and they are not summary statistic like means. I know, many of the statistical tests will not be applicable here. I just wanted to make sure.
You know you can always just present / try to minimize the percentual error.

100*(sqrt((simulated-observed)^2)/observerd)

Simple is often better you know.
 
#7
Thanks a lot TheEcologist! A last request.....

Can you give a reference for the expression you cited or for any similar expression, please!

100*(sqrt((simulated-observed)^2)/observerd)

Many thanks....!

Deb.