Which method to compare models with truth?

[seanelvidge]

Hi all,

Edit: First I must say sorry for the ridiculous spelling mistake in the title! I wish I could go back and change 'two' to 'to'.

I have only just come across this forum while searching the net for some info - it looks fantastic so I'm hoping I'll get plenty of use out of it in the future! If I have posted this in the wrong area I apologise, and will happily delete it and reopen it in the correct one.

The problem: I have a large set of "truth data" i.e. data that I consider to be correct, and I have two other sets of data which are estimates of said data. The methods used to derive these two sets are different. What I am looking for is a way to say if one is statistically significantly better than the other.

Currently I am creating two plots, of the mean plus and minus one standard deviation of the errors (i.e. estimated set 1 - truth and estimated set 2 - truth). From these plots I can see clearly that one method is better than the other (i.e. the mean error and standard deviations are smaller) but I am looking for a method of determining this statistically.

I'm not looking for anyone to walk me through the process, I am more than happy to go and look up how to do the process - just not sure which one to do! I thought about doing an F test, but I can't make my mind up if that is really suitable for what I want to do or not.

Thanks in advance for any help anyone can provide!

Kind Regards,

- Sean

 

[Link]

Re: Which method two compare models with truth?

I don't understand your explanation of the data.

In statistics, there is a "true" assumed distribution. From this true distribution, there are samples of data we use to (sometimes) estimate parameters from this distribution.

What you're saying is that there is true "data". If this is correct, then it should be the only data that you used. I'm confused by what you mean when you say data that are estimates of the true data.

Continuing, it appears currently that you're interested in determining which of your two empirical data is closer to the original data. There are a few ways to do this. One way is to compare the means from each distribution to see if they're statistically significant.

HTH

 

[seanelvidge]

Re: Which method two compare models with truth?

Hi,

Thanks for your quick reply. Sorry that I wasn't clear enough in my description! Let me try and clear up the confusion:

I have two programs which estimate a system in real time, which changes constantly, and I want to tell which of these two programs is better (or at least statistically significantly better). The way I have done this is by taking a previous time period (which I now know the exact state of the system at that time - this is my "truth" data) and am trying to compare it with my programs estimate of that that time period.

So I can't use the the true "data" in the running of my program because I only have the truth in hindsight, but having it does mean I can compare my existing models to see which is better. Does that make more sense?

But I think you got the jist of what I am trying to achieve since I want to determine which of my empirical data is closer to the original, "truth" data. Comparing the means seems like a good idea to me (again I thought of using a T Test for this, but wasn't sure if that was allowed since I do not know that the errors are Gaussian), can you suggest any tests to compare if the means are statistically significant? It would also be good to find a test which takes into account the standard deviations, since sometimes it seems (from visually inspecting the plots) that the means are very similar, yet the s.d. can be much larger in one than the other.

Once again, my thanks.

- Sean

 

[TheEcologist]

Re: Which method two compare models with truth?

It will certainly depend on what your model output is.

For distributions functions, a natural measure would be Kullback-Leibler.
For sample distribution you could use the Kolmogorov-Smirnov
You could also calculate the proportional error; esitmate/truth and test if the proportional errors differ.

So it depends on what it exactly is being returned, and on what defines a better system.

 

[seanelvidge]

Re: Which method two compare models with truth?

It will certainly depend on what your model output is.

For distributions functions, a natural measure would be Kullback-Leibler.
For sample distribution you could use the Kolmogorov-Smirnov

Thanks for your input. Although I am confused by what kind of test I should do with my outputs!

All of the data (truth and estimates) are simply two-dimensional arrays of numbers. Which I can plot against one another (see the picture for a small example (of actual results!))

The lines are just piecewise fit on to the data points - I could just plot the points, but the number of them usually make these plots more viewable.

In this situation could you advise me further on what would be a good test. The idea of calculating the proportional error is an excellent one but could I use this to tell if one estimate was significantly better than another?