Wrong model is right (annoying)

[noetsi]

As old time posters here know I am a data analyst rather than a statistician. So in December 2013 I created a budget prediction tool based on basic time series, historic spending, and essentially pure logic since were making a major process change no one had done before.

Since then I made two changes in the prediction system because of logic or empirical problems with some of its input. Recently we got the 11th month of data (so we know how much we spent in the last year). The fixed budget is not bad, about 5 percent off. But the budget prediction model that just can't be right (the original one) is perfect. It misses spending by less than a half percent predicting 15 months in advance (which given the complexity and variability of our processes is impossibly correct).

So the wrong model gets it perfect and the right model based on logic and empirical analysis of certain key features of the original model gets it wrong. Which is...really annoying. To make it worse the May spending, the last month, had results that are a total outlier. And they canceled out, nearly perfectly, our error in the original model.

It is like someone knew what our error would be and deliberately made spending for a large state agency such that it corrected that error.:shakehead

 

[hlsmith]

They typically say go for the simpler model. Is it incorrect or just simpler. Well by chance even a broken clock is correct twice a day!

 

[noetsi]

None of these models are really simpler. The original model had an inflation factor in it the later one does not (because we inflated one trend and did not inflate another with this inflator which makes no sense). This is why the latter model is logically more correct. It does not have a "correction" factor that should never have been in the model. Secondly they make different assumptions about how many customers will enter service (different, but not more or less complex).

The inflator reduced spending projections in practice and in fact we spent less. That is why it is correct I imagine even though the assumptions behind the inflator were wrong. We reduced spending by a constant with the inflator and because spending was less in fact this worked even though the logic behind the inflator was wrong.

So what I really need to figure out is why we spent less. But many many many things could have caused that. But sometimes I guess (the lesson is here) you can be right for the wrong reasons. One error canceled out another.

 

[rogojel]

hi noetsi,
maybe you could feed the models with sensible random data as in bootstrapping, to get the range of the predictions for both models? Probably the perfect prediction is just a lucky shot.

regards

 

[noetsi]

I actually think it structurally corrects for other errors in the model, but how and what it corrects is totally unknown to me.