As old time posters here know I am a data analyst rather than a statistician. So in December 2013 I created a budget prediction tool based on basic time series, historic spending, and essentially pure logic since were making a major process change no one had done before.
Since then I made two changes in the prediction system because of logic or empirical problems with some of its input. Recently we got the 11th month of data (so we know how much we spent in the last year). The fixed budget is not bad, about 5 percent off. But the budget prediction model that just can't be right (the original one) is perfect. It misses spending by less than a half percent predicting 15 months in advance (which given the complexity and variability of our processes is impossibly correct).
So the wrong model gets it perfect and the right model based on logic and empirical analysis of certain key features of the original model gets it wrong. Which is...really annoying. To make it worse the May spending, the last month, had results that are a total outlier. And they canceled out, nearly perfectly, our error in the original model.
It is like someone knew what our error would be and deliberately made spending for a large state agency such that it corrected that error.:shakehead
Since then I made two changes in the prediction system because of logic or empirical problems with some of its input. Recently we got the 11th month of data (so we know how much we spent in the last year). The fixed budget is not bad, about 5 percent off. But the budget prediction model that just can't be right (the original one) is perfect. It misses spending by less than a half percent predicting 15 months in advance (which given the complexity and variability of our processes is impossibly correct).
So the wrong model gets it perfect and the right model based on logic and empirical analysis of certain key features of the original model gets it wrong. Which is...really annoying. To make it worse the May spending, the last month, had results that are a total outlier. And they canceled out, nearly perfectly, our error in the original model.
It is like someone knew what our error would be and deliberately made spending for a large state agency such that it corrected that error.:shakehead