Time series opinion

#1
There is no time series theory here. It is based on the opinion of some experts who do time series which has worked reasonably in the past for me.

I have a data going back to 2014 by month. There were six models in the past all are ESM so the model structure for each in SAS is always the same (the algorithm is always the same). Recently I have decided to add additional models based on the theory, rule of thumb whatever you want to call it, that mixing additional models will improve the forecast.

What I have done in the past is select a hold out of the last 12 month and predict that with the six models, take the 3 models that best predict and then use them, with the whole data set including the hold out data set to predict the future.

That works with the ESM because the model itself (the algorithm) never changes. But the two models I am thinking of could change. One is an ARIMA model the other a ESM model that has 30 possible forms (I am using R tools to do this, auto.arima for arima).

So say I take the hold out data set and the new ESM or ARIMA model is one of the best predictors. I decide to add it to the model. Now the optimal Arima model (the pdq estimates) that predicts the hold data set might well vary than if I use the whole data set including the hold out data set rather than the data I did use.

I am not sure if I should use the original Arima model or estimate a new one with all the data.

I understand there is no theoretical answer to this. I am just asking opinions.
 

hlsmith

Less is more. Stay pure. Stay poor.
#2
Interesting question. I would fit a new ARIMA model - otherwise you are wasting data and projecting further into the future from where the model ends. You could always attempt to validate this process when you eventually collect more data in the future.

As a test, you could compare the MAPE, etc. in both models to the full dataset - I think it would be pretty intuitive that the new model will outperform the small series model given no major shocks are occurring.