Error distribution and non linear models.

Ok, I am reformulating this question because it started rather silly. As I always point out, my background is not statistics so please excuse my ignorance.

I am currently experimenting with non-linear models. I would like to know how does the accuracy of (real) predictive error estimation is influenced by the size of the testing set (that is the sample size). Since I am estimating a real predictive error by drawing a Root Mean Squares of the Errors and also Mean Absolute Error on my test set, I would guess the bigger the test set, the more accurate the estimation, is this right?

Also, can I gain some insight on how accurate my error estimation is by looking at the (absolute) error distribution? Is there a statistical test that can tell me if my error estimation is accurate? I want to know because I am comparing different models and I would like to know if I am really comparing values that are close to the real error and not just estimations that are way off because of chance.

Thanks in advance!

PS: The absolute error distribution looks like a half gaussian, but this is lost when the test set is small.
Last edited: