split halves test

I just read there was something called a split halves test: If you get a 95% or so significance that means youll still be wrong in 5% of the cases. But if its right in 2 samples its probably true.

So you supposedly you can split your sample into two halves calculate the same thing for those 2 and voilà if both are statistically significant youre probably right...the only problem with this being that the significance drops slightly.

The site looked pretty nice and all, but....if this was true and it allowed to make a better prediction couldnt I just split the halves in two more halves and so on?:). I assume the statistical significance level drops proportionally so that in the end this splitting in halves method doesn't help you, at all?