pvalue Model Terms

hlsmith

Less is more. Stay pure. Stay poor.
#1
So if I simulate a bunch of random independent variables and select one as the DV and the rest as IVs, I should expect 1 in 20 will be significant at p value </= 0.05 as the number of terms increases toward infinity, right? By chance, the number of significant terms may be higher than 1\20 at first but should settle into 5% after awhile. This seems like the Law of Large Numbers, right? Am I missing anything?
 

Dason

Ambassador to the humans
#2
Nope - not missing anything. As long as you're simulating properly and using an exact test the p-value distribution will be uniform. So if you choose .05 for alpha then that will be the true probability of rejecting the null and thus the value your rejection rate will converge to under the law of large numbers.
 

hlsmith

Less is more. Stay pure. Stay poor.
#3
Ah you got my attention. Exact p value test. I know Fisher, exact logistic, I think a complete permutation test may be exact. What would it be for multiple linear regression, with continuous vars? Just adding some exact line of code?