pvalue Model Terms

[hlsmith]

So if I simulate a bunch of random independent variables and select one as the DV and the rest as IVs, I should expect 1 in 20 will be significant at p value </= 0.05 as the number of terms increases toward infinity, right? By chance, the number of significant terms may be higher than 1\20 at first but should settle into 5% after awhile. This seems like the Law of Large Numbers, right? Am I missing anything?

 

[Dason]

Nope - not missing anything. As long as you're simulating properly and using an exact test the p-value distribution will be uniform. So if you choose .05 for alpha then that will be the true probability of rejecting the null and thus the value your rejection rate will converge to under the law of large numbers.

 

[hlsmith]

Ah you got my attention. Exact p value test. I know Fisher, exact logistic, I think a complete permutation test may be exact. What would it be for multiple linear regression, with continuous vars? Just adding some exact line of code?

 

[Dason]

If you're simulating according to the usual assumptions then it should be exact already.

 

[hlsmith]

@noetsi I simulated this. Boom.

 

[hlsmith]