How to determine which factors/interactions to remove from a model?

How do you determine which factors/interactions to remove from a model? Should I be dropping higher-order interactions from my model if they aren't significant and aren't increasing my R-square/F-ratio? Asking specifically for an ANCOVA that I'm running (two categorical IV's and one covariate) but am curious about for other models as well (for example a generalized linear model). Can we run a number of models and decide what to include based on on AICc? In the case of GLM's, can we simply rely on selection methods (i.e., Lasso, elastic net, etc.) to remove factors/interactions? Or are we going to have to use our background knowledge of the system to determine what is important (and should be included) and what might simply be "nuisances" (and can be removed if they're not adding anything to the model)?


Not a robit
First and foremost is using background knowledge. Some times you will leave one in not based on significance but known rationale. Second all of the criteria you mention can be used along with likelihood tests. Don't get to overly married to strict alpha cutoffs if you have background knowledge or it seems close to alpha level. Also graphing the relationship can be a good idea to look at unparell slopes.

For regularization approaches don't forget to use group option to force the contributing terms into the model. So run with grouping options and without and compare results.