LASSO

[noetsi]

I know hlsmith does this. How do I do LASSO in SAS.

 

[hlsmith]

I believe you can only do it with a continuous outcome in SAS if you don't have the DataMiner platform. Is that what you want to do? I haven't used SAS, but I am sure I can help. I have done it with binary outcome a bunch in R.

 

[noetsi]

The DV will be both categorical and continuous. I will use it only for the continuous ones.

 

[hlsmith]

It is a process for feature selection, so model building. Ideally, you use it for figuring out terms using a training set and then fit that model to a holdout set. Do you have enough data to do that? You dont have to do it like that, but it drastically helps for generalizability.

If I get time this weekend i will see about setting up a n example using work.heart set.

 

[Buckeye]

I've used lasso in the caret package (in r) if it's any help. I've used it in the context of predictive modeling and the regression coefficients shrink to 0. Kind of plays a role in variable selection.

 

[noetsi]

I probably will have enough data to do a training and hold out set. I don't know how you split this with cross sectional data.

 

[noetsi]

My goal is it to chose variable since stepwise is not a good system and I rarely have theory in my analysis to build on.

 

[Dason]

Are you just looking for optimal predictive ability or do you want explanatory abilities as well

 

[noetsi]

For cross sectional data I don't use models to predict, just to show relationships between variables.

For time series I think there is agreement that multivariate approaches have little value to predict relative to univariate models [or at best add little] so I will not be using multivariate approaches to predict, only to show relationships between variables.

 

[hlsmith]

Running out of time this weekend. I am sure you saw this:

of note, you can just jam all variables into model and it will identify the true/best model. It just optimizes given presented terms. So for example, if you put effects of both covariates and the outcome, the model doesnt know this and will treat them as a covariate.

 

[noetsi]

thanks hlsmith. Is it considered valid to use lasso the way your brought up (I ask because using stepwise to do this is obviously attacked a lot in the research).

How in the world does it know the best model Spend lots of time reading the theoretical litature?

 

[Buckeye]

Lasso imposes a constraint on the beta coefficients. This is a modification of the typical least squares optimization problem.
See this wiki link under "Geometric Interpretation". https://en.wikipedia.org/wiki/Lasso_(statistics)

In this modified optimization there is a tuning parameter, lambda. When lamba = 0 we get the ols beta coefficients. But sometimes lambda is tuned such that the betas are exactly zero and this is due to the constraint region. Basically, lasso is a trade-off between bias and variance. We trade more bias for less variance in our predictions. I think ridge regression can be used to mitigate multicollinearity.

 

[hlsmith]

LASSO will set terms to zero, ridge regression wont. It will just shrink them down really close.

 

[noetsi]

John Fox argues that ridge regression generates biased parameters which is a heavy trade off for dealing with Multicolinearity.

 

[noetsi]

". It is necessary to standardize variables before using Lasso and Ridge Regression. Lasso regression puts constraints on the size of the coefficients associated to each variable. However, this value will depend on the magnitude of each variable. The result of centering the variables means that there is no longer an intercept. This applies equally to ridge regression."

If you get rid of the intercept how do you deal with reference levels of the dummy variables?

 

[Dason]

John Fox argues that ridge regression generates biased parameters which is a heavy trade off for dealing with Multicolinearity.

As does LASSO. But I don't see dealing with multicollinearity as the main reason one would use one of these methods.

 

[noetsi]

I agree dason I was responding to a previous poster who mentioned the use of ridge regression to deal with multicollinearity a strategy some support.

Why would one want to use lasso to chose variables when you know it biases the results. Biasing the results of the parameters would seem to be the cardinal sin of regression. Nothing is worse.

 

[Dason]

Because it can improve your predictive ability. https://en.wikipedia.org/wiki/Bias–variance_tradeoff

 

[Buckeye]

There is a bias-variance trade off in any estimator of any parameter. The sample standard deviation (MLE) is a biased estimate of the population standard deviation. That doesn't stop us from using it.

 

[noetsi]

One thing I am not clear on is if you chose the variables with LASSO, then run those variables through the OLS model. Do you need to check regression assumptions when running LASSO (like normality, equal error variance).

This is on page 5 of this article about LASSO. It made me wonder....

"The following are OBSOLETE!• When testing interactions, main effects should also be included"
http://www.misug.org/uploads/8/1/9/1/8191072/bgillespie_variable_selection_using_lasso.pdf