External Validation of Cox Model

[Jyde]

I have two data set (training and validation) for building and validating a Cox model.

With the training data set I fitted a cox model using stepwise selection method.

The significant variables in the model were the only variables included in the validation model. Is this the right approach?

While validating the model I realized that the variables are not significant in the validation model and also the assumptions of the cox model do not hold (I checked the assumption on the validation data). Should I ignore the fact that the variables are insignificant and go ahead in making corrections for the problem with model assumptions in validation data?

Thirdly,in both training and validation data I have a variable 'treatment' with three groups. In training the groups are Standard, New drug and mixture, while in validation data the groups are Standard, New drug and X (is a treatment which is different from mixture in training data). Is it right to include this variable in both model or should I eliminate the groups that are not match; mixture from training data and X from validation data or should I work with it like that? I am not sure how this affects my analysis.

Thanks for your responses.

 

[hlsmith]

I am guessing the you did not randomly split the data into a training and validation set?

 

[Jyde]

No I did not. The data were collected from two different experiment.

 

[hlsmith]

Systematic bias (selection bias, measurement bias, and possibly confounding) along with chance is gonna mess with you. Yeah, you usually take the coefficients from the test set and score the validation set and see how accurate the model is in the validation set. So you wont rerun the model. I don't use PHREG regularly, so I can't remember the assumptions or if you would care too much about them given the above approach.


What assumptions seem to be broken?

 

[Jyde]

Systematic bias (selection bias, measurement bias, and possibly confounding) along with chance is gonna mess with you. Yeah, you usually take the coefficients from the test set and score the validation set and see how accurate the model is in the validation set. So you wont rerun the model. I don't use PHREG regularly, so I can't remember the assumptions or if you would care too much about them given the above approach.


What assumptions seem to be broken?

Proportionality. Though I can stratify the affected variable but then I would loss information on it. I could introduce an interaction but then I don't want to complicate my interpretation.

 

[hlsmith]

Hmm. I have not cross-validated for PHREG. You could go back and make this change to the original model as well, but then you are letting validation influence testing and it may not even effect the original model, but to take up degrees of freedom.


I think it just comes back to the differences in the datasets. This is an interesting problem, though I don't know enough to move you forward. Can you explain the differences between the datasets or anything that may make them unique. Is it possible to merge them or use a shrinkage approach. However, these may not help you out given the problem.

 

[Jyde]

Hmm. I have not cross-validated for PHREG. You could go back and make this change to the original model as well, but then you are letting validation influence testing and it may not even effect the original model, but to take up degrees of freedom.


I think it just comes back to the differences in the datasets. This is an interesting problem, though I don't know enough to move you forward. Can you explain the differences between the datasets or anything that may make them unique. Is it possible to merge them or use a shrinkage approach. However, these may not help you out given the problem.

That was my thought (letting validation influencing testing)!

The only difference about the data set is that one of the treatment arm in validation data set contains a different treatment from we have in testing data set. Though out of curiosity I decided to remove observations from the different arms and used treatment arms that were common to both data set still it couldn't fix the problem.

 

[hlsmith]

So test dataset has 2 treatments and validation has 3 treatments? Were treatments randomized?

 

[Jyde]

So test dataset has 2 treatments and validation has 3 treatments? Were treatments randomized?

test dataset has 3 treatment as well. The only difference is that the third treatment differs in both dataset.

Yes treatment was randomized.