The bootstrap method for medical modelling

[Lindsey]

I am part of a team creating a model for paediatric hospitalised patients and would appreciate some help. We are looking to produce an accurate model for weight estimation, using input variables of gender, age and mid-upper arm circumference. This would be helpful as some children arrive in hospital too unwell for a set of scales but weight is required for medication doses etc.

So far we have used multiple regression analysis and come up with a fairly good model. We produced our model using data from a large cohort of UK children and cross-validated it in large cohorts of Greek and Dutch children. However, a reviewer has suggested we use bootstrapping to produce a better model.

We have a few questions about this and would really appreciate some help.
1) Should we use a combination of the UK, Greek and Dutch data to produce the model?
2) Do we need to do further validation / cross-validation once we have the model from bootstrapping? And if so which data should we use for this?

Thanks very much in advance

Lindsey

 

[Dason]

Would it be possible to introduce any other variables? My guess is that if you could get an approximate height measurement you could improve any model significantly

 

[hlsmith]

Who do you hope to generalize the results to? That is your first question in my opinion. Are data from hospital patients or community non-ill children?


How do you know that children who can't be weighed are comparable to the sample you are using to come up with weights


How is it possible you can't weigh a peds patient? I can't think of scenarios where this would be true and if their was some type of NICU patient that can't be moved, that patient would likely not have a comparable person to create a prediction rule from.


What is your age range? Do you have children and adolescents?


Also, what was their exact comment?? Just saying "bootstrapping" is pretty vague. They may just want bootstrap confidence intervals.

 

[Lindsey]

Hi

Thanks very much for the responses.

We do have height data for all patients. This has improved the model a little but not significantly with the method we have used so far (multiple regression with quadratic input variables). We feel measuring height is difficult in a resus situation which is why we are not keen to use height, unless the advantage in terms of accuracy is large, but we will present these results in the paper.

The results should be applicable to children attending ED. This will be a mix of children with co-morbidities and previously well children (who may have presented due to trauma etc). Our sampling data reflects this, as we have included a cross section of all children attending ED. We have sampled children well enough to be weighed only (as we required weight to produce our equation) but these children should be as similar as possible to those presenting too unwell to be weighed. There is no reason to believe the two groups should be different.

Commonly, paediatric patients cannot be weighed when they arrive in hospital very unwell eg. trauma or peri-arrest situations. Estimated weights are routinely used in these situations, but the current equations are inaccurate (according to numerous literature sources and as we have shown with our data). The purpose of our equation is for the paediatric cohort, rather than NICU patients (NICU patients almost never go home and then return to ED unwell). Even when NICU patients are extremely unwell, every effort is made to obtain a weight in the delivery suite for prognostic purposes as well as for drug calculations (those < 500g are very unlikely to survive). This is easier in newborns due to their small size and lack of 'down time' before medical support arrives. We do not suggest weight should be estimated here. We have included other children in the neonatal period (< 1 month old) who have made it home and returned to ED. These are not extreme prems etc. as this population will not have made it to the community and will be resident in NICU.

Our age range is 0.1 months to 17.9 years.

Our comment regarding bootstrapping is:
"Development of model on one cohort(UK), then validation on another cohort (danish) and another (Greece). From the manuscript, it appears the authors think the populations are different - hence the value in comparing the three groups. but if the groups are inherently different, it is usually best to "trial and test" on one cohort, then re-validate on a different population. As it stands, you are at risk of overfitting one population, and being underfit on the validation cohort, and thus not achieving an algorithm for all children worldwide. Suggest grouping UK, greece and dutch cohorts together and develop/validate using a random sampling method (such as bootstrap or jackknife method). Secondary analysis that demonstrates this doesn't significantly impact the derivation and validation of the model would also be acceptable."

Thanks for the input. Any help appreciated!

 

[Lindsey]

Hi

Thanks very much for the responses.

We do have height data for all patients. This has improved the model a little but not significantly with the method we have used so far (multiple regression with quadratic input variables). We feel measuring height is difficult in a resus situation which is why we are not keen to use height, unless the advantage in terms of accuracy is large, but we will present these results in the paper.

The results should be applicable to children attending ED. This will be a mix of children with co-morbidities and previously well children (who may have presented due to trauma etc). Our sampling data reflects this, as we have included a cross section of all children attending ED. We have sampled children well enough to be weighed only (as we required weight to produce our equation) but these children should be as similar as possible to those presenting too unwell to be weighed. There is no reason to believe the two groups should be different.

Commonly, paediatric patients cannot be weighed when they arrive in hospital very unwell eg. trauma or peri-arrest situations. Estimated weights are routinely used in these situations, but the current equations are inaccurate (according to numerous literature sources and as we have shown with our data). The purpose of our equation is for the paediatric cohort, rather than NICU patients (NICU patients almost never go home and then return to ED unwell). Even when NICU patients are extremely unwell, every effort is made to obtain a weight in the delivery suite for prognostic purposes as well as for drug calculations (those < 500g are very unlikely to survive). This is easier in newborns due to their small size and lack of 'down time' before medical support arrives. We do not suggest weight should be estimated here. We have included other children in the neonatal period (< 1 month old) who have made it home and returned to ED. These are not extreme prems etc. as this population will not have made it to the community and will be resident in NICU.

Our age range is 0.1 months to 17.9 years.

Our comment regarding bootstrapping is:
"Development of model on one cohort(UK), then validation on another cohort (danish) and another (Greece). From the manuscript, it appears the authors think the populations are different - hence the value in comparing the three groups. but if the groups are inherently different, it is usually best to "trial and test" on one cohort, then re-validate on a different population. As it stands, you are at risk of overfitting one population, and being underfit on the validation cohort, and thus not achieving an algorithm for all children worldwide. Suggest grouping UK, greece and dutch cohorts together and develop/validate using a random sampling method (such as bootstrap or jackknife method). Secondary analysis that demonstrates this doesn't significantly impact the derivation and validation of the model would also be acceptable."

Thanks for the input. Any help appreciated!

 

[hlsmith]

I am guessing you have huge datasets? Also,do you have different models for different ages and genders (ie, BMI values for PEDs and adolescent populations)? I thought skin volumn was also used for dosings?

I agree with the reviewer, it would be like creating a prediction rule in an urban ed and assuming it would work in a rural center. I would pool all patients then pull out 60% to develop model and test on the remaining random 40%.