First time poster: question on statistical analyses of treatment groups

#1
Hello all,

I wonder if you can help me understand the following comment I received on a study:

So, there are 3 patient groups (A, B and C):
Group A receives treatment X
Group B receives treatment Y
Group C received treatment Y followed by treatment Z.

Results (the numbers are made up):

Group B patients (n=50) have a median survival time of 36 months (range 11-49 months)
Group C patients (n= 31) have a median survival time of 43 months (range 21-69 months)

Statistical analyses showed that the median survival times of group B&C are not statistically different (P=0.06). Since they are not statistically significantly different, I want to combine groups B+C for further analyses. The feedback I received on that was: Since patients in group C only received treatment Z only if they survived long enough to do so and the survival range in group B is lower than group C, I have to exclude all group B patients that survived less than 21 months (i.e. 11-21 months survival). Is this correct or can I refute this argument?

Thanks a lot and I hope this is not too basic

Jan
 

Karabiner

TS Contributor
#2
Since patients in group C only received treatment Z only if they survived long enough to do so
Was this the actual the case? How exactely were patients allocated to groups?

And what are the further analyses you want to undertake (with which goal), so that
you think you should combine B and C?

With kind regards

Karabiner
 
#3
Was this the actual the case? How exactely were patients allocated to groups?

And what are the further analyses you want to undertake (with which goal), so that
you think you should combine B and C?

With kind regards

Karabiner
Thank you for getting back to me!

Regarding your first question:
Well, yes, but in some cases they had progressive disease during treatment Y, so treatment Z was started earlier in some than in other patients.

Regarding your second question:
Patients were not randomised. Basically the allocation was based on clinicians' and patients' preference. I know this is not ideal...

Regarding your third question:

I want to combine groups B+C because patient numbers in groups B+C are quite small. Once combined I want to compare that joint BC group to group A with regards to a potential prognostic factor:

Group A
a) prognostic factor positive
b) prognostic factor negative

Group BC
a) prognostic factor positive
b) prognostic factor negative

Thanks for your help!
 

Karabiner

TS Contributor
#4
So, as far as I understan,d you want to compare patients in treatment X with patients in treatment Y,
i.e. alls patients from B and C. There was no need to make a statistical comparison in order to justify that.

With kind regards

Karabiner
 
#5
So, as far as I understan,d you want to compare patients in treatment X with patients in treatment Y,
i.e. alls patients from B and C. There was no need to make a statistical comparison in order to justify that.

With kind regards

Karabiner
Thanks again.

How about if I want to compare group B vs group C for the potential prognostic factor:

Group B
a) survival in prognostic factor positive patients
b) survival in prognostic factor negative patients

Group C
a) survival in prognostic factor positive patients
b) survival in prognostic factor negative patients

I.e. comparing group "Ba vs group Ca" and "group Bb vs group Cb".
Would the comment by the reviewer than hold true? (Since patients in group C only received treatment Z only if they survived long enough to do so and the survival range in group B is lower than group C, I have to exclude all group B patients that survived less than 21 months (i.e. 11-21 months survival)).
 

hlsmith

Less is more. Stay pure. Stay poor.
#7
Your biggest issue is the non-randomization. Without controlling for background covariates that may be associated with treatment assignment and outcome (i.e., confounding by indications) you can't identify treatment effects. Your small sample size likely prohibits you from successfully addressing this issue due to issues related to power and a positivity violation - a lack of patients with comparable overlap in background characteristics.

The idea of merging group B and C together would be less than desirable. Basing a decision on p-value is always terrible - is there really no difference between the groups or did you just fail to reject the null given your small sample sizes. Establishing non-superiority of a treatment is a formal process not based on a pvalue from a underpowered comparison. Also, did your analyses control for multiple comparisons? The reviewer is correct on their comment about threats to conclusions - especially given the nonrandomization.

Your best and most prudent action is to not report any of these results and acknowledge the limitations - trying to find something in your data that wasn't stated a priori would be unethical and only contribute towards the replicability crisis. Look up HARKing and its impact on science.

Thanks and welcome to the forum.
 
#8
What exactely do you mean by this? Which comparisons do you intend to make?

With kind regards

Karabiner
See the picture on what I want to test for:
1633702801525.png

Would the comment by the reviewer hold true? (Since patients in group C only received treatment Z only if they survived long enough to do so and the survival range in group B is lower than group C, I have to exclude all group B patients that survived less than 21 months (i.e. 11-21 months survival)).
 
#9
Your biggest issue is the non-randomization. Without controlling for background covariates that may be associated with treatment assignment and outcome (i.e., confounding by indications) you can't identify treatment effects. Your small sample size likely prohibits you from successfully addressing this issue due to issues related to power and a positivity violation - a lack of patients with comparable overlap in background characteristics.

The idea of merging group B and C together would be less than desirable. Basing a decision on p-value is always terrible - is there really no difference between the groups or did you just fail to reject the null given your small sample sizes. Establishing non-superiority of a treatment is a formal process not based on a pvalue from a underpowered comparison. Also, did your analyses control for multiple comparisons? The reviewer is correct on their comment about threats to conclusions - especially given the nonrandomization.

Your best and most prudent action is to not report any of these results and acknowledge the limitations - trying to find something in your data that wasn't stated a priori would be unethical and only contribute towards the replicability crisis. Look up HARKing and its impact on science.

Thanks and welcome to the forum.
Thanks a lot for your valuable comments! :)

This is a retrospective study and yes, small numbers (underpowering) and non-randomisation are valid concerns, but they are not uncommon in our field. We certainly do not want to "produce" effects with bad statistics, nor do we want to claim an effect is absent when in fact the study is merely under-powered. If at the end of the day we cannot show an effect and honestly state this may be due to a type 2 error, then this may be acceptable for our purpose.
I just try to wrap my head around the comment by the reviewer about adjusting groups B+C for the lower ends of the survival range. Do you have any comments on that? Is this necessary? (see the picture in the post above and my first post for explanation)
 

hlsmith

Less is more. Stay pure. Stay poor.
#10
Are you performing actual survival analysis models with Kaplan Meier or survival plots. Not sure if the reviewers comments can be thought of as some type of quasi-competing event, this would require additional thought.
 
#11
Are you performing actual survival analysis models with Kaplan Meier or survival plots. Not sure if the reviewers comments can be thought of as some type of quasi-competing event, this would require additional thought.
Yes, we perform Kaplan Meier survival analysis.

I am not sure what you mean by "quasi-competing event" (?) but I do not think there is any "competition" (?)

My first post was maybe a bit confusing and unnecessarily complicated as I was not totally sure what the reviewers was referring to. So, let me try again:

Group B patients treated with Y (n=50) have a median survival time of 36 months (range 11-49 months)
Group C patients treated with Y followed by Z (n= 31) have a median survival time of 43 months (range 21-69 months)

Statistical analyses showed that the median survival times of group B&C are not statistically different (P=0.06).

The reviewer: Since patients in group C only received treatment Z only if they survived long enough to do so and the survival range in group B is lower than group C, I have to exclude all group B patients that survived less than 21 months (i.e. 11-21 months survival).

I would understand the reviewer if he/she asked us to exclude all patients from group B that died before actually having a chance to start treatment Z, but there are no such patients o_O

The way I currently understand the reviewer is that he/she wants us to equalize the starting point of the the survival range since there may be "bias" (??)

We are grateful for any further input from anyone on this forum!
Have a great weekend you all!




1633717722562.png
 

hlsmith

Less is more. Stay pure. Stay poor.
#12
You likely should draw something like the following out to convey your context, plus it could help convey information to the reviewer and eventual target audience.

But color code groups and show where Z was implemented in time in patients on Y.

1633726741089.png