two-sample assuming unequal variances - too sensitive?

[confused_pleasehelp]

Hi there,

I'm comparing image values in a tumour region before and after treatment. I'm trying to see if the value of the pixels is significantly different after treatment. The histogram distributions look like this (for one tumour):


I assumed using the 'two-sample test assuming unequal variances' option in excel would be appropriate here, but it seems overly sensitive - as in, it's saying pretty much almost all of them are statistically significant differences (except for a few). This would be fine, except that it says this is p<0.000001:

And to me, that just.... seems wrong? As in, those two distributions are almost identical.. it seems dishonest to call that change statistically significant.

Please help, should I be using something else? Or changing what alpha threshold I consider significant?

 

[Karabiner]

The test concerns the question whether we can reject the Null hypothesis.

Here, the Null hypothesis is: "the mean difference between the populations
from which the two samples are drawn is exactely = 0.00000000000..."

This is rejected if the sample data seem implausible, assuming the null
hypothesis.

If you have a huge sample size, then you can reject such a Null hypothesis
even if the sample man difference is small. Mind that "statistically significant"
has nothing to do with important, relevant, or large. Just with the
question "might the difference be exactely zero in the population, or not?"

You should additionally consider the 95% confidence intervals for each
mean, and the 95% confidence interval for the mean difference, to get an
idea of how large or small your sampling error is.

With kind regards

Karabiner

 

[confused_pleasehelp]

thank you, you are a hero. it's definitely an issue of having a very large sample size. I'm now wondering how to report these p values as it's very common in my field to just report p<0.0001 but it feels disingenuous. It is still very low p value even after adjusting the 95% confidence interval to alpha=0.99

 

[hlsmith]

If it is a pre post design, you don't have two independent samples, right?

 

[confused_pleasehelp]

yeah it's pre-post, but the tumour is slightly different size after so it's not like each of the thousand pixel values from before can pair to a pixel value after - does that make sense? what test should I use @hlsmith?

 

[Karabiner]

Ops. I missed that part. If you measure exactely the same subjects twice,
then you have to perform a dependent measures t-test (which ist essentially
a test of whether the intra-individual differences have a mean of zero).

With kind regards

Karabiner

 

[confused_pleasehelp]

Ah, except there are a different number of data points in the "after" - it's not a 1:1 matching... I just tried to do a paired t-test but it said the number of samples has to be the same. Is there a way to perform dependent measures t-test without each sample in 'before' needing to match up with a respective sample in the 'after'?