Bootstrap procedure to test difference between correlation coefficients

[smndpln]

Hi everyone,

I have two correlation coefficients (r1 and r2), obtained within the same sample (20 subjects). My aim is to test it they are significantly different.
r1 is the correlation between a neurophysiological parameter and a behavioural parameter in condition A; r2 is the correlation between the same neurophysiological parameter and the same behavioural parameter in condition B.

I was thinking to apply a bootstrap procedure for each condition, in order to obtain two distributions of correlations. Then, I can simply run a two-sample t-test, to test for a significant difference.
My questions:
1) Does this procedure seem reasonable to achieve my purpose? (test if r1 and r2 are significantly different);
2) There is a way to decide the number of iterations, or is it totally arbitrary? (for example..I can go with 1000?).

Thanks in advance,
Simone

 

[hlsmith]

Hmm, I am not an expert in this area but this seems off. Your general intentions are fine, though I don't think you need to run a t-test. Can you describe the groups and variables in more detail? It seems like you are saying they are the same variables and people, so I am not seeing the distinction between the two r's. And why can you use the same sample to calculate both parameters?


A thousand resamples is usually fine. You can also run say 500, 1000, 5000, and 10000, then see if they all converged to about the same distribution (that is the rule of thumb). If you think about the resampling and then doing a t-test, its outcome can change based on the number of resamples. So you can decrease your standard errors more and more just be taking more resamples. That doesn't seem right to use a your deciding factor.


What is the explicit question you are trying to answer and what is your hypothesis about that question?

 

[smndpln]

Hi hlsmith, and thanks for the reply.

Actually, the two r's are different because they come from different condition in an experimental paradigm. So I have r1, which is the correlation between reaction times in the negative condition and brain connectivity in the negative condition, and r2, which is the correlation between reaction times in the neutral condition and brain connectivity in the neutral condition.

The explicit question is: are the two correlations different? Hypotesis: yes, at least for the connectivity in specific brain regions.

P.S. Probably, you're going to think this is a job for an ANOCOVA with separate regression slopes. Anyway, I noticed that ANCOVA is likely to accept null hypotesis with small sample sizes. Also Cohen's d may be useful, but does not give a statistical significance.

 

[hlsmith]

I believe you can do this, however using the following stipulations:


-Sent your seed for the bootstrap resamples and run both correlations on the same data sets.


-Find the differences between the two calculated r's, so match them based on the same resamples (pairs) and subtract.


-Run a paired t-test.


Though, as I mentioned earlier the number of resamples is unbound. So, I guess you end up powering to the level you are comfortable with. So you could do 50 our 10,000, which the later would end-up being an pretty steep distribution, but could get you close to the truth, you may try not to report the p-value, but an effect size. This would also benefit from graphing a final histogram of the correlations as well as their differences.

 

[katxt]

You probably want to keep the fact that the data comes from the same subjects. Here is a suggestion.
Put the data in columns with the subject as an index column. Find the two correlations and find the difference.
Resample the rows with replacement. This is really just resampling the subjects. Record the new difference. Do this 1000 times, or 2000 it only makes 10 seconds difference. Find the percentile that ZERO is. If this is over 0.5 then take it away from 1. This is a one sided p value. Doubler it for your two sided value.

 

[smndpln]

hlsmith, I think you are right and maybe the estimation of effect size is better. I'm using the classic Cohen' d for this.