Constant sum question - results analysis

#1
I ran a study with a constant sum question: participants had to divide 100 units (I called the units USD) between 5 possibilities. In other words, they had to bet on five possibilities: 1 object out of 5 has a property vs. 2 objects out of 5 have a property vs. 3 objects out of five have a property vs. 4 objects out of five have a property vs. 5 objects out of 5 have a property.

Next, I ran a paired samples t-test to compare the answers (or 'bets') on two possibilities, namely that 4 objects out of five have a property vs. that 5 objects out of 5 have a. property. A reviewer pointed out that I cannot use a paired-samples t-test because the answers are not independent of one another: your next bets are dependent on the previous ones as it all must sum to 100 (the bet on 5 is dependent on how much you already bet on 4). Which significance test should I use then? Please help!
 
#2
Am I right to assume that I should first transform the data to a log ratio and then perform a MANOVA + Hoteling t square test on CoDaPack? reference: https://ddd.uab.cat/pub/sort/sort_a2015m7-12v39n2/sort_a2015m7-12v39n2p231.pdf

I ran a study with a constant sum question: participants had to divide 100 units (I called the units USD) between 5 possibilities. In other words, they had to bet on five possibilities: 1 object out of 5 has a property vs. 2 objects out of 5 have a property vs. 3 objects out of five have a property vs. 4 objects out of five have a property vs. 5 objects out of 5 have a property.

Next, I ran a paired samples t-test to compare the answers (or 'bets') on two possibilities, namely that 4 objects out of five have a property vs. that 5 objects out of 5 have a. property. A reviewer pointed out that I cannot use a paired-samples t-test because the answers are not independent of one another: your next bets are dependent on the previous ones as it all must sum to 100 (the bet on 5 is dependent on how much you already bet on 4). Which significance test should I use then? Please help!
 

katxt

Active Member
#3
This sounds interesting but so far I cam't follow what you are doing. Can you give a more specific example, a show us what the data looks like, please
 

katxt

Active Member
#4
Further thoughts. It is true that the 4 group and the 5 group responses may be weakly (and negatively) correlated, but the paired sample t test is designed specially for responses which are not independent. So long as the differences are reasonably well behaved I can't see a problem, especially if you have plenty of subjects.
Alternatives would be Wilcoxon, or a bootstrap test for a difference of 0.
 
#5
This sounds interesting but so far I cam't follow what you are doing. Can you give a more specific example, a show us what the data looks like, please
Many thanks for your reply! Here's how it looks like with more precision and examples from the data (if more information is helpful I will be happy to provide it): participants (N= 600, which is enough according to a predictive power analysis) read an utterance for instance 'some of the five apples are red'. Next, participants had to bet by dividing the sum of 100 between six options: 0 objects out of 5 have a property vs. 1 object out of 5 has a property vs. 2 objects out of 5 have a property vs. 3 objects out of five have a property vs. 4 objects out of five have a property vs. 5 objects out of 5 have a property. For instance, one person bet 0, 10, 40, 40, 10, 0. I took the mean bets for all answers.
Here's an example graph, it presents the mean bets on each of the options, I compared with a paired samples t-test the mean bets on 4 and 5, but the Reviewer pointed out that this is not the right test for this sort of compositional data. Thus my question: which test is appropriate then?


1597393127568.png
 

katxt

Active Member
#6
OK. Thanks issta. In my opinion, there is nothing wrong with the paired t test for this. With 600 subjects it will give you p values that can be relied on, even if the data isn't perfectly normal. (Other forum members may disagree. We'll see.)
My view is to keep the analysis simple - the paired t test is easy to do, easy to understand and easy to explain. One problem could be convincing the editors that the reviewer is mistaken in their contention that the dependence of one score on another for the same person invalidates the paired t test. We know, in fact that the pairing actually takes that dependence into account. It would be interesting to see the actual correlation between successive groups.
A more pressing problem is that of multiple p values if you are doing multiple comparisons. How many tests do you plan to do? How do you plan to control the false positive rate? kat
 
#7
OK. Thanks issta. In my opinion, there is nothing wrong with the paired t test for this. With 600 subjects it will give you p values that can be relied on, even if the data isn't perfectly normal. (Other forum members may disagree. We'll see.)
My view is to keep the analysis simple - the paired t test is easy to do, easy to understand and easy to explain. One problem could be convincing the editors that the reviewer is mistaken in their contention that the dependence of one score on another for the same person invalidates the paired t test. We know, in fact that the pairing actually takes that dependence into account. It would be interesting to see the actual correlation between successive groups.
A more pressing problem is that of multiple p values if you are doing multiple comparisons. How many tests do you plan to do? How do you plan to control the false positive rate? kat
Thanks again for the reply Kat! I only plan the paired samples t-test comparing bets on 4 and 5. I also did a repeated measures anova to check if there was an effect of scenario. I could run a Bonferroni correction, do you think this would be a good method of controlling the false positive rate?
 

katxt

Active Member
#8
With one test there is no problem. If your final analysis is bristling with p values, some of which are hovering just below 0.05, then some caution is needed. Bonferroni is conservative, but the fancy alternatives aren't much better. You can always explain the problem and report p = 0.04 as "probably significant but not yet proven."