Basic questions about pre-/post-treatment comparison and z-standardization

#1
Assume I have three treatment-groups, each with pre- and post-treatment measurements:
Code:
set.seed(42)
n <- 100
df <- tibble(
group_a_pre = rnorm(n, mean=0, sd=1),
group_b_pre = rnorm(n, mean=50, sd=10),
group_c_pre = rnorm(n, mean=5, sd=2),
group_a_post = rnorm(n, mean=0.36, sd=1.2),
group_b_post = rnorm(n, mean=55, sd=13),
group_c_post = rnorm(n, mean=4.4, sd=2.3)
)
In each of the three groups (a,b, and c), the dependent variable is measured by using a different instrument. All instrument measure the same phenomenon, but using different units.
I have the following questions:
  • Would I use a two-way ANOVA (factors being group and treatment), or would I use one-way ANOVA with group as the only factor, transforming the dependent variable to the difference between pre- and post-treatment?
  • I assume I would z-transform the values in each group? For z-transformation, e.g., for group a, would I z-transform e.g., group_a_pre and group_a_post separately (based on the mean and standard deviation in each of the two vectors) or would I perform the transformation based on the overall mean and standard deviation for c(df$group_a_pre, df$group_a_post)?
Thank you!
 

Karabiner

TS Contributor
#2
Please have a look at the mean and standard deviation of a z transformed variable.

-- Treating three different measurements for a dependent variable, as if they were the same,
sounds a bit awkward. What is the topic and what are the reserach questions here?
Did the three groups receive exactely the same treatment? And could you describe the
three measurements and their scales in detail?

With kind regards

Karabiner
 
#3
Thank you for your response! My situation is as follows:
The data actually comes from a within-subject design: Each individual is asked to rate stimulus material using three different instruments (assume that all three instruments have previously been validated). Manipulation takes place in the second phase of the trial and the hypothesis is, that one of the three instruments (c) will be less susceptible to the manipulation. Thereafter, the individual uses the same three instruments to rate the stimulus material. I suspect differences in the dependent variable to be significantly higher between the two conditions for instruments a and b, as compared to c.

Would I use a two-way ANOVA or is it legitimate to calculate the differences pre- / post-manipulation for the three instruments and perform a one-way ANOVA between these three groups?

Also, the scores yielded by the three instruments are in different units, hence the question regarding standardization.

Thank you and kind regards


Please have a look at the mean and standard deviation of a z transformed variable.

-- Treating three different measurements for a dependent variable, as if they were the same,
sounds a bit awkward. What is the topic and what are the reserach questions here?
Did the three groups receive exactely the same treatment? And could you describe the
three measurements and their scales in detail?

With kind regards

Karabiner
 

hlsmith

Less is more. Stay pure. Stay poor.
#4
Would you need multiple level model to cluster measurements in the person? Also, I would wonder if the three instruments had a different number of items, would this spread impact comparisons between instruments. What is the conclusion specifically, is there a difference between the change values, and if so who wins. Also, you may need to control for multiple comparisonsGen

You packed about three complex questions into one study - good work in creating a bizarre use case.
 

Karabiner

TS Contributor
#5
Each individual is asked to rate stimulus material using three different instruments (assume that all three instruments have previously been validated).
You still did not explain how the instruments look like, but this is crucial.
In addition, you say they are validated - do also normative values exist?
 

katxt

Well-Known Member
#6
You can make things a little simpler by working directly with the differences. The data could then be a column of 100 subject IDs, followed by three columns of differences. These three columns are in different units but presumably correlated.
The question you ask is (I think), are the numbers in of one of the columns in some sense significantly closer to zero than the other two?
My instinct is simply to divide each column by the SD of the column and do a one way repeated measures anova on the transformed columns. That will tell you if there are significant differences between the columns, and if there are, then a post hoc test will tell you what you want to know.
 

katxt

Well-Known Member
#7
Extra note. This is effectively finding if there is a significant difference in effect size for the three instruments.
If it sounds a little fishy, then you could test between two instruments using the raw data and a bootstrap test. The test statistic would be the difference in the effect sizes again, and the difference rows are resampled with replacement. Check to see if 0 is inside the 95% CI for the sampling distribution of the difference in the effect size.