Workshop evaluation with control group and pre/post/follow-up questionnaire

[Beano]

Hello,

Thanks for taking the time to read this rather long message. It will become clear that I have a terrible grasp of data analysis and need some help. I have read and read and read and am still very confused. Any help is greatly appreciated.

I have to design the evaluation for a health workshop that has 3 outcome measures:
Increase in condom use, increase in knowledge and change in attitude.
In total 36 people will receive the workshop and 36 people will be in the control group. Subjects will complete the questionnaire pre-, post- and 3 months after the intervention.

I have a questionnaire which collects demographic data, a 15-item knowledge section and 35 questions on a 5-point Likert-scale to measure 4 constructs.

What kind of analysis should I unleash on this? The workshop is only for women who are aged between 45-65 with recruitment done through volunteer and snowballing. They are randomly assigned to treatment or control group

So far, I thought I would create baseline data by creating a mean composite value for each of the 4 constructs and for the knowledge section simply see how many % of the questions are answered correctly. This will give me a basis for comparison with post-test and follow-up.

After this, I am lost though.....

- Should I do a chi-squared analysis of the pre-test data to see if the control/treatment groups can be compared?

- Is an ANCOVA appropriate here? Should I do anything at all to take possible differences in things such as education level, marital status and income level into account or is this not relevant for what I am trying to find out?

- Am I right in thinking the ANCOVA only deals with the co-variables so I need to do some other test to make the actual comparison of pre-post and between subjects? If so, which test would I need to apply?

- I am initially only looking to measure between-subjects effects although am I right in thinking that, once the data is in SPSS, I can simply run a within-subject test as well?

- What is a t-test/student t-test? Someone said that was more appropriate for this? Is that in addition to the ANCOVA or in place of the ANCOVA??


Please keep in mind I am an undergrad nursing student with no research experience so simple words go a long way

Thank you very much for any help at all.

Beano

 

[staassis]

Repeated Measures ANCOVA. It is most easily run in SPSS.

 

[Beano]

Will the Repeated Measures ANCOVA also show the difference and significance of this difference between treatment and control groups? I.e. will it show that the workshop has lead to a change in knowledge, behaviour and attitude in the treatment group? I mean a straightforward comparison?

I heard of the Mann Whitney U test and was told this might be a good one to use but I don't understand the difference?

 

[Beano]

Or even, if I turn the likert scale responses into a mean composite score for each construct, I am no longer working with ordinate data, right? So could I do a straightforward t-test instead?

 

[Karabiner]

What kind of analysis should I unleash on this?

You should state your research questions before asking this.

- Should I do a chi-squared analysis of the pre-test data to see if the control/treatment groups can be compared?

Don't know why Chi², but anyway, why should you want to do this?
You said you had a random assignment. Therefore, the null hypothesis
"group1 = group 2" MUST be true (or don't you trust your randomization
procedure?).

- Is an ANCOVA appropriate here? Should I do anything at all to take possible differences in things such as education level, marital status and income level into account or is this not relevant for what I am trying to find out?

Since you have a randomized, repeated measures (mixed) design,
in my opinion there's no need to sacrifice degrees of freedom
without a research question concerning those variables.

Please keep in mind I am an undergrad nursing student with no research experience

So you could just perform a t-test between groups
post-treatement and a t-test at follow-up. You will
certainly find t-tests explained in many books and
on the internet.

With kind regards

K.

 

[Beano]

Thank you for this very clear explanation!! You make perfect sense. I guess I was just getting mired in the amount of data I could collect and analyse. Just because people tell me their income does not mean I have to add that to the things I am analysing at this particular moment. I mistakenly thought data analysis meant you had to link ALL bits of data into your analysis. Which meant I could not pick the appropriate analysis technique. The t-test certainly is the most 'simple'.

The only thing I still wonder about it this: I will be using a small sample size so there is a chance that even with 'proper' random assignment to control/treatment, it might end up with, for example, more 45-50 year olds in the treatment group than in the control group. I understand this is not a problem that might occur in a very large sample but I only have 72 so if the groups are very unequal, it can really screw up the results. Hence I was wondering if I should do a chi-squared analysis on the treatment/control group. Or: do away with the random assignment and assign in such a way that control/treatment are fairly balanced? I am assigning only AFTER the pre-test questionnaire so I would have enough demographic information to create a reasonably balanced group for control & treatment.

Or is that against all rules of research?

 

[Karabiner]

The only thing I still wonder about it this: I will be using a small sample size so there is a chance that even with 'proper' random assignment to control/treatment, it might end up with, for example, more 45-50 year olds in the treatment group than in the control group.

Have a look at stratified sampling and block randomization.
In principle, you identifiy a few important co-variates which might
affect the outcome, and then take measures that ensure that
these co-variates are distributed about equal between groups.

Hence I was wondering if I should do a chi-squared analysis on the treatment/control group.

Again: such an analysis makes no sense, because it tests the null
hypothesis that the two samples are from two populations (!) with
different parameters (e.g. different mean age). But it is impossible,
in case of random allocation from one an the same population,
that the null is not true. So all you can achieve is a type 1 error.
At the same time, important differences between the
samples (!) could turn out "non-significant" due to lack of power.
I'd rather judge the differences between the samples from descriptive
statistics (and/or perform procedures as mentioned above).

Or: do away with the random assignment and assign in such a way that control/treatment are fairly balanced? I am assigning only AFTER the pre-test questionnaire so I would have enough demographic information to create a reasonably balanced group for control & treatment.

Yes, you could even use the pretest score for matching.

With kind regards

K.

 

[Beano]

Once again, thank you for the quick response. I realise now that I misinterpret what the chi-squared is for so thanks for clearing that up.

I will follow your advice which sounds pretty sound to me (and more importantly: I understand what you said!).

I have had sleepless nights over not understanding this so I am very grateful for the help!!

Beano

 

[Beano]

Last thing....


Once I performed the t-test, I will have to do something with a p value to determine if the outcome are likely to be due to intervention or not, right?

 

[Beano]

So, I have had a looooong thought about this and realised I still did not know what I was even asking for. So..... here is a re-cap.

I have to design the evaluation for a health workshop that has 4 outcome measures:
Increase in condom use, increase in knowledge, improved confidence and change in attitude as measured on a Likert-scale.
In total 36 people will receive the workshop and 36 people will be in the control group. Subjects will complete the questionnaire pre-, post- and 3 months after the intervention.

I have a questionnaire which collects demographic data and 35 questions on a 5-point Likert-scale to measure the 4 constructs mentioned above. I am going to treat the Likert scores as interval data and calculate a mean composite score for each of the constructs. So I will end up with 4 mean composite scores per questionnaire.

I want to see if attending the workshop is positively related to higher means in the test scores in the treatment group. The control group will get just a leaflet so I am expecting a small change in their scores too but I am looking to see if the workshop participants have a bigger change from their baseline (t1), pre-test, compared to the post-test and the follow-up test. I want to answer the question if the workshop improves knowledge and results in a positive change in attitude.

What kind of analysis should I unleash on this? The workshop is only for women who are aged between 45-65 with recruitment done through volunteer and snowballing. They are randomly assigned to treatment or control group. I feel a t-test is too limited because I believe age, education level and relationship status play a large part in the topic I am looking at, even though I am not controlling for these in the study. Thus I feel I need to take them into account when doing the analysis?

To start right at the basis, I am struggling a bit with all the different terms used for the various components of the analysis. So can someone please let me know if I have them right in my head?

Independent variable: The Workshop
Dependent variable: The results of the tests at t2 and t3? (Does this mean I have 2 dependent variables or 4: 2 for treatment and 2 for control?)
So what are things such as age, education level, relationship status? They are variables too? Or are they covariates? And what are the factors in this design? This obviously determines if I am conducting a 1-way analysis or two way.
And if ANCOVA is for comparing 3 or more groups (as it seems to say everywhere), how does it apply to my study where I am only comparing treatment to control (i.e. 2 groups)? Or does the word 'group' apply to something else here?


I have read that it is best in this situation to do between-subjects ANCOVA and enter the pre-test scores as a covariant?

Am I right in thinking that if I do this, then I will be able to determine if:
There is any difference between the control and treatment in the amount of change from the pre-test to the post-test?
There is any difference between the control and treatment in the amount of change between pre-test-test and follow-up?
There is any difference between the control and treatment in the amount of change between post-test and follow-up?

For the analysis of within-subjects, I would then run the repeated measures ANCOVA as I wish to determine if things such as education and relationship status are correlated to outcome scores.

Does this make sense or am I still barking completely up the wrong tree?

Thanks.