Same sample, comparison of knowledge before course and after

#1
Hello everyone. I am clueless and I need a push to the Right direction.

i want to compare the test scores of the same sample of students before and after taking a course. Basically I want to test if the course has increased their knowledge. How can I do that?

Can someone please direct me to the right method? And the size sample?
 

Karabiner

TS Contributor
#2
It crucially depends on how knowledge was actually measured, and on which scale level (ordinal? interval?).

What do you mean by "and the size sample"? Are you free to choose the sample size?

And how are the data sampled, e.g. is there just 1 course you investigate, or are there several schools, and
several courses in each school?

With kind regards

Karabiner
 
#3
Ok, each participant will answer 10 multiple questions. Gets 1 Point for each correct answer. It is only one course. The course is voluntary. So I guess I can recruit students until I run out of time. Thank you, the more I am studying statistics the more I get lost. I was looking at paired t test or wilcoxon. But I might totally wrong.
 

Karabiner

TS Contributor
#4
Paired samples t-test should be o.k.

The needed sample size will be somewhere beween 35 and 150, I suppose.
Do already have some pretest and/or posttest data, or at least educated
guesses about performance before and after? This could help determine
how many subjects to include.

With kind regards

Karabiner
 

hlsmith

Less is more. Stay pure. Stay poor.
#5
How long is the course, if enough time elapsed you would know what their scores would have been give they hadn't taken the course without a control group. What do you think the distribution of test scores look for pre and post?
 
#6
How long is the course, if enough time elapsed you would know what their scores would have been give they hadn't taken the course without a control group. What do you think the distribution of test scores look for pre and post?
The course is 30 minutes online. The idea is to test them right before the course and then test them immediately after it.

I saw that for paired t the distribution must be normal. But I will not know until they take it.
 
#7
Paired samples t-test should be o.k.

The needed sample size will be somewhere beween 35 and 150, I suppose.
Do already have some pretest and/or posttest data, or at least educated
guesses about performance before and after? This could help determine
how many subjects to include.

With kind regards

Karabiner
No I don’t. I am in the design phase and I want to get everything right from the beginning. Thank you very much.
 

Karabiner

TS Contributor
#8
If you assume a "small effect" in the population from which your sample was drawn,
then you'll need to analyse 200 cases, if you want to be quite sure that you actually
can demonstrate an improvement (technical details: effect size d=0.2, power 80%, alpha
[two-tailed] = 0.05). If you assume a medium sized effect (d=0.5), then 35 cases. If
something in between (d=0.35), then 67 cases. http://statulator.com/SampleSize/ss2PM.html
Maybe you need some cases more, if subjects fail to complete the post test.

With kind regards

Karabiner
 
#9
If you assume a "small effect" in the population from which your sample was drawn,
then you'll need to analyse 200 cases, if you want to be quite sure that you actually
can demonstrate an improvement (technical details: effect size d=0.2, power 80%, alpha
[two-tailed] = 0.05). If you assume a medium sized effect (d=0.5), then 35 cases. If
something in between (d=0.35), then 67 cases. http://statulator.com/SampleSize/ss2PM.html
Maybe you need some cases more, if subjects fail to complete the post test.

With kind regards

Karabiner
Thank you very much for the information. It did not occur to me to check effect size.