Some questions regarding an experiment

#1
Hey everyone,

I'm going to conduct an experiment and have some questions and hope this is the correct place to ask them :)

Part 1 of the experiment: There are 2 kinds of groups of participants in the experiment (let's say A and B). The participants have to look at 10 videos in total and for each video, they have to answer yes or no.
Each video has a truth value: either the video is correct or not (true or not true). So the participant is right with their yes/no answer or not.

Part 2 of the experiment: And after they performed part 1 of the experiment, there is some additional information that is provided by me.

Part 3 of the experiment: Then they have to do the same (look at 10 other videos (of the same kind with the same subject)) and answer yes or no for each video.


Q1: How many participants do I need for this experiment to be able to do statistical analysis (how many of A and B or combined)? Is it 30? And 30 in total or 30 of A and 30 of B?

Q2: Is showing 10 videos enough trails per part of the experiment or what is the minimum required number of trails needed? I tried to google this a lot of times but am still unsure.

Q3: And what kind of statistical analysis is best to use to see if there is a difference between the number of times A vs B got the correct truth-value (so correctly say yes when the answer is indeed yes and no when the answer is indeed no), and to see if there is a difference between before and after part 2 of the experiment for both A and B and between them? Maybe a mixed ANOVA within- and between-factors? Or maybe something additional to that of something completely different?

Thanks a lot :)
 

Karabiner

TS Contributor
#2
Part 1 of the experiment: There are 2 kinds of groups of participants in the experiment (let's say A and B).
Are these natural groups (such as males/females; with/without high school degree), or are the groups
created by the experimenter?
Part 2 of the experiment: And after they performed part 1 of the experiment, there is some additional information that is provided by me.
Do A and B receive the same information?
How many participants do I need for this experiment to be able to do statistical analysis (how many of A and B or combined)? Is it 30? And 30 in total or 30 of A and 30 of B?
You did not explictly state your research questions.
Generally speaking, the more the better.
The minimum sample size required depends on your assunptions about
how large the effects are.
Maybe a mixed ANOVA within- and between-factors?
Yes, maybe.

With kind regards

Karabiner
 
#3
Are these natural groups (such as males/females; with/without high school degree), or are the groups
created by the experimenter?

Do A and B receive the same information?

You did not explictly state your research questions.
Generally speaking, the more the better.
The minimum sample size required depends on your assunptions about
how large the effects are.

Yes, maybe.

With kind regards

Karabiner
Thank you for responding!

The first group is a group of random members of the public (aging from 20 to 60, both females and males) who are willing to participate. The other group is a group of Artifical Intelligence experts (aging from 23 to 25, both females and males).

Yes, A & B receive the same information.

The main research questions are (may still be tweaked a little bit):
-Is there a difference in the ability of general members of the public versus AI experts to differentiate between deepfakes and non-deepfakes?
-What is the effect of a little amount of training on characteristics of how to recognize deepfakes on the ability of both general members of the public versus AI experts to differentiate between deepfakes and non-deepfakes and is there a significant difference between both groups?

What do you advise the minimal number of trails and total participants to be? Will 30 participants be enough for reliable statistical testing (finding enough participants is hard)? (I assume that there won't be an extremely big effect)

With kind regards,

Persco
 
Last edited:

Karabiner

TS Contributor
#4
The great advantage is the repeated-measures design, which will reduce
error variance. Mabe you can use some co-variates to further reduce
error variance. But if you don't expect an effect of at least medium size,
then I suppose that total n = 30 will be a bit sparse. You could maybe
use a sample size calculator such as g*power to get an idea of what
would be necessary to obtain a reasonable statistica power.

With kind regards

Karabiner
 
#5
Thanks!
The great advantage is the repeated-measures design, which will reduce
error variance. Mabe you can use some co-variates to further reduce
error variance. But if you don't expect an effect of at least medium size,
then I suppose that total n = 30 will be a bit sparse. You could maybe
use a sample size calculator such as g*power to get an idea of what
would be necessary to obtain a reasonable statistica power.

With kind regards

Karabiner