Quick Question on Hypothesis Testing

#1
Let's say you test hypothesis at alpha=.05, does that mean that 5% of your rejections will be wrong?

Is this statement true? If not, can you give an example?

I know that alpha= P(type 1 error), which is the probability of rejecting given Null is true. But I still can't tell whether that statement is true or not. Some of my friends are saying that it is false, but I am not totally convinced.

Am I missing something? Can someone please help me understand this?
 

Karabiner

TS Contributor
#2
Let's say you test hypothesis at alpha=.05, does that mean that 5% of your rejections will be wrong?
If you only test true null hypotheses, then 5% of your rejections will be wrong.
If you test only false null hypotheses, 0% of your rejections will be wrong.
If you test a mixture of wrong and false null hypotheses, then between 0% and 5% of your rejections will be wrong.


With kind regards

K.
 

Karabiner

TS Contributor
#4
If you only test true null hypotheses, then 5% of your rejections will be wrong.
If you test only false null hypotheses, 0% of your rejections will be wrong.
If you test a mixture of wrong and false null hypotheses, then between 0% and 5% of your rejections will be wrong.


With kind regards

K.
Sorry, but that was misleading, as I see now.

If you test only true Null Hypotheses, then of course
all of your rejections must be wrong. But you can expect
that you will make wrong (i.e. false-positive) decisions
in 5% of tests (while false-negative decisions are impossible
if all Null Hypotheses were true).
 

Karabiner

TS Contributor
#6
If all Null Hypotheses are true, then it is impossible to make a type II error.
Therefore, the expected proportion of false decisions is 5%.

(at least, if I didn't confuse things once more)