p-value and alpha confusion

#1
I’m trying to clarify my thinking with regards to these. I’m comfortable (I think) with the idea that the p-value is the probability of obtaining a sample at least this extreme given the null hypothesis being true, and that alpha can be understood as the maximum probability of making a type 1 error given the null hypothesis being true and rejecting the null hypothesis.

But I’m struggling to understanding why those two things aren’t interchangeable? Of course with alpha this is decided when setting up a hypothesis test (or CI) whilst a p-value comes from a specific study in which the null is either true or not.

But suppose I obtain a p-value of 0.04. Since I don’t know in reality whether the null is true or not, could I not express that, assuming the null is true and that I reject it, there is a 0.04 probability that I will have done so incorrectly? And that, in fact, the maximum probability of a type 1 error is 0.04?

Thanks - I’ve read a bunch of posts about p-values and alpha but I’ve struggled to find anything that clarifies this for me.
 

Miner

TS Contributor
#3
Alpha is a threshold value that you set that defines the amount of risk that you are willing to accept that you will commit a Type 1 error. You then compare the p-value to this threshold to determine whether you will reject the null hypothesis.

If the consequences of committing a Type 1 error are high, set alpha low (e.g., 0.01). If the consequences of committing a Type 1 error are low, set alpha higher (e.g., 0.05 or 0.10). I realize that an alpha of 0.10 is not commonly used, but in my field of industrial statistics, we may use it under experimental conditions when we plan on a series of sequential experiments to explore more deeply and confirm the results of earlier experiments.
 
#4
All of this goes away if your are a Bayesian!
Ha, this is true. I’m keen to learn more about Bayesian methods, but I figured I should take some time to revise/understand the main frequentist methods first. My stats knowledge is whatever I can remember from school/college and the odd uni module, having gone down the applied route.
 
#5
Alpha is a threshold value that you set that defines the amount of risk that you are willing to accept that you will commit a Type 1 error. You then compare the p-value to this threshold to determine whether you will reject the null hypothesis.

If the consequences of committing a Type 1 error are high, set alpha low (e.g., 0.01). If the consequences of committing a Type 1 error are low, set alpha higher (e.g., 0.05 or 0.10). I realize that an alpha of 0.10 is not commonly used, but in my field of industrial statistics, we may use it under experimental conditions when we plan on a series of sequential experiments to explore more deeply and confirm the results of earlier experiments.
Thanks, I understand alpha as maximum permissible probability of a type 1 error. I’m just not sure how that substantially differs from the p-value being the probability of a sample at least that extreme.
 
#6
As I understand it P-value is long run probability of an event occuring under independent, random selection from the same population, and alpha is the probability of a type 1 error, so I use alpha and 1-beta to estimate an appropriate sample size and, if the output P-value is very low, subject to assumptions being met and the model being adequate, it suggests the result is unlikely to be by chance alone and accept a type 1 error may happen 5% of the time if a=0.05 when building the sample size
 
Last edited:

hlsmith

Less is more. Stay pure. Stay poor.
#7
I was too lazy to type anything longer yesterday. But the combination of these two responses is my perception of it. Frequentist says estimates are based on repeated samples, slash, realizations from the super population. So the doubt (probability) comes from that process. And the alpha is the threshold based on the doubt.