What P value level do you suggest for this study?
- Thread starter Cameron
- Start date
There is no such thing as "marginally" significant. This is a term used by researchers in an attempt to paint nonsignificant tests as more favorable. Same goes for the bs term of "trending toward significance" for non significant test outcomes.
A test of hypothesis is significant or it is not.
In either case, I think a p-value near .1 constitutes weaker evidence against the null hypothesis. If you or someone else conducted the test, it would be dishonest to not report it, particularly on the basis of it's p-value. This is part of why research is in a reproducibility and replication crisis.
A test of hypothesis is significant or it is not.
In either case, I think a p-value near .1 constitutes weaker evidence against the null hypothesis. If you or someone else conducted the test, it would be dishonest to not report it, particularly on the basis of it's p-value. This is part of why research is in a reproducibility and replication crisis.
The initial post was misleading then, as your wording asked if you should "...take them into consideration and report a marginally significant in a manuscript?" And so, I discussed the popular, but ill-conceived, idea about "marginal" significance (and the similar "trend toward" significance). It also prompted me to point out that irrespective of a p-value, you should report in the manuscript any analysis that was conducted even if you don't find them interesting.
It depends on the alpha level you set prior to seeing your data and conducting analyses. What alpha level did you choose ahead of time? What do the graphs look like?
It depends on the alpha level you set prior to seeing your data and conducting analyses. What alpha level did you choose ahead of time? What do the graphs look like?
I practice in industrial statistics where there is no pressure to publish. When I see p-values in that region, I will target that factor for further investigation. As an outsider, observing the reproducibility and replication crisis, the biggest failing that I see is the rush to publish based on a single study. In industrial statistics, you typically will run a confirmation experiment to validate the results before running a pilot then scaling up to full production. If you skipped these steps and jumped straight to the end (equivalent to publishing after a single study) you will quickly end up on unemployment.
Your approach is more aligned with the scientific process and with how p-values were intended to be used. I agree with you.
I think part of the issue is pressure to publish but I think part of it is lack of understanding as I have seen firsthand. I literally had a well respected researcher (who is a master in his primary field, but not at all in stats) refer to an observational study, without any attempts to adjust for confounding, as "bullet proof" because "...Fisher's exact test is so robust. You don't need fancy, technical methods such as logistic regression..." The guy clearly was missing out on many things.
I think it is okay to publish after a single study as long as conclusions are tempered to reflect the quality and strength of evidence against a null hypothesis. This would just allow readers to see groundwork that has been done and how it has been done and allow for possible improvement on or replication of a study.
I think part of the issue is pressure to publish but I think part of it is lack of understanding as I have seen firsthand. I literally had a well respected researcher (who is a master in his primary field, but not at all in stats) refer to an observational study, without any attempts to adjust for confounding, as "bullet proof" because "...Fisher's exact test is so robust. You don't need fancy, technical methods such as logistic regression..." The guy clearly was missing out on many things.
I think it is okay to publish after a single study as long as conclusions are tempered to reflect the quality and strength of evidence against a null hypothesis. This would just allow readers to see groundwork that has been done and how it has been done and allow for possible improvement on or replication of a study.
Well, I think this whole thing is silly and supports not running NHST and just reporting effect sizes. With the effect size you will be able to show the magnitude, direction, and most importantly uncertainty via precision values. The latter will also help convey sample size concerns.
I am more inclined these days to agree with @Miner - in that you cannot completely disregard a p-value of 0.0501. There is information there. Should you make a decision based on it - hmm, up to you. You also have to keep in mind possible study design problems (e.g., misclassification, selection bias) and that all models are incorrect and just a proxy to understanding the phenomenon. Given these things, is there truly not an effect or is it lost a little bit in the noise. Many biases bias toward the null and repeat studies are always needed.
I am more inclined these days to agree with @Miner - in that you cannot completely disregard a p-value of 0.0501. There is information there. Should you make a decision based on it - hmm, up to you. You also have to keep in mind possible study design problems (e.g., misclassification, selection bias) and that all models are incorrect and just a proxy to understanding the phenomenon. Given these things, is there truly not an effect or is it lost a little bit in the noise. Many biases bias toward the null and repeat studies are always needed.