High statistical significance with low R squared coefficient

#1
Hi all, I recently created a regressor using lm() to establish the linear regression of a dependent variable over an independent one using this code:
regressor = lm(formula = Dependent var ~ Independent var, data = dataset)
After running its summary to evaluate significance and coefficient correlation I was quite surprised to find these results:
summary(regressor)
Pvalue: < 2e-16 ***
Multiple R-squared: 0.5338
Adjusted R-squared: 0.5328

In my mind a high statistical significance is tied to a high R squared close to 1, but this seems not to be the case.
Plotting the data indeed I see dots all over the place, although there is a weak tendency towards positive regression...
Could anyone help me to interpret these results and the validity of this model?

Thanks
Alex
 
Last edited:

hlsmith

Omega Contributor
#2
You are testing that their is a positive or negative relationship (slope). With a big enough sample you can prove the smallest of relationships (is not zero). You likely have a large enough sample to prove the slope just isn't zero. I will point out that in most fields an r^2 of around 0.5 would be considered a large value. It would mean a correlation > 0.7 (0.5 > (0.7*07)).
 
#3
Thanks for your input hlsmith. It is a 'data.frame': 480 obs. of 6 variables. I was surprised because on similar datasets over the same variables I used to have higher R squared coefficients lingering around .7. Would you know what is an approximate expected R squared value from similar data sources when there is a 3 stars *** statistical significance? Thanks
 
#4
There is no "expected" R-squared. R^2 and the p-value measure two completely different things. A small p-value for a variable in a regression implies that it is improbable that the observed association between the independent and dependent variables is due to a chance. R^2, on the other hand, is the proportion of the variability in the dependent variable that is explained by the independent variable. There is no reason that R^2 should be high just because the p-value is small. (And who says R^2=.5 is "low," anyway.)
 
#5
There is no "expected" R-squared. R^2 and the p-value measure two completely different things. A small p-value for a variable in a regression implies that it is improbable that the observed association between the independent and dependent variables is due to a chance. R^2, on the other hand, is the proportion of the variability in the dependent variable that is explained by the independent variable. There is no reason that R^2 should be high just because the p-value is small. (And who says R^2=.5 is "low," anyway.)
The bold part isn't accurate. A small p-value for the case you provided would indicate that it's improbable to see a coefficient at least as contradictory to the null hypothesis, IF what we saw is entirely due to chance (null is true). In other words, p-values don't tell you any sort of probability that "observed results are due to chance" because a p-value is calculated with the assumption that observed results are entirely due to chance.
 
#6
The bold part isn't accurate. A small p-value for the case you provided would indicate that it's improbable to see a coefficient at least as contradictory to the null hypothesis, IF what we saw is entirely due to chance (null is true). In other words, p-values don't tell you any sort of probability that "observed results are due to chance" because a p-value is calculated with the assumption that observed results are entirely due to chance.
Yes, that's a more accurate way to put it.
 
#7
Thank you all for your inputs. In essence I gathered that P value and R squared are separate values, but the fact that the P value defeats the null hypothesis (in this instance the P-value is lower than the null hypothesis, therefore there is an extreme unlikelihood that the result is due to pure chance) indirectly implies that there is indeed a correlation between independent and dependent variables. After all that is what the P value is after. I should recalibrate my estimation of R squared since I heard on different forums that .5 is high indeed. It is just that on similar datasets from the same variables I used to get higher values (around .7), which brought me to the question: what is the R squared range value to be expected on a data of 500 observation circa over 6 variables when there is a high statistical significance ***? But probably this question is too vague and undefined for an answer. Thanks again for your help
 
#8
Regardless of sample size, R^2 can vary from 0 to 1. R^2 is the amount of variance explained by the model. There is no direct relationship between R^2 and sample size.
 
#9
the P-value is lower than the null hypothesis, therefore there is an extreme unlikelihood that the result is due to pure chance
This is what I was saying is incorrect. The p-value doesn't tell you anything regarding the probability the null hypothesis is true or false. I can design an experiment where the null hypothesis is true but where the p-value is very low. This illustrates why it's incorrect to say the p-value tells us anything about the likelihood the result is due to chance. In this example, we know 100% the result was due to chance, so it's completely inaccurate to say the low p-value indicates a low likelihood the result is due to chance.
 
#11
Really? With a fixed sample size? I’d like to see that.
Sure, if you want a real experiment versus a simulation, either can be done. Failure to randomize people into groups in a way that leads to bias away from the null, or introducing some other form of bias that typically occurs in studies and you can get a p-value less than alpha, but the null is still true. If you don't like the idea of building bias into an experiment despite bias occurring in real life, we can create an easy experiment where we toss a fair coin 10 times with the null that proportion of heads is .5. It would be relatively easy to achieve lower p-values in repeated experiments, possibly even the first try. We can then also run simulations where we make this easy and design the experiment to run but with violated assumptions or bias as happens in real life and a low p-value still doesn't indicate the null is extremely unlikely. Again, we can take out added bias or assumption violations and in repeated simulation, you will get some low p-values. You can do all of this with a fixed sample size.
 

hlsmith

Omega Contributor
#12
Well back to a couple components, I bet you can slap 95% confidence intervals on the prior model of 500 units if you really wanted and see the estimated variability of the estimate.
 
#13
Sure, if you want a real experiment versus a simulation, either can be done. Failure to randomize people into groups in a way that leads to bias away from the null, or introducing some other form of bias that typically occurs in studies and you can get a p-value less than alpha, but the null is still true. If you don't like the idea of building bias into an experiment despite bias occurring in real life, we can create an easy experiment where we toss a fair coin 10 times with the null that proportion of heads is .5. It would be relatively easy to achieve lower p-values in repeated experiments, possibly even the first try. We can then also run simulations where we make this easy and design the experiment to run but with violated assumptions or bias as happens in real life and a low p-value still doesn't indicate the null is extremely unlikely. Again, we can take out added bias or assumption violations and in repeated simulation, you will get some low p-values. You can do all of this with a fixed sample size.
In other words, in the absence of systematic error, if the null hypothesis is true, you cannot design a study to have a low p-value.

The bigger issue is that your assertion that "[t]he p-value doesn't tell you anything regarding the probability the null hypothesis is true or false" is false. If it were true, frequentist hypothesis testing would be useless. The p-value is a measure, albeit a flawed one, of how inconsistent the data is with the null hypothesis. A very small p-value, such as the one that the OP got, says that the observed relationship between the predictor and outcome is wildly improbable if the null hypothesis is true. In practice that can usually be interpreted to mean that it is wildly improbable that the null hypothesis is true. More technically, it must be the case that a p-value provides information about the probability that the null is true, because the p-value and sample size can be converted to a posterior probability of the null hypothesis.

Much more discussion of this subject is probably off-topic. Perhaps the moderators could transfer these posts to a new thread about interpretation of p-values in the appropriate forum.
 
Last edited:
#14
In other words, in the absence of systematic error, if the null hypothesis is true, you cannot design a study to have a low p-value.

The bigger issue is that your assertion that "[t]he p-value doesn't tell you anything regarding the probability the null hypothesis is true or false" is false. If it were true, frequentist hypothesis testing would be useless. The p-value is a measure, albeit a flawed one, of how inconsistent the data is with the null hypothesis. A very small p-value, such as the one that the OP got, says that the observed relationship between the predictor and outcome is wildly improbable if the null hypothesis is true. In practice that can usually be interpreted to mean that it is wildly improbable that the null hypothesis is true. More technically, it must be the case that a p-value provides information about the probability that the null is true, because the p-value and sample size can be converted to a posterior probability of the null hypothesis.

Much more discussion of this subject is probably off-topic. Perhaps the moderators could transfer these posts to a new thread about interpretation of p-values in the appropriate forum.
As far as I'm aware, the easiest way would be introducing biases that readily occur in practice. If you don't want to, you could certainly repeat the experiment as many times as you want to keep interpreting p-values and end up making many different claims regarding the probability of Ho based solely on the p-value, which isn't correct as far as I know. It's poor logic (circular) to assume Ho is true to calculate the p-value, then try to use the p-value to tell you the probability of Ho. There is also an important difference of saying the p-value measures inconsistency between observation and an assumed state of nature versus saying the p-value tells you how likely your assumption is to be true. I think you're towing the line between inverse probability misinterpretations by saying if Ho is true, this set of outcomes occurs with a low probability. We observed a member of the unlikely set and therefore, Ho is unlikely. Event A being unlikely with the assumption of B doesn't necessarily make B unlikely given the observation of A.

Sure, with additional information, such as priors, you can transform a p-value to get a posterior, but a p-value alone can't tell you the probability of Ho (hence why I said "the p-value"-- it could be used in junction with other information). And further, you're no longer using the Frequentist p-value in the Frequentist framework for inference.

Now, if people were to repeat, redesign to remove bias, and continue an experiment many times and keep seeing low p-values, this would be a correct framework to being questioning the null hypothesis, at least, according to Fisher.
 
#15
As far as I'm aware, the easiest way would be introducing biases that readily occur in practice. If you don't want to, you could certainly repeat the experiment as many times as you want to keep interpreting p-values and end up making many different claims regarding the probability of Ho based solely on the p-value, which isn't correct as far as I know. It's poor logic (circular) to assume Ho is true to calculate the p-value, then try to use the p-value to tell you the probability of Ho. There is also an important difference of saying the p-value measures inconsistency between observation and an assumed state of nature versus saying the p-value tells you how likely your assumption is to be true. I think you're towing the line between inverse probability misinterpretations by saying if Ho is true, this set of outcomes occurs with a low probability. We observed a member of the unlikely set and therefore, Ho is unlikely. Event A being unlikely with the assumption of B doesn't necessarily make B unlikely given the observation of A.

Sure, with additional information, such as priors, you can transform a p-value to get a posterior, but a p-value alone can't tell you the probability of Ho (hence why I said "the p-value"-- it could be used in junction with other information). And further, you're no longer using the Frequentist p-value in the Frequentist framework for inference.

Now, if people were to repeat, redesign to remove bias, and continue an experiment many times and keep seeing low p-values, this would be a correct framework to being questioning the null hypothesis, at least, according to Fisher.
I did not say that the p-value "tells you" the probability of the null hypothesis. As I wrote, what I take issue with is your assertion that the p-value "doesn't tell you anything regarding the probability of the null hypothesis." All else equal, the smaller the p-value, the less probable the null hypothesis. The OP reports a p-value for a regression coefficient of 2*10^-16. If the null is true, there is 1 chance in 1 quintillion of seeing a regression coefficient at least that extreme. What odds would you require to bet that the null hypothesis is true?
 
#16
I did not say that the p-value "tells you" the probability of the null hypothesis. As I wrote, what I take issue with is your assertion that the p-value "doesn't tell you anything regarding the probability of the null hypothesis." All else equal, the smaller the p-value, the less probable the null hypothesis. The OP reports a p-value for a regression coefficient of 2*10^-16. If the null is true, there is 1 chance in 1 quintillion of seeing a regression coefficient at least that extreme. What odds would you require to bet that the null hypothesis is true?
So you're trying to move from a Frequentist interpretation to a Bayesian one. I think your argument, in the Frequentist framework, is flawed in that the null is either true or it is not, "probability" 1 or 0. The p-value doesn't change that in the slightest. It also doesn't imply that [small p-values indicate a highly improbable null], which is more or less what you were saying. If you want to change the framework to a Bayesian probability, then I agree with you that a smaller p-value would imply a less likely null, all else constant. But, a p-value and it's interpretation are Frequentist in nature.
 
#17
I'm not making an argument in the frequentist framework. Questions about probabilities of hypotheses are inherently Bayesian. So, yes, it's a Bayesian statement to say that all else equal, the smaller the p-value the less probable the null hypothesis.
 
Last edited:
#18
I'm not making an argument in the frequentist framework. Questions about probabilities of hypotheses are inherently Bayesian. So, yes, it's a Bayesian statement to say that all else equal, the smaller the p-value the less probable the null hypothesis.
Right, but the OP is presenting a Frequentist statitstic, and encouraging the interpretation in the Bayesian framework contributes to the poor literacy we see today and the illusion that a p-value of .08 means there's an 8% chance the null is true or that a "mistake" has been committed. You and I may know the issues or problem with that, but someone asking how to interpret a Frequentist measure should be provided with an accurate, Frequentist interpretation. If you want to give them a Bayesian answer, you should probably clarify for their knowledge that this isn't the widely accepted interpretation of that particular measure, but it may be valid in some cases and that additional assumptions are needed to arrive at the solution you're suggesting.

Staying on the OP's point would have been to provide a Frequentist answer to a Frequentist question (how do I interpret this p-value).