- Thread starter joeb33050
- Start date

Hi Joe,

I showed that when you estimate the standard deviation, the t-distribution, give better results than the normal distribution, even when n>30.

I showed that when you estimate the standard deviation, the t-distribution, give better results than the normal distribution, even when n>30.

Code:

```
n <- 10
N <- 10000000
mu <- 0
sigma <- 1
# Tests against ho: mu=0, we'll wrongly calculate
# the sd from the data
ztest_fromdat <- function(dat){
xbar <- mean(dat)
s <- sd(dat)
z <- (xbar - 0)/(s/sqrt(n))
pval <- 2*pnorm(-abs(z), 0, 1)
return(pval)
}
single_trial <- function(n, mu, sigma){
dat <- rnorm(n, mu, sigma)
t.p <- t.test(dat)$p.value
z.p <- ztest_fromdat(dat)
output <- c(t = t.p, z = z.p)
return(output)
}
output <- replicate(N, single_trial(n=n, mu=mu, sigma=sigma))
# Over repeated trials the probability that the p-values
# should be below our alpha (here 0.05) should equal alpha.
# that's the definition of what alpha is.
colMeans(t(output) <= 0.05)
```

Code:

```
> colMeans(t(output) <= 0.05)
t z
0.05068 0.08229
> # This means that in our 100000 replications we rejected the null in 5.068% of the reps for the t-test
> # and we rejected the null... 8.229% of the reps for the z-test. We used alpha=0.05 so it should be 5%.
> # the value we see in our experiment is within sampling error of the true value. The value we see
> # for the z-test isn't.
>
> # If you're interested in the math...
> # The value for z...
> # is awefully close to this...
> # (because it's the value it's estimating)
> pt(qnorm(0.025), 9)*2
[1] 0.08164913
```

From there I just conduct the t-test and z-test like we would with that data. I then do this many times and see what happens.

For our particular example every time I generate new fake data I generate a sample of size 10 (that's the 'n' in my script). I repeat the whole experiment 100000 times I guess (I think I changed that particular value after copying the code here but before running the script). Seems a bit overkill but I had time to let it run and wanted a fairly accurate sample I guess.

You're free to modify the code to see what happens for different values. The important thing to note though is that the z-test isn't correct and why do we say that? Because 1) it's just mathematically incorrect but I don't think you'd be able to follow the arguments for why but that results in 2) the actual type-I error rate isn't the same as the level of alpha we choose for the test.

n = 10? As I have mentioned, Z is about n > 30, or 50, or 100, depending on the source. Z doesn''t work well below some n like 30. It doesn't compute when n = 10. Nobody I know would use Z in an n = 10 situation.

How about n = 30, then n = 100?

I have no clue about your script/program.

Thanks;

joe b.

Sigma is the actual standard deviation for the population. I generate data from the theoretical population. Once that happens we treat that just like we would with real data - basically I pretend like I don't know the parameters of the distribution that generated the data. It's just data at that point. So if we are going to do a test of any kind we need to estimate those parameters. We use the sample mean and sample standard deviation for this test because we don't know what the parameters actually are (or least that's what we're treating the situation like).

From there I just conduct the t-test and z-test like we would with that data. I then do this many times and see what happens.

For our particular example every time I generate new fake data I generate a sample of size 10 (that's the 'n' in my script). I repeat the whole experiment 100000 times I guess (I think I changed that particular value after copying the code here but before running the script). Seems a bit overkill but I had time to let it run and wanted a fairly accurate sample I guess.

You're free to modify the code to see what happens for different values. The important thing to note though is that the z-test isn't correct and why do we say that? Because 1) it's just mathematically incorrect but I don't think you'd be able to follow the arguments for why but that results in 2) the actual type-I error rate isn't the same as the level of alpha we choose for the test.

How about n = 30, then n = 100?

I have no clue about your script/program.

Thanks;

joe b.

From there I just conduct the t-test and z-test like we would with that data. I then do this many times and see what happens.

For our particular example every time I generate new fake data I generate a sample of size 10 (that's the 'n' in my script). I repeat the whole experiment 100000 times I guess (I think I changed that particular value after copying the code here but before running the script). Seems a bit overkill but I had time to let it run and wanted a fairly accurate sample I guess.

You're free to modify the code to see what happens for different values. The important thing to note though is that the z-test isn't correct and why do we say that? Because 1) it's just mathematically incorrect but I don't think you'd be able to follow the arguments for why but that results in 2) the actual type-I error rate isn't the same as the level of alpha we choose for the test.

Last edited:

You might not think that the difference is large between the actual error rates but the difference is big enough that you should believe the z-test isn't hitting the stated type-I error rate. And yes the difference decreases as the sample size increases. That's partially why the common suggestion that if n > 30 then it doesn't really matter. But if you ask me... if it doesn't *really* matter why would you use the more inappropriate version when it's just as easy to do the appropriate version in any software?

You might not think that the difference is large between the actual error rates but the difference is big enough that you should believe the z-test isn't hitting the stated type-I error rate. And yes the difference decreases as the sample size increases. That's partially why the common suggestion that if n > 30 then it doesn't really matter. But if you ask me... if it doesn't *really* matter why would you use the more inappropriate version when it's just as easy to do the appropriate version in any software?

Not much. At n=30 the actual type-I error for the z-test is 0.0596722 and at n=100 it is 0.05281139. The point is that the t-test is valid and the z-test isn't.

I guess I'm wondering what your questions still are?

I guess I'm wondering what your questions still are?

We make believe that we know variences, ANOVA, and that's sorta OK. But we don't know variances, else we'd have **** few other questions. And to suggest that a Z test is not "correct" unless we know the variance, is nibbling on the fringes, confusing the students and pissing me off. The difference with n>30, between t and Z , is inconsequential. We're in the area of claiming that the accelerator, brake and steering wheel are accelerators; because velocity is a vector quantity. Standing tall in the weeds doesn't impress or help the student. I've only been bothered by this since 1972. However, she has reported that there is pie.

joe b.

(Swiss rich folk are happier than Swiss poor folk. The told me. It's the yodeling.)

We make believe that we know variences, ANOVA, and that's sorta OK

The normal distribution IS important for a lot of things. And it's a great teaching tool. It gets used in teaching situations... and in a lot of complex situations where asymptotics are the best we can do. But nobody I know really uses a simple z-test. But you know what's a lot easier to teach initially than a t-test? That's right - the z-test. So maybe don't get so mad and just understand that it's a much simpler thing to teach conceptually - there are a lot less moving pieces when teaching that particular test.

I'm legitimately not sure why you seem so perplexed and angry about this. What's the big deal? The t-test is the appropriate thing to do. Maybe you weren't taught that but it's pretty much taught in every intro class I've ever seen.

You also seem to go on tangents and just make a lot of weird unrelated points. Not sure if that's a language barrier issue or something but you might want to work on staying on topic.

Not sure what you're talking about. ANOVA doesn't pretend to know the variances. In an ANOVA we calculate the variance from the data - just like we do with a t-test. As a matter of fact if you do an ANOVA on just two groups it's equivalent to doing a t-test.

The normal distribution IS important for a lot of things. And it's a great teaching tool. It gets used in teaching situations... and in a lot of complex situations where asymptotics are the best we can do. But nobody I know really uses a simple z-test. But you know what's a lot easier to teach initially than a t-test? That's right - the z-test. So maybe don't get so mad and just understand that it's a much simpler thing to teach conceptually - there are a lot less moving pieces when teaching that particular test.

I'm legitimately not sure why you seem so perplexed and angry about this. What's the big deal? The t-test is the appropriate thing to do. Maybe you weren't taught that but it's pretty much taught in every intro class I've ever seen.

You also seem to go on tangents and just make a lot of weird unrelated points. Not sure if that's a language barrier issue or something but you might want to work on staying on topic.

The normal distribution IS important for a lot of things. And it's a great teaching tool. It gets used in teaching situations... and in a lot of complex situations where asymptotics are the best we can do. But nobody I know really uses a simple z-test. But you know what's a lot easier to teach initially than a t-test? That's right - the z-test. So maybe don't get so mad and just understand that it's a much simpler thing to teach conceptually - there are a lot less moving pieces when teaching that particular test.

I'm legitimately not sure why you seem so perplexed and angry about this. What's the big deal? The t-test is the appropriate thing to do. Maybe you weren't taught that but it's pretty much taught in every intro class I've ever seen.

You also seem to go on tangents and just make a lot of weird unrelated points. Not sure if that's a language barrier issue or something but you might want to work on staying on topic.

Sorry if I wasn't clear when posting the attached chart, it should answer your question.

The chart shows the actual type I error (rejecting correct H0)

Blue Z - The actual type I error for the Z test when using the sample standard deviation.

Red T - The actual type I error for the T-test.

The actual type I error for the T-test is around 0.05, as expected if the distribution is correct.

View attachment 2204

Last edited:

Ps if you use the Z-test with the population standard deviation, which is the correct use for the test, you get type I error similar to the significance level. (0.05)

This is the expected result for any test used properly.

This is the expected result for any test used properly.

If you need to hammer a nail you can use a hammer or a stone, what will you use?

Seriously, you may use the correct test with the same effort, and get an exact result, or use the incorrect test, and get an inaccurate result. so what will you use?

Correct, when using the incorrect test with a larger n you may get smaller inaccuracy, but why not using the correct test?

When you only have a stone you may use the stone, but we have the hammer