EXPLANATION: Z AND t TESTING
Both Z and t tests estimate the probability that samples from two Normal distributions, 1 and 2, have equal means; we then make inferences about whether the two distributions have equal means; that µ 1 = µ 2.
With both tests, test statistics, named, here, “Z test” and “t test”, are calculated.
t test = (x̄ 1 - x̄ 2) / (s / √n)
Z test = (x̄ 1 - x̄ 2) / (σ / √n)
The commonly stated instructions are:
Use the t test when n < about 30 and σ is unknown.
Use the Z test when n > about 30, σ is known, or n is large enough that s is an adequately accurate estimator of σ.
What follows is the WHY behind these instructions.
THE VARIANCE AND STANDARD DEVIATION are explained in that section of this book, above.
One estimator of σ, called “s”, the sample standard deviation, is described and explained. The formula for s is:
This formula is used to estimate σ, the population standard deviation. The s that we are talking about here, call it “s rms” for this discussion, is AN estimator of σ, the most-frequently-used estimator of σ, one of many/several estimators of σ.
t testing
s rms is a biased = incorrect estimator of σ.
A Monte Carlo simulation of 400,000 estimates of s rms led to the table below.
S RMS is the average estimate of σ = 1; at n = 2, S RMS = .797, the S RMS estimate of σ = 1.
1 / S RMS is the correction factor; at n = 2, 1 / S RMS = 1.254. 1.254 * .797 = 1.
With n = 2, S RMS / 1 is .797 = 79.7% of σ. By n = 30, S RMS / σ is .992 = 99.2% of σ. The bias = error in S RMS is of most importance when n is small.
There is a t distribution for every n, where n > 1.
Each t distribution has, built into it, the value of the appropriate 1 /S RMS correction factor, and uses that to correct s rms to σ.
t test = (x̄ 1 - x̄ 2) / (s / √n)
s is one input to t test, s rms must be that input.
The t distribution corrects s rms to a closer/better estimate of σ; then solves for Z.
The result is that P (t test ≤ t) = P (Z test ≤ Z), which can only be true if
t test = (x̄ 1 - x̄ 2) / (
σ / √n).
Then, the instruction: “
Use the t test when n < about 30 and σ is unknown.” should read:
Use the t test when n < about 30 and s is s rms, s is an estimate of σ using the s formula above.
If any estimate of σ other than s rms is used, the t test result will be incorrect.
Z testing
Use the Z test when n > about 30, σ is known, or n is large enough that s is an adequately accurate estimator of σ.
Z test = (x̄ 1 - x̄ 2) / (σ / √n)
σ = s rms * (1 /S RMS)
Then multiplying s rms by 1 / (S RMS)) corrects s rms to σ and means that Z testing is as accurate as t testing, even when n < 30.
Comments about the instruction: “
Use the Z test when n > about 30, σ is known, or n is large enough that s is an adequately accurate estimator of σ.”
If σ is known and used; then for
any n, P (Z ≤ Z test) = P (t ≤ t test), t test is not required.
If σ is known; then the < > 30 business can just go away.
If σ is estimated by s rms * (1 / S RMS); then P (Z ≤ Z test) = P (t ≤ t test), and a Z test is always appropriate and a t test is never required.
It is
not true that Z testing is less accurate than t testing, at
any n, P (Z≤ Z test) = P (t ≤ t test). What is true is that if s rms is used in Z testing, then, P (Z ≤ Z test) < P (t ≤ t test). Z testing requires knowing and using σ; using large n reduces but never eliminates the estimating error.
Conclusions
If Gosset had this table; then he would not have had to calculate the many t distributions, get into degrees of freedom, t testing and the rules for Z and t testing. We could just cut that section out of the texts and Z test away.
s rms is one of many estimators of σ, there are several. One of my favorites is the range estimator; the range, at each n, on average, is “c” standard deviations wide. c varies with n, so range / c is an estimator of σ, an unbiased estimator of σ, call it “s r”. s r is easy to calculate and unbiased; but has a larger standard error than s rms. I am trying to make a sensible comparison.
