I was thinking about z-scores and I'm curious about their usage when data are skewed/non-normal. I often see zscores being used to identify outliers, e.g. with z>1.96, 2.58, etc. HOWEVER: the z-score calculation of z = (x - mean(x)) / stdev(x) is dependent on the mean, and the mean is not an appropriate measure of central tendency when data are skewed. So when data are skewed, I think it's inappropriate to use z-scores and instead we should favour an alternative like Tukey's classification (how boxplots define outliers, can't remember exactly what it's called). Is this correct?
Appropriate use of z-scores
- Thread starter ooostats
- Start date
- Tags non-normality outliers tukey z-scores
I don't agree. I think that the mean is an appropriate measure of central tendency. Not only because the sample mean is an unbiased estimate of the population mean, but because you are often interesed in the mean or the sum. Suppose you apply for a job, then it is the wage sum over a longer period that is relevant, not the week by week median. You will want to know the sum i.e. the mean.
I should have said that it was just this statement that I disagreed with.
If you are interested in the mean or median (of course both are good measures of localisation) depends on your objective. And if you apply for a job and are going to work there for 100 weeks, then which would you be more interested to be informed about? The weekly median salary or the mean salary? I would like to know the mean since it corresponds best to the wage sum.
Or if you are at a hospital department, and you want to cure patients with treatmen A or B, then you are more interested in the mean since if is related to the long run sum.
Consider there two results for a and b:
Clearly the median is the same, 4, but mean of b is much higher.
Also, there is a tendency to talk about outliers as some kind off error. But there are lots of natural data that are highly skewed, e.g the income distribution and substances in environment pollutants. There is nothing wrong with these values. But to delete them would be wrong.
If you are interested in the mean or median (of course both are good measures of localisation) depends on your objective. And if you apply for a job and are going to work there for 100 weeks, then which would you be more interested to be informed about? The weekly median salary or the mean salary? I would like to know the mean since it corresponds best to the wage sum.
Or if you are at a hospital department, and you want to cure patients with treatmen A or B, then you are more interested in the mean since if is related to the long run sum.
Consider there two results for a and b:
Code:
> a <- c(1 , 2, 4, 8, 16)
> b <- c(1/4, 1, 4, 16, 64)
> mean(a)
[1] 6.2
> mean(b)
[1] 17.05
> sum(a)
[1] 31
> sum(b)
[1] 85.25Also, there is a tendency to talk about outliers as some kind off error. But there are lots of natural data that are highly skewed, e.g the income distribution and substances in environment pollutants. There is nothing wrong with these values. But to delete them would be wrong.