Question: making a positive case for "no effect"

[dwilson]

I have the results of a basic web-based marketing test. Visitors were randomly sent to one of several pages with a binary outcome of "sale" or "no sale" for each visitor. The result fails to reject the null hypothesis of "the page variation has no effect on sales."

However, in my organization, there are existing beliefs (and disagreements) about the value of the different test cases. So I expect to face the question, "But how confident are you that there's really no effect?"

General question (1): what is the appropriate statistical test to answer this question?

I expect that this boils down to (2): what statistical test will allow me to say something like, "I'm 95% confident that the magnitude of the effect size is less than X?"

[terzi]

In fact, all statistical test are based in probability distributions so you can make that kind of statement for all of them. What analysis did you perform? How did you find out that there wasn't a relationship? If you used hypothesis testing, you should have used a significance level, right?

Now, there's another thing called "power" of a test, but this concept is different and is usually not recommended to measure the power after the analysis has been made.

[dwilson]

Yes, I used hypothesis testing with a 95% significance level in my original analysis. My understanding of the strict interpretation of my result is that I "failed to reject the null hypothesis (that there was no relationship)," which is not the same as ruling out the possibility of a relationship.

Am I getting this right? From your reply, it sounds like hypothesis testing is the way to go. I've considered setting up different null hypothesis: that the effect is larger in magnitude than some value (not sure what). Rejecting it at 95% confidence would allow me to put a reasonably confident upper bound on the effect magnitude. Is this advisable? Is there a better way?

[terzi]

Rejecting a null hypothesis is not the same as ruling it out. When you reject the hypothesis is because, based on your data, the probability attached to your null hypothesis is low enough to consider it inviable. There's still a probability of error involved with every hypothesis test, yet that is another subject.

Now, a one-sided hypothesis test can be helpful in your case. Just be careful when choosing which test to use. Remember that almost all tests (t-test, correlation measures, anova) requires data to fill certain assumptions. When those are not met, any conclusions will be incorrect.

[CowboyBear]

Originally Posted by dwilson

I expect that this boils down to (2): what statistical test will allow me to say something like, "I'm 95% confident that the magnitude of the effect size is less than X?"

Exactly. You can't 'prove' the null hypothesis, but you can show that the effect is beneath some level where one can consider its size to be negligible. The idea is basically just to use whatever test you would otherwise, but tailor the null/alternative hypotheses around to suit the question. E.g. Null hypothesis: effect size over 0.1, alternative hypothesis effect size < 0.1. Or to make things a little easier you could just generate a 95% confidence interval - if the upper (and potentially lower) bounds are within whatever cutoff criteria you choose for a negligible effect, then you have evidence that to indicate the effect is negligible.

This kind of question requires a lot of statistical power though (i.e. big sample size!).

[TheEcologist]

Originally Posted by dwilson

I have the results of a basic web-based marketing test. Visitors were randomly sent to one of several pages with a binary outcome of "sale" or "no sale" for each visitor. The result fails to reject the null hypothesis of "the page variation has no effect on sales."

This is binary data with multiple categorical variable, so I'd like to know what test you used in the first place. Give us the # of categorical variables (web pages), and sample sizes.

Often an effective way to 'prove' that a process is likely to occur randomly, is to use a form permutation analysis; let a computer choose each category with equal probability and then choose 'sale' or 'no sale' with the average probability of a sale over all categories. After you calculate how often this random process gives you comparable results to your own.

Now any decent statistical test (fishers exact test, chi-squared) will give you exactly the same information.. however when working with people who dont have a strong statistics background, such a simulation process has a stronger 'psychological effect' than simply telling them some test they dont know and dont understand predicts this or that.

e.g. "after 1 000 000 computer simulations 99% of all web-pages, where the computer had no form of conscious reasoning and was completely unbiased, had a similar or larger effect than the effect displayed"

Ergo there is a 99% chance of getting this by simply clicking away randomly... Although this is the equivalent of the information you get from a p-value, it does carries more meaning with a 'normal person'.

Such a process doesn't enable you to make better informed choices, but it does help in conveying the message of a stats test.

[terzi]

Originally Posted by TheEcologist

When working with people who dont have a strong statistics background, such a simulation process has a stronger 'psychological effect' than simply telling them some test they dont know and dont understand predicts this or that.

I'll have to disagree on that one I don't think a simulation they don't understand will have more impact than a test they don't understand either. The only option where that would be useful is for people with a strong computational background. I'd suggest to use the tool YOU understand the most, since you are the one who will defend it and the one that may somehow explain it to people.

It is hard to make someone understand or even believe your results, but that's part of the statistician's job. In my personal experience, the only thing that will have a 'psychological effect' is the confidence you have in your own work.

[TheEcologist]

terzi and I hijacked dwilson post (this one). The thread was going completely off topic and was of no more use to dwilson. Ergo the move to
http://www.talkstats.com/showthread.php?t=9942

See that thread for the continued discussion.