Determining the power of a 2-Sample T-Test

[smithosaurus]

So I am running a two-sample t-test , and wanted to set some preliminary accuracy goals.

Ideally, I wanted to end the test when there was a statistically significant difference, using an alpha level of 1%, and power of 90%.

However, this will be my first time setting a power criteria for this test. So for calculating this, I was going to use the sample mean of each sample to calculate the power.

Is this going the right direction? Or is there some huge flaw in my reasoning?

 

[trinker]

Generally effect size is used. So in a 2 sample t-test I'd use a Cohen's d or Hedges g.

PS that's a low alpha and high power. If you're doing this for a real study that may be over calculating sample size needed.

 

[smithosaurus]

For my particular test, I am looking at binomial data with fairly large sample sizes.

Could I still apply Cohens d? I read in the wiki page that Odd's ratio would be appropriate.


An example could help:

Two samples sizes of 10,000 each. p1= .50 and p2= .51

When I calculated Cohens d, I get a value of ~2, but that leads me to think I cant apply it in this situation...

 

[trinker]

Yeah I assumed your data was different from your description. Maybe you'd better back up the trolley and explain what the IVs and DVs are and you Ho.

But yeah in this type log odds, odds ratios etc may be more appropriate. May I ask did you use a t-test on data with a binary outcome?

 

[smithosaurus]

Ok yeah I should give a better overview of what I'm asking:

Basically I want to run an A/B split test for optimizing a website. The metric I want to test is clicks. So its binomial data, p= P(click) & q=P(Dont click)

The only changes between the two is a slight layout change. Sample sizes easily get in the 5,000 to 10,000 range.

I've done hypothesis tests on this in the past, but when I wanted to apply a power criteria is where it got tricky.

And yes you are right about t-tests with binary outcome. This test was related to another based on non-binary data, but I still said t-test where I believe I should be referencing a two-sample proportion test.

 

[trinker]

In a test of proportions I've often seen Cohen's h used as an effect size though I've never conducted this sort of study myself. Power isn't terrible to calculate by hand for something simplier but why not use the free Gpower or R to calculate it?

 

[Karabiner]

I want to run an A/B split test for optimizing a website. The metric I want to test is clicks. So its binomial data, p= P(click) & q=P(Dont click)

http://tinyurl.com/khky2pb

HTH

K.

 

[smithosaurus]

In a test of proportions I've often seen Cohen's h used as an effect size though I've never conducted this sort of study myself. Power isn't terrible to calculate by hand for something simplier but why not use the free Gpower or R to calculate it?

Perfect! The R package was exactly what I needed. Thanks!


I do have one last question though; Say I want to calculate the effect size for a two sample t-test. Quick R says to use the formula of

d=|(sample mean a)-(sample mean b)| / common error SD

I was unsure how they meant to calculate common error variance in this situation, and the other resources I found used different equations.

Would the the following equation work (pooled standard deviation, equal sample sizes)?

Common Error SD = SQRT( .5 * (Sample_Var1 + Sample_Var2))