Stuck with my darn P value

[CB]

I haven't used SAS University, specifically (because I have access to a paid, licensed version at my program), but I've heard good things from people who have used it. You might want to look into it because it is free from SAS (at least in some cases). Any objections to using SAS University? I would imagine functionality might be slightly reduced from a full-paid version, but it's one heck of a program (so I can't imagine the free version totally lacking).

Not familiar with SAS University, sorry. SAS is ok but R can do so much more and is completely free, so I'd generally suggest that

 

[ondansetron]

@PKrazda,



But the sample size is large (120 = 60 + 60), so by the central limit theorem one can expect the difference in the means to be approximately normally distributed (although the original data are skewed.) Do a bootstrap simulation if you doubt it.

I agree with you, that 120 should be "sufficiently large" to invoke the CLT unless the distribution is incredibly odd and nonnormal (it might take a larger sample size to invoke the CLT in some cases). The thing that matters is you're using the sampling distribution to help make your inference, so if it's approximately normal, you can use those nice tests we're familiar with. Further, I think your bootstrap idea is a good idea to buttress the findings from the initial parametric test (i.e. bootstrap the p-value for the test [or simply to visualize an estimate of the sampling distribution] in addition to invoking the CLT on a traditional parametric test for a difference in means). Reasonable concurrence between the two would give more weight to the appropriateness of invoking the CLT.

 

[CB]

But then I saw that the variance was much lower than mean, so I thought (incorrectly!) that it can not be Poisson distributed. But when I did som simulations and took away the zeros, I noticed that the variance would be lower than the mean. So it can be Poisson distributed.

Yeah, it's not-quite Poisson distributed because by design everyone has at least one surgery... The absence of zeros results in the underdispersion (variance < mean). It could be worth defining the DVs as the number of subsequent surgeries to the initial surgery (i.e., subtract 1 from all the counts) to deal with this, though I doubt it'd change the conclusions.

 

[katxt]

You are just regressing number of surgeries on condition.
That's three different reasonable methods providing similar conclusions. Each directly tests the research question of interest (does new method result in fewer surgeries?)

Thanks CowboyBear. I guess I asked for that. I was expecting something more definite like coefficients, dfs and p values that I could inspect.
However, I think the Poisson refression is overkill.
PKrazda is a practitioner with a new clinical method to share with other practitioners. The summary data virtually speaks for itself. The number of single surgeries is clearly well up. The number of 3 or more surgeries is obviously down, What PK needs now is some stats to go with those conclusions. Something simple, clear and correct, and which hopefully other practitioners will recognize. To me a Chi squared test will do just that.
So, at this stage, I will bow out of the discussion, attaching a spreadsheet with the Chi squared formulas on it which, after looking at all the posts PK may or may not think appropriate.
kat

 

[CB]

Thanks CowboyBear. I guess I asked for that. I was expecting something more definite like coefficients, dfs and p values that I could inspect.
However, I think the Poisson refression is overkill.

It's R code. You can just put it into the R console, or http://www.r-fiddle.org/ if you don't have R installed. Sorry, I could have included output, but it just makes the post really big.

I don't think Poisson regression is overkill. Clinical trials are high stakes research, and there's an obligation to do this kind of research really thoroughly. That said on reflection I think the question of the right DV distribution model is much less crucial than the issue of controlling for baseline characteristics of the participants (the ones who got the new method may have been in better shape to start with). It'd be good to hear what the OP is doing about this.

Btw even if we accept the "Poisson is overkill" argument, note that my code includes two other simple tests (t-test, permutation test) that directly test the hypothesis of interest here. Chi-square doesn't. I think it's great you've made the effort to produce that sheet, but it's not the right test, even if we want something super simple.

 

[PKrazda]

Hey Guys

Firstly thanks for all of your help! I've downloaded R and started playing around with it, and whilst I can't exactly say I'm full gotten to grips with it, using the code you've provided I'm trying to grasp the basics.

So far I've decided to use GretaGarbo's suggestion and use the difference of the means to calculate the p value and CI. Thank you for taking the time to show me this method using the code, I've paid a lot of attention to what you did and tried to understand it.

I also looked at the Poisson regression code posted by CowboyBear but I think that's still a bit over my head not through lack of trying though!

Something else I would like to explore is the use of a p value on each line as suggested by katxt...

True, the Chi square test says there is some difference somewhere. Having established that, then you can do post hoc tests at each level, again using Chi square. (This is exactly the process you follow with an anova. Include Bonferroni if you like.)
So, here's a plan. Make three groups 1, 2, 3+ vs new, old, and test for some difference somewhere with a 3x2 Chi square. p = 0.0002 so yes, there is a difference somewhere.
At level 1, test old 14 vs new 35 with 1x2 Chi square. p = 0.003, so yes, new is significantly higher.
At level 2, test old 25 vs new 18 with 1x2 Chi square. p = 0.29, so can't really tell.
At level 3, test old 21 vs new 7 with 1x2 Chi square. p = 0.003, so yes, new is significantly lower.
It convinces me.

Is there any way I can out more about how Chi square and how anova allows the p to be presented by this? I'm guessing this is done in R or is there another way you can do this without using it?

Regardless of any further help from anyone, I want to thank you all again! It so nice that you've taken the time to humour a (clearly) real newbie to this like me and it's really peaked my interest in stats.

 

[oli123]

That's an important discussion.
Given that the null hypothesis is true, as discussed by the researcher of XXXXXX a p-value is the likelihood of getting a result as extreme as or more extreme than the one observed.
Let's look at the text in bold more closely because it's a key component of the definition that's easy to misunderstand—or completely overlook.
Take note of the word "given that." This indicates that the likelihood is dependent on a presumption. The null hypothesis is assumed to be correct.