Post-hoc tests' properties

[Analyst1981]

Hi all,
when we read descriptions of several post-hoc tests, we get actually lost, because everyone has its advantage/disadvantage. And sometimes you reach to a point where you can't decide which one is better to use. Even if you use all the candidate post-hoc tests that most likely fit your situation, you might get different significance results.

If we read researches done on them, we usually come across some terminologies that describe them, namely:
Liberal test, Conservative test, a test with tight/loose control over alpha, a test with low/high power.

Actually I can know what high/low power is.
but what is the difference between having a conservative test and a test with tight control over alpha. Aren't they same?

 

[hlsmith]

They could be. I would imagine a conservative test errors on the side of minimizing type I errors. The secret is to think about your setting and select accordingly in balancing threats of errors. Also, don't emphasize null significance hypothesis testing. Look at the estimates themselves. That is what is important.

 

[noetsi]

I think this is a general issue with statistics. Made a lot worse because of disagreement among statisticians. Some say X works better, and some say Y is the way to go and X is horrifically wrong.

Which leaves non-statisticians like me lost on the best way to go. In theory simulations would answer questions like this, but you rarely find any so if you can't simulate yourself (and there are no trade offs to be considered) you are....