meta-analysis

rach

New Member
#1
Hi I am supposed to conduct a meta-analysis of 30+ studies regarding specific training programs. The majority of studies use different measures and different analyses (t-test, ANOVA, Chi Square, correlations etc). Am I able to combine all together in one analysis? And if so, how is this done? Or do I just conduct analysis on a couple that have used similar methods in the initial studies? Any advice would help!
 

JohnM

TS Contributor
#2
That's often how meta-analysis is "sold" -the fact that you can magically combine results from several studies into one big study or one big analysis.....but common sense should cause warning bells to go off - this isn't really possible.

What you need to do is try to group together or map out how each study contributes to the total understanding of the effectiveness of the particular training program.

Understand that all of the methods you listed (t-tests, Chi-square, ANOVA, correlation) are actually correlational tools, so they are very strongly related to each other - they all try to use explanatory variables to explain what factors affect the success/lack of success of the training program - although this usually doesn't come out in a lot of stats courses.

So, for each of the 30+ studies, you need to list the independent variables, the dependent variables, and whether or not there is a significant relationship - look for patterns, and use these patterns to "weave" together the meta-analytic story.

Remember, however - with meta-analysis, there is a "file drawer" effect - you are usually combining the results from published studies - but what about all of the studies on this topic that never got published because they didn't find a significant or interesting relationship?
 

rach

New Member
#3
Meta-analysis help?????????????????

Thanks John M. That's a big help. Now just say that I can establish some common patterns/variables (eg knowledge gained, perceived confidence) how do I conduct the analysis when the correlations are done differently ie chi-square vs t-test vs Pearsons r vs ANOVA. My understanding is that I have to combine all effects from the different studies into the same units. How do I do this? The next aspect of concern is that many of these studies use instruments/questionnaires that have been designed by the study author/s. Is it OK to lump them all together as long as they're measuring the same thing (eg knowledge gained). I presume that this is not ideal, but may be necessary if mention is made in the final review write up. Thanks again for your help so far.
 

JohnM

TS Contributor
#4
You can combine all effects into the same units by transforming them into standardized effect sizes - it's basically converting an effect size into a z-score - you don't report the actual effect size, only its size in terms of standard deviations from the mean.

You can combine surveys designed by study authors, but you'll need to make clear mention of this in your report.

There isn't just one way to do a meta-analysis - it's a general analytic procedure and it needs to be tailored to your specific situation.

A really good overview and explanation of the method can be found in the book "Reading and Understanding Multivariate Statistics" by Grimm and Yarnold. Meta-Analysis is discussed in Chapter 10.