Thursday, February 28, 2019

Reimagining Meta-Analysis

Fruit and Meta-Analysis

The fruit in my house weigh on average 93 grams.  I know this because I weighed them.  The process of doing so is a good analogy for meta-analysis, though a lot less painful.

I bet you find the value of 93 grams rather uninformative.  It reflects what my family likes to eat, say bananas more than kiwis and strawberries more than blackberries.  In fact, even though I went through the effort of gathering the fruit, setting fruit criteria (for the record, I excluded a cucumber because cucumbers, while bearing seeds, just don't taste like fruit), and weighing them, dare I say this  mean doesn't mean anything.  And this is the critique Julia Haaf (@JuliaHaaf), Joe Hilgard (@JoeHilgard), Clint Davis-Stober (@ClintinS) and I  provide for meta-analytic means in our just-accepted Psychological Methods paper.   Just because you can compute  a sample mean doesn't mean that it is automatically helpful.

Means are most meaningful to me when they measure the central tendency of some naturally interesting random process.  For example, if I was studying environmental impacts on children's growth in various communities, the mean (and quantiles) of height and weight for children of given ages is certainly meaningful.  Even though the sample of kids in a community is diverse in wealth, race, etc., the mean is helpful in understanding say environmental factors such as a local pesticide factory.

In meta-analysis, the mean is over something haphazard...what type of paradigms happen to be trendy for certain questions.   The collection of studies is more like a collection of fruit in my house.  And just as the fruit mean reflects my family's preferences about fruit as much as any biological variation among seeded plant things, the meta-analytic mean reflects the sociology of researchers (how they decide what data to collect) as much as the phenomenon under study.

Do All Studies Truly?

In our recent paper, we dispense with the meta-analytic mean.  It simply is not a target for us for scientific inference.   Instead, we ask a different question, "Do All Studies Truly...."  To set the stage, we note that most findings have a canonical direction.  For example, we might think that playing violent video games increases rather than decreases subsequent aggressive behavior.  Increases here is the canonical direction, and we can call it the positive effect.  If we gather a collection of studies on the effects of video game violence, do all truly have an effect in this positive direction, that is do all truly increase aggression or do some increase and others truly decrease aggression.  Next, let's focus on  truly.  Truly for a study refers to the what would happen in the large-sample limit of many people.  In any finite sample for any study, we might observe a negative-direction effect from sampling noise, but the main question is about the true values.  Restated, how plausible is it that all studies have a true positive effect even though some might have negative sample effects?  Using Julia's and my previous work, we show how to compute this plausibility across a collection of studies.

So What?

Let's say, "yes," it is plausible that all studies of a collection truly have effects in a common direction, say violent video games do indeed increase aggression.  What is implied is much more constrained than some statement about the meta-analytic mean.  It is about robustness.  Whatever the causes of variation in the data set, the main finding is robust to these causes.  It is not that just the average shows the effect, but all studies plausibly do.  What a strong statement to make when it holds!

Now, let's take the opposite possibility, "no."  It is not plausible that all studies truly have effects in a common direction.  With high probability some have true effects in the opposite direction.  The upshot is a rich puzzle.  Which studies go one way and which go the other way?  Why?  What are the mediators?

In our view, then, the very first meta-analytic question is "do all studies truly."  The answer will surely shape what we do next.

Can You Do It Too?

Maybe, maybe not.  The actual steps are not that difficult.  One needs to perform a Bayesian analysis and gather the posterior samples.  The models are pretty straightforward and are easy to implement in the Bayes Factor package, stan or JAGS.  Then, to compute  the plausibility of the "do all studies truly" question, one needs to count how many posterior samples fall in certain ranges. So, if you can gather MCMC posterior samples for a model and count, you are in good shape.

We realize that some people may be drawn to the question and may be repelled by the lack of an automatic solution.   Julia and I have unrealized dreams of automating the process.  But, in the meantime, if you have a cool data set and an interest in the does-every-study-truly question, let us know.

Rouder, JN, Haaf, JM, Davis-Stover, C, Hilgard, J (in press) Beyond Overall Effects: A Bayesian Approach to Finding Constraints Across A Collection Of Studies In Meta-Analysis. Psychological Methods.


Unknown said...


With a joy in my heart I want to use this medium to tell the world about Lord gbojie who helped me in getting my lover back with his powerful spell, my ex and i was having misunderstanding which led to our breakup and i went to beg him several times to please forgive and accept me back because i know i offended him but each time i went i always feel more deeply in pain and agony because he always walk out on me and would not want to listen to what i have to tell but on a faithful day as i was browsing i came across  a testimony of a woman whose problem was more than mine and yet Lord gbojie helped her with his powerful and effective spell so i was happy and also contacted Lord gbojie for help via email and then i told him my story but the only thing he said was that i will wipe you tear with my spell so lucky for me everything want well just as he promised and right now i have got my lover back and we are both living happily. there is nothing Lord gbojie can not do with his spell and just as promise my self i will keep testifying on the Internet of how Lord gbojie helped me. Are your problem greater that mine or less i give you 100% guarantee that Lord gbojie will put an end to it with his powerful spell, contact Lord gbojie for help you can reach him no his email : or you can also call him or whatsapp  +2349066410185  website :  

şevki said...

milf porno
Katja Krasavice Porno

Donald Scott said...

HOW I WAS ABLE TO FLUSH OUT THE HERPES VIRUS WITH THE HELP OF DR. KHAM NATURAL PRODUCT, HIS EMAIL I was diagnosed with Herpes a year ago. It’s the worst thing that’s ever happened to me and  I feel like my life is over. Living with constant pain is something no one should have to go through, Herpes has also had a huge impact on my sleep . Since being diagnosed I’ve fallen so far into depression I see no way out because i have been to different hospital. I’m taking a high dose of antidepressants, as well as sedatives and sleeping-pills. I don’t know how I’ll ever accept that this is my life now…I keep asking why me because know changes? But I guess reading your testimony really made me feel happy because i believe what i had from people out there who you also help to cure the herpes virus, and i believe so much with your herbal medicine i am going to be out of this bad situation like others you also help before, what we i have done if not for the herbal medicine that i requested for that was able to flush out the bad virus that was making me go crazy. people out there how will i thank this man called DR. KHAM, i will advice you to email him direct if you have disease like herpes etc. or whatsapp him +2348159922297

Elvis Rose said...

Greetings to every one here. I want to let every one know that HPV is a serious and deadly disease which can't be cured through drugs or injections by the American doctors but the best way to deal with HPV is by taking natural herbs medicine for it to flushes out the virus and is only few American doctors that know about this herbal medicine from doctor Afrid a great herbalist doctor. I have read about Dr Afrid the great herbalist doctor from African who can cure HPV with his powerful herbal medicine. for the people suffering from the following diseases Epilepsy, Hiv, Herpes, Cancer, Als, Hepatitis, Diabetes, Hpv,Infections ETC should contact him for his herbal medicine because his herbal medicine cured all kinds of diseases or virus, i am telling you all this because i am a living testimony and i was cured of Hpv. and my child was cured of epilepsy. Although, i sent him what he requested and he sent me his medicine which i took for two weeks and today when i went for test i was tested Hpv negative. you can reach him through his EMAIL;[] And you can also whatsapp/call him through his number +2349057260738 or website   

Relanes Ihab said...

Hello viewers around the Globe, I was despondent because i had a very small penis, about 2.5 inches soft and 4 inches hard not nice enough to satisfy a woman, i have been in so many relationship, but cut off because of my situation, i have used so many product which doctors prescribe for me, but none could offer me the help i searched for. i saw some few comments on the internet about this specialist called Dr, OLHIA and i decided to contact him on his website i saw on the : so I decided to give his herbal product a try. i emailed him and he got back to me, he gave me some comforting words with his herbal pills for Penis Enlargement, Within 3 week of it, i began to feel the enlargement of my penis, " and now it just 4 weeks of using his products my penis is about 9 inches longer, and i had to settle out with my Ex girlfriend Ella, i was surprised when she said that she is satisfied with my sex and i have got a large penis. Am so happy, thanks to Dr OLIHA I also learn that Dr OLIHA also help with Breast Enlargement Hips and Bums Enlargement etc.. If you are in any situation with a little Penis, weak ejaculation, small breast_hips_bums do get to Dr OLIHA now for help on his email {} or whats app him number: +2349038382931