Monday, April 2, 2018

Advancing The Research Question

To those concerned about methodological practice in psychology,

You are our people. We are a tribe of kindred spirits wondering in the social science wilderness.  

Two difficult issues in this wilderness are the lack of constraint from theories and the lack of constraint in data.  The typical theory predicts that "there is an effect in a certain direction," and the typical analysis is "yes, p<.05."   Even when hacked, we still haven't risked or learned much.   

Many of you have focused on cleaning up the field by improving when we may claim there is an effect (or an invariance).  Your efforts in promoting preregistrations, awareness of QRPs, and more thoughtful statistical analysis are admirable and efficacious.

Nonetheless, the basic testing question---is there an effect---hasn't changed.  We are still playing low-theory, low-risk science.  So what to do?

We came up with what we consider the next question after asking "is there an effect?" It is, "does everybody?"  

Take evaluative conditioning.  Participants judge the emotional valence of relatively neutral objects, say tables.  Some neutral objects are repeatedly paired with negative images (think bloody decapitated puppies), while others are repeatedly paired with positive images (think smiling children playing with adorable baby goats).   Not too surprisingly, tables are rated more positively when paired with smiling children than with decapitated puppies.  The next question is whether this evaluative conditioning effects is universal---does everybody plausibly show an evaluative conditioning effect in the same direction?

Some phenomena clearly hold universally.  If people can hear, then they respond faster to unexpected loud tones (startle) than unexpected soft tones.  Startle is a low-level, subcortical phenomenon, and nobody has a reverse startle where they respond faster to unexpected soft tones than loud ones.  Some phenomena clearly are not universal.  Handedness is a good example---most people can throw a ball further with their right hand; others throw further with their left hand.

Why does it matter?  If a phenomenon is universal, that it, it holds for everyone (or everyone in a subpopulation), we can seek a common explanation.  Further questions might even be metric---is there a metric relationship between the intensity of the sound and the intensity of the startle response. If a phenomenon is not universal, the next questions are: Why do people differ?  What are the correlates of, say, left-handedness?  Why are some people left-handed?  For evaluative conditioning, a universal answer begs the question of whether the mechanism is the same as ordinary associative learning; variation begs the question of why some people would view a table more positively when paired with decapitated puppies.

Of course, not every question lends itself to "does everyone?"  Questions about preference, for example, will not be universal.  Areas where the does-everyone idea is fruitful include perception, cognition, and social cognition.

The hard part of this question is statistical.  In commonly sized samples, we always observe some people who reverse the effect.  But the real question is whether these reversals are due to sample noise (trials are noisy) or variation in true values.  So, one needs to ask the more nuanced question, "does everyone plausibly?" and use latent variable models, with true and observed values, to answer the question.  The hard part is deciding how to evaluate the evidence because the analyst is assessing whether an ordering holds for each individual simultaneously.  

We were very proud to present the "does everybody plausibly" question and solve the statistical problem.   Our first paper on it was published in Psychological Methods.  

So, fellow methodological terrorists, research parasites, and allies.  Please consider the does-everyone question in your work.  If you need help with the statistics, we are here.


The Struggle Continues,
Jeff and Julia