Sunday, May 17, 2015

To Better Know A Bayesian

The Self-Propagated Myth of Bayesian Unity

Substantive psychologists are really uncomfortable with disagreements in the methodological and statistical communities.  The reason is clear enough----substantive psychologists by-and-large just want to follow the rules and get on with it.

Although our substantive colleagues would prefer if we had unified and uniform set of rules, we methodologists don't abide.  Statistics and methodology are varied fields with important, different points of view that need to be read, understood, and discussed.

Bayesian thought itself is not uniform,.  There are critical, deep, and important differences among us, so much so that behind closed doors we have sharp and negative opinions about what others advocate,  Yet, at least int he psychological press, we have been fairly tame and reticent to critique each other.   We fear our rule-seeking substantive colleagues may use these differences as an excuse to ignore Bayesian methods altogether.  That would be a shame.

In what follows, I give the briefest and most coarsest description to the types of Bayesians out there.  In the interest of being brief and coarse, I am going to do some points-of-view an injustice.  Write me a nice comment if you want to point it out a particular injustice.  My hope is simply to do more good than harm.

Also, I am not taking names.  You all know who you are:

Strategic vs. Complete Bayesians:

The first and most important dimension of difference is whether one uses Bayes Rule completely or strategically.  

Complete Bayesians are those that use Bayes rule always, usually in the form of Bayes factors.  They are willing to place probabilities on models themselves and use Bayes rule to update these probabilities in light of data.  The outline of the endeavor is that theories naturally predict constraint in data which are captured by models.  Model comparison provides a mean of assessing competing theoretical statements of constraint, and the appropriate model comparison is by Bayes factors or posterior odds.  In this view, models predict relations among observables and parameters are convenient devices to make conditional statements about these relations.  Statements about theories are made based on predictions about data rather than about parameter values.  This usage follows immediately and naturally from Bayes rule.

Strategic Bayesians are those that use Bayes rule for updating parameters and related quantities, but not for updating beliefs about models themselves.  In this view, parameters and their estimates become the quantities of interest, and the resultants are naturally interpretable in theoretical contexts. These Bayesians stress highest density regions, posterior predictive p-values, and estimation precision.  Strategic Bayesians may argue that the level of specification needed for Bayes factors is difficult to justify in practice especially given the attractiveness of estimation.

The Difference: The difference between Complete and Strategic Bayesians may sound small, but it is quite large.  At stake are the very premise of why we model, what a model is, how it relates to data, what counts as evidence, and what are the roles of parameters and predictions.  Some statisticians, philosophers, and psychologists take these elements very seriously.  I am not sure anyone is willing to die on a hill in battle for these positions, but maybe.

I would argue that the difference between Complete and Strategic Bayesians is the most important one in understanding the diversity of Bayesian thought in the social sciences.   It is also the most difficult and the most papered over.

Subjectivity vs. Objectivity in Analysis

The nature of subjectivity is debated in the Bayesian community.  I have broken out here a few positions that might be helpful.

Subjective Bayesians ask analysts to query their beliefs and represent them as probability statements on parameters and models as part of the process of model specification.  For example, if a researcher believes that an effect should be small in size and positive, they may place a normal on effect size centered at .3 with a standard deviation of .2.  This prior would then provide constraint for posterior beliefs.

A variant to the subjective approach is to consider the beliefs of a generic, reasonable analyst rather than personal beliefs. For example, I might personally have no faith in a finding (or, in my case, most findings), yet I still may assign probabilities to parameters and hypotheses values that I think capture what a reasonable colleague might feel.  This process is familiar and natural---we routinely take the position of others in professional communication.

Objective Bayesians stipulate desirably properties of posteriors and updating factors and choose priors that insure these desired properties hold.  A simple example might be that in the large-sample limit, the Bayesian posterior of a parameter should converge to a true value.  Such a desirada would necessitate priors that have certain support, say all positive reals for a variance parameter or all values between 0 and 1 for a probability parameter.

There are more subtle examples.  Consider a comparison of a null model vs. an alternative model.  It may be desirable to place the following constraint on the Bayes factor.  As the t-value increases without bound, the Bayes factor should favor without bound the alternative.  This constraint is met if a Cauchy prior is placed on effect size, but it is not met if a normal prior is placed on effect size.

There are many other desiderata that have been proposed to place constraints on priors in a variety of situations, and understanding these desiderata and their consequences remains the topic of objective Bayesian development.

The Difference:

My own view is that there is not as much difference between the objective and subjective points of view as there might seem.

1.  Almost all objective criteria yield flexibility that still needs to be subjectively nailed down.  For example, if one uses a Cauchy prior on effect size, one still needs to specify a scale setting.  This specification is subjective.

2.  Objective Bayesian statisticians often value substantive information and are eager to incorporate it when available.  The call to use desiderata is usually made in the absence of such substantive information.

3.  Most subjective Bayesians understand that the desiderata are useful as constraints and most subjective priors adopt some of these properties.

4.  My colleagues and I try to merge and balance subjective and objective considerations in our default priors.  We think these are broadly though not universally useful.  We always recommend they be tuned to reflect reasoned beliefs about phenomena under consideration.  People who accuse us as being too objective may be surprised by the degree of subjectivity we recommend; those who accuse us as being too subjective may be surprised by the desiderata we follow.

Take Home

Bayesians do disagree over when and how to apply Bayes rule, and these disagreements are critical.  They also disagree about the role of belief and more objectively-defined desiderata, but these disagreements seem more overstated, especially in light of the disagreements over how and when Bayes rule should be used.  


John K. Kruschke said...

Hi Jeff.

No "injustice out-pointing" here, just discussion.

Seems to me that it's impossible to be a "Complete" Bayesian. To be a Complete Bayesian, wouldn't every analysis have to begin with all possible models? In reality, all analyses are conditional on some specified family of models, with a finite set of discrete and/or continuous parameters and their priors. Doesn't that make all Bayesian analysis "Strategic" not "Complete"?


Jeff Rouder said...

Hi John, My goal was to come up with terms that were positive rather than derogatory, and yet captured the differences transparently and straightforwardly. I hope the emphasis is on the distinctions more so than the absolute accuracy. I had never thought about your point. Thanks.

I guess a complete Bayesian may still put zero prior probability on models that are not theoretically important. With this, she or he may still compare the ones that are important (as well as estimates parameters in these models). Strategic Bayesians avoid placing probabilities on models themselves.

Again, my explicit goal here is to avoid loaded terms like "orthodox" or "partial." Complete and strategic both sounded positive to me. Hope you read it as so.

Best, Jeff