Monday, November 19, 2018

Preregistration: Try it (Or not)

So, as the Statistical War and Tone War are in a lull, the Preregistration conflict has flared up yet again.   A few thoughts on the airplane back home from Psychonomics.


A. To be honest, it has taken me quite a long time to sort out my thoughts on preregistration.   I am not telling you to preregister or not.  Moreover, how I read your work is not dependent on whether you preregistered or not.  Perhaps you might find my thoughts helpful in your decision; perhaps not.

B. I don't believe in the usefulness of the exploratory/confirmatory distinction.  All of my research is motivated by some theoretical issue (so it is not exploratory) and I am always open to alternatives that I have not considered (so it is not confirmatory).  Arguments that rely on the exploratory/confirmatory distinction are not persuasive to me, and I will not be using them here (or elsewhere).

Why I Preregistered, the story:

I used preregistration because my students forced me to.  I found the experience rewarding and will preregister again.  Perhaps the strongest argument for preregistration is that it may clarify the researcher's thinking before seeing the data.   I think most of us can agree that writing is hard, and one of the reasons it is hard is that it forces you to clarify your thinking on things.   Preregistration in some sense provides the opportunity for that type of clarification before the data are collected.    As we wrote the preregistration, my team realized we hadn't though enough about what type of models could instantiate one of the theoretical alternatives.  So, we made a set of additional model specifications before seeing the data.  That was quite helpful.

Why I don't Take the Preregistration Too Seriously:

I feel no hesitation to break my preregistration.  In fact, I do not know if we did or did not break our preregistration because I never went back and read it!  I don't care if we did or not, to be honest.   I actually think this is not such a bad idea.  Here is why:

As Shiffrin notes, science requires good judgment.  In fact, being open-minded, flexible, and judicious are probably more important characteristics than being smart, industrious, or meticulous.  Now, what I like about preregistration is that it summons me to provide my best judgment at a particular point in time.  But, as new information come in, including data, I need to exercise good judgment yet again.  Hopefully, the previous efforts will make the current efforts easier and more successful.  But that is where it stops.   I will have no contract with my preregistration; instead I will use good judgment.  Preregistration is used to improve the pre-data steps, and hopefully that will improve post-data steps too.

So if you preregister, consider the following:

1. Try not to substitute your preregistration for your best judgment.  You can add value judiciously.  Don't trade in what you know now for what you knew then.

2. Don't forget to have a conversation with your data.  Nature only whispers, you need to communicate with her softly and subtly.  You gently ask one thing, it whispers something else.  And you go back and forth.  Please do not downgrade this conversation because it might be the most important thing you do with your data.

If You Want Others To Preregister:

Tell your story.   Maybe in detail.  What might you have done differently?

No comments: