tag:blogger.com,1999:blog-1754053998399059707.post3262184371443702102..comments2017-06-10T05:54:19.578-05:00Comments on Invariances: The Bayesian Guarantee And Optional Stopping.Jeff Rouderhttp://www.blogger.com/profile/12042232118911308833noreply@blogger.comBlogger12125tag:blogger.com,1999:blog-1754053998399059707.post-11173976697942010372016-04-06T14:39:44.957-05:002016-04-06T14:39:44.957-05:00The "data integrity problem" is associat...The "data integrity problem" is associated with a relatively simple quantitative rule, I think. The sampling plan matters for Bayesians iff it affects the kernel of the likelihood (and matters for frequentists even if it doesn't). Some censoring rules will have that effect.Joachim Vandekerckhovehttp://www.blogger.com/profile/00457442016473274970noreply@blogger.comtag:blogger.com,1999:blog-1754053998399059707.post-60030593204108554902016-04-06T12:23:22.863-05:002016-04-06T12:23:22.863-05:00Yeah, I think we are there. Thanks for the commen...Yeah, I think we are there. Thanks for the comments---I learned a lot. I think I would write the blog post a bit differently if I could do it over again. JeffJeff Rouderhttp://www.blogger.com/profile/12042232118911308833noreply@blogger.comtag:blogger.com,1999:blog-1754053998399059707.post-7041746669356729622016-04-06T11:37:08.298-05:002016-04-06T11:37:08.298-05:00I agree with all of the above. I don't see the...I agree with all of the above. I don't see the issue of sampling scheme (which I agree can be thought of as a form of model specification) as being intrinsically related to the choice of frequentist vs. Bayesian statistical methods. Which is why I object to the argument that optional stopping is irrelevant if you're a Bayesian. It's true that the <em>machinery</em> of Bayesian inference works just fine no matter what data you feed it (whereas the frequentist machinery won't always), but that doesn't mean that a researcher who's using Bayesian inference doesn't need to think about the sampling scheme. The fact that one camera might work perfectly well underwater while another one doesn't is no guarantee that the photographer is going to get the best shot of the beach from below the water line--and depending on many other factors, it could turn out that the less resilient instrument produces more useful results.<br /><br />I agree that people should be able to explain why they're using whatever approach they're using.Tal Yarkonihttp://talyarkoni.orgnoreply@blogger.comtag:blogger.com,1999:blog-1754053998399059707.post-51845264728689815602016-04-06T09:38:40.692-05:002016-04-06T09:38:40.692-05:00Well, now get your excellent point. It is a stron...Well, now get your excellent point. It is a strong challenge. We can make it even stronger. Suppose an unscrupulous researcher collected data and changed all the negatively-vaued observations to positively-valued observations. I could simulate data from this process, and I can even show that given a set of observations, Bayes' Theorem accurately describes the true values that generated these data. Now your point is "so what." We have a data integrity problem here and the fact that Bayes' Theorem works is at best inconsequential. <br /><br />My attention is captured and my thinking could stand some sharpening. Thanks.<br /><br />Some points:<br /><br />1. If your question is how well you capture a hypothetical truth should it hold, by all means please use frequentist methods. In most cases they are tuned in efficiency and bias, we have a lot good sense about how they behave, and they control long-term error rates well. <br /><br />2. If you choose Bayesian methods because of the frequentist properties, say so. Say I am a frequentist and I am using Bayesian because it is more tractable and in some cases. Also some people choose certain prior because they have excellent frequentist properties. This is fine, but, in my view, at heart this is frequentist endeavor with Bayes used as a convenient computational engine. But let's be honest about the goal. Is it about updating beliefs or about error control for select hypotheticals. Mine is the former.<br /><br />3. I think the issue with the unscrupulous researcher who changes the sign of negative values is a model issue rather than an analysis issue. The Bayesian research is learning exactly about the parameters in the model, but the model is not well describing the data-generation and fraud process. What we need (in addition to honest, open reporting) are more useful models. Model specification to me is a substantive issue not a statistical one. Our inference about parameters is only as good as our models. That point probably needs to be said a few times because most researchers reify these estimates as properties of the world. To me, they are collectively fabricated.<br /><br />Jeff Rouderhttp://www.blogger.com/profile/12042232118911308833noreply@blogger.comtag:blogger.com,1999:blog-1754053998399059707.post-20562976963142013802016-04-06T09:08:14.069-05:002016-04-06T09:08:14.069-05:00Thanks a lot for posting the R code! Much apprecia...Thanks a lot for posting the R code! Much appreciated. Felix Thoemmeshttp://www.blogger.com/profile/02139944750643002756noreply@blogger.comtag:blogger.com,1999:blog-1754053998399059707.post-13400574140984757502016-04-06T06:23:16.219-05:002016-04-06T06:23:16.219-05:00Tal, the question Jeff answers here is not one tha...Tal, the question Jeff answers here is not one that "nobody cares about". It's a question that scientists using data in order to learn about the truths of their beliefs *should* care about.JP de Ruiterhttp://www.blogger.com/profile/09640729547040033538noreply@blogger.comtag:blogger.com,1999:blog-1754053998399059707.post-51954778773524618672016-04-06T00:35:25.333-05:002016-04-06T00:35:25.333-05:00Jeff, I think I understand your goals, and my poin...Jeff, I think I understand your goals, and my point is basically the same as John's: you're answering a question that nobody really cares about. And I don't actually think it's a frequentist/Bayesian split in this case; I suspect that many people who call themselves Bayesians would also conclude that the parameter estimates are biased in a perfectly meaningful sense if you use optional stopping. The fact that Bayesians are trying to infer plausible values from data and not the other way around doesn't mean that a Bayesian should reject the idea that there is a single asymptotic parameter estimate that one would converge on given all of the data in the world. It should be clear from your setup that if you use an optional stopping rule like the one you propose, you will never converge on the true parameter estimate no matter how much data you collect. Now, you can insist that this idea of convergence on some true parameter value in the long run is a frequentist notion if you like, but my point (and John's too, I think) was that if you do that, the main consequence will be to push people away from Bayesian inference, because it will just seem to most people like Bayesian inference can't tell them what they want to know (which is not true--it <em>can</em> tell them what they want to know, provided one accepts that the stopping rule does matter).<br /><br />In effect, what you're saying is: given any prior, any data, and any sampling plan, Bayes will give you the optimal answer. Well, sure. But that's not what researchers want; I have a hard time seeing many people wanting a posterior estimate that depends on the sampling plan! I mean, suppose I decided that my sampling plan is to visually inspect my results every 10 subjects and drop all the observations that don't fit with my hypothesis. Strictly speaking, you would still be correct to say that "Bayes Theorem tells you where the true values are given your prior and the data, and it doesn't matter how the data were sampled". But this would be a completely useless analysis; nobody in their right mind would trust the resulting parameter estimate--not because of Bayes (or lack thereof), but because it's a stupid and clearly biased stopping rule. The same is true (though to a lesser degree) of the stopping rule in your example. Nobody is going to care if your estimate is optimal conditional on some really bad decisions if the upshot is that no amount of data will ever lead to the asymptotically correct estimate you'd get from sampling the entire population in one shot.Tal Yarkonihttp://talyarkoni.orgnoreply@blogger.comtag:blogger.com,1999:blog-1754053998399059707.post-30848369141377615422016-04-05T23:26:45.932-05:002016-04-05T23:26:45.932-05:00John, Thanks for your comments. I think the quest...John, Thanks for your comments. I think the question asked depends on whether the analyst is Bayesian or frequentist. I am Bayesian. I ask, given my data, what are my beliefs about the plausibility of different truth statements. Bayes' Theorem answers this question. I am heartened to know that Bayes' Theorem holds regardless of how the data were collected. <br /><br />The context for the blog is a simulation on Facebook by John Cleavenger who simulated Bayesian estimation from a single ground truth value as in the top analysis. The claim is that Bayesian estimation is problematic. I replied then that it is problematic if you interpret the Bayesian quantities as a frequentist rather than as a Bayesian. So here I offer what I consider to be the proper Bayesian interpretation. Jeff Rouderhttp://www.blogger.com/profile/12042232118911308833noreply@blogger.comtag:blogger.com,1999:blog-1754053998399059707.post-57593166853210513462016-04-05T23:22:36.629-05:002016-04-05T23:22:36.629-05:00Tal, I am having a hard time understanding your q...Tal, I am having a hard time understanding your question/points. I think perhaps you misunderstand my goals. Sometimes people simulate how optional stopping affects Bayesian quantities by simulating data from known truth and then plotting histograms of the resulting Bayesian estimates and seeing how they deviate from the ground truth. Bayesian queries are not focused on this conditional but on the reverse one. What can we learn about the plausibility of various true values conditional on data (and our prior). So, in simulation, you need to condition on some data value, I chose .3 but analogous results hold for any value. I collected all the truth values that produced a sample mean near .3. I used the simulations to construct one distribution of true values conditional on some sample mean, and then I showed that Bayes Theorem describes this distribution. And this good description always holds, which is, well, quite neat. In practice, you would have a single sample mean---the result of your experiment---and you would like to know the plausibility of true values. Bayes Theorem well describes these plausibilities even optional stopping. And that too, is quite neat.Jeff Rouderhttp://www.blogger.com/profile/12042232118911308833noreply@blogger.comtag:blogger.com,1999:blog-1754053998399059707.post-44458937125385210622016-04-05T19:54:30.986-05:002016-04-05T19:54:30.986-05:00This demo and its underlying math are clear, but t...This demo and its underlying math are clear, but this Bayesian framing isn't answering the question that I think is being asked. <br /><br />The question is not, For a specific data value, what posterior parameter distribution is inferred? (And is that parameter distribution biased relative to the specific data value?)<br /><br />The question is, When the Bayesian posterior distributions for each data value are integrated across <i>all</i> the sample data sets, is that <i>aggregate</i> distribution biased relative to the known generating prior?<br /><br />I don't think it helps the asker of that question to tell them they shouldn't ask that question, or that it's not a Bayesian question. They still want an answer to the question.<br />John K. Kruschkehttp://www.blogger.com/profile/17323153789716653784noreply@blogger.comtag:blogger.com,1999:blog-1754053998399059707.post-54338743686306897612016-04-05T19:05:18.203-05:002016-04-05T19:05:18.203-05:00"I don't understand the rationale for con..."I don't understand the rationale for conditioning your analysis on a sample mean of around .3." <br />The point of a Bayesian analysis is to infer which values of the parameter plausibly led to the actually observed data (represented by the posterior). In this example the actually observed data is .3, so that's what the posterior is conditioned on. <br />Alexander Etzhttp://www.blogger.com/profile/07682503927955600974noreply@blogger.comtag:blogger.com,1999:blog-1754053998399059707.post-27611207577730554692016-04-05T16:49:35.254-05:002016-04-05T16:49:35.254-05:00I don't understand the rationale for condition...I don't understand the rationale for conditioning your analysis on a sample mean of around .3. In the real world, it would be rather un-Bayesian to condition this way if you believed that all samples were being generated by the same underlying process (which they are in your simulation). Surely samples with a mean other than .3 are just as useful in updating our estimate of the "true" parameter.<br /><br />The salient question is whether, in your optional stopping simulation, the posterior distribution <em>over all samples</em> (i.e., not just ones selected post-hoc based on the sample mean) matches the prior distribution. Ideally, it should given that you're sampling the true population values according to the prior. But it won't in this case, because of the bias introduced by the optional stopping rule. Of course, you're welcome to argue that bias is a frequentist property that Bayesians shouldn't care about it, but I suspect that in that case your simulation results will look more like a modus tollens than a modus ponens to most people.Talhttp://talyarkoni.orgnoreply@blogger.com