Replication

Show notes

Science earns the right to call something knowledge only after the result survives being produced again — by other hands, in another lab, from the same setup. That is what makes the reproducibility crisis so disquieting: a large share of published findings, across fields, turn out not to reproduce when someone actually tries. The usual diagnosis points at statistics — underpowered studies, flexible analysis, the quiet hunt for a significant p-value. But the deeper issue is not that the numbers are wrong. It is that a finding which cannot be reproduced has never cleared the bar that would make it admissible in the first place. It is a claim, often an accurate-looking one, that no one has yet been allowed to test. A result no one can reproduce is a rumor with a p-value.

The reforms that the field has converged on are, read closely, a discipline of provenance. Preregistration fixes the question and the analysis before the data arrive, so the hypothesis cannot be quietly rewritten to fit the result — it is a commitment recorded in advance, the way a chain of custody records who touched the evidence and when. Sharing the data and the code does the rest: it turns a printed conclusion back into the inputs and the steps that produced it, so that the path from raw observation to reported number is open to inspection. Each of these is the scientific analog of a reconstructable record — not a louder assertion that the finding is real, but the materials a skeptic would need to find out for themselves.

And the strongest check of all is not a better explanation; it is a replay — taking the same data and the same code and re-running the analysis to see whether the same number comes out. A study whose author can hand you the inputs and the procedure, and whose result reappears when you execute it yourself, has done something a beautifully argued discussion section never can: it has made its conclusion contestable on equal terms. This is the same move the rest of the show keeps returning to. Legitimacy does not come from the prestige of the journal or the authority of the lab. It comes from the fact that the route to the result can be rebuilt and checked — replays, not explanations. The book this series accompanies, Admissible Reality, is built on exactly that standard: a conclusion counts when its warrant survives reconstruction.

What makes the unreproducible result genuinely dangerous is that it does not look broken. It is accurate-looking — a clean effect, a tidy figure, a confident abstract — and that surface plausibility is precisely what lets it travel. It gets cited, it shapes the next grant, it becomes the premise other work is stacked upon, all before anyone has confirmed it holds. The conversation puts this to a metascientist: if the cure for the crisis is provenance and replay rather than rhetoric, then reproducibility is not a hygiene step at the end of science but the moment a claim actually becomes evidence — and everything that cannot pass it, however polished, is still only a rumor with a p-value.