Root Cause

Show notes

When a serious system fails, the engineers who keep it running do something deceptively ordinary: they write it down. The post-mortem is a hard-won discipline for reconstructing what actually happened — not a verdict but a record. It lays out a timeline, the evidence that fixes each moment, the contributing factors that lined up to make the failure possible, and the decisions people made with the information available at the time. That last clause is the whole craft. A good post-mortem refuses hindsight; it asks what a reasonable operator knew at 03:14 when the page fired, not what an omniscient reader knows after the fact. The goal is a reconstruction faithful enough that someone who was not there could replay the failure and understand why each step seemed right when it was taken.

Blamelessness is not politeness. It is an evidentiary safeguard. People hide what they fear punishment for, and a record built on hidden steps is worthless — it will be confidently wrong about the very moments that matter most. The blameless post-mortem exists to keep the account honest by removing the incentive to lie: name the failure, never the failing person, and you get the truth about the path. Strip that discipline away and the record fills with omissions and self-serving edits; the timeline survives but the causes do not. So the convention that looks soft is doing the hardest evidentiary work in the room — it is the rule that lets the reconstruction be trusted at all.

Set that beside how most automated systems handle their own failures. They emit an outcome — a score, a decision, a label — and discard the path that produced it. There is no timeline, no preserved evidence, no record of which factors contributed or what was known when. The answer arrives without its reasons, and when it turns out to be wrong there is nothing to reconstruct, because the steps were never kept. A post-mortem culture and a discard-the-path system are opposite theories of failure: one treats the path as the thing worth saving, the other treats it as exhaust. This show keeps returning to a single claim — that a decision is legitimate not because it was reached but because it can be reconstructed and replayed, the way a careful adversary could walk each step and challenge it. Incident review is that claim already operating in the wild. The book this series accompanies, Admissible Reality, calls the property these post-mortems protect legible failure: a system fails well when its failure can be read, traced, and argued with — and fails worst when it fails silently, leaving an outcome and no path.

So the conversation asks a site-reliability engineer to treat the post-mortem as more than an operations ritual — to take the actual practice, its timelines and contributing-factors analyses and its insistence on what was knowable at the time, and hold it against systems that report conclusions with no reconstructable history. What does a blameless review preserve that an automated decision pipeline throws away? What would it take to make a machine's failure as legible as a well-run incident report — replayable, contestable, honest about the moment of the mistake? The wager of the episode is that engineers running real systems have already built much of the accountability machinery that machine reasoning still pretends it can do without. Continue the argument in Dispatches, where the show's longer threads live between episodes.