stories · operations
Four famous process failures, and the one thing they share
Knight Capital's $440 million, an AWS outage from a typo, the crash that invented the checklist, and the checklist that cut surgical deaths. Different industries, the same lesson about documented procedures.
Process failures make the news when the number is big enough. Behind the number there is almost always a step that was written down and not taken, or taken on the wrong branch. Four public cases, told briefly and from the public record, and the thread that runs through them.
Knight Capital, 2012: one server out of eight
Knight Capital was a market maker handling a large share of US equity trading. In August 2012 it deployed new trading software to its servers ahead of a market change. The deployment procedure covered eight servers. On one of them, the new code wasn't installed, and an old, long-dormant routine was reactivated by a reused flag.
When the market opened, that server started sending orders no one intended. In roughly 45 minutes the firm accumulated positions that cost it about $440 million and, within days, its independence. The SEC's later order described the absence of adequate controls around the deployment.
The step existed. The procedure said eight servers. The failure was that nothing checked, at the moment of action, that the step had been completed — and no one could see, in time, that it hadn't.
Amazon S3, February 2017: a command with a wider blast radius than intended
Amazon's own post-event summary is unusually candid. An engineer on the S3 team was following an established playbook to remove a small number of servers from a billing subsystem. One of the command's inputs was entered incorrectly, and a much larger set of servers was removed, including ones supporting critical index subsystems. S3 in the US-East-1 region was effectively unavailable for hours, and a large part of the internet that depended on it went with it.
Amazon's fix wasn't "be more careful". They changed the tool so it removed capacity more slowly and refused to take a subsystem below its minimum, and they broke the subsystems into smaller cells. In other words: the process was followed; the process itself lacked a guard at the step where a human types a number.
Boeing, 1935: the crash that produced the checklist
The story is well told in Atul Gawande's The Checklist Manifesto. In October 1935 the Army Air Corps evaluated Boeing's Model 299, the aircraft that would become the B-17. On its test flight it climbed, stalled and crashed, killing two of the five crew, including the pilot — one of the most experienced in the Army. The cause was a locked elevator control that nobody had released before takeoff.
The aircraft wasn't judged too dangerous to fly; it was judged too complex for one person to hold in their head. A group of pilots produced a short, step-by-step checklist for takeoff, flight, landing and taxiing. The B-17 went on to fly a huge number of hours without a serious accident attributable to pilot error of that kind, and the checklist became part of aviation.
The step — release the elevator lock — was known to every pilot. It was skipped by one of the best. Expertise doesn't protect against the step that's easy to forget; a visible, at-the-moment prompt does.
The WHO surgical safety checklist, 2008: the success case
Gawande's own experiment. A 19-item checklist, used at three moments in an operation (before anaesthesia, before incision, before the patient leaves the room), piloted in eight hospitals across rich and poor countries. The published result: major complications fell by about a third and deaths by nearly half across the sites. Nothing on the list was unknown to the surgical teams. Confirming the patient's identity, checking the antibiotic was given, counting the sponges — all documented, all "obvious", all skipped often enough to kill people.
What made it work wasn't the list. It was that the list was read aloud, at the moment, by a named person, and that every team member was expected to speak up. Adherence became visible inside the room.
What they share
- The step was documented. In every case, the procedure existed and was known.
- The failure was at the moment of action, not at the moment of writing. Distance between the document and the hands doing the work.
- Nobody could see, in time, that the step was missed. Knight found out from the market. Amazon found out from the internet. The B-17 crew found out at 300 feet.
- The fix was never "try harder". It was a guard at the step (Amazon), a prompt at the moment (the checklist), and a way to see adherence (the operating-room read-aloud).
Support and engineering teams don't crash aircraft, but they run the same shape of process every day: documented, branching, dependent on facts about the item, and invisible when skipped. The fixes are also the same shape: bring the step to the work, ask for the fact before acting, and make adherence visible so the document can improve. That is what process guidance means in practice.
Sources: SEC order against Knight Capital Americas LLC (2013); Amazon's "Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region" (2017); Atul Gawande, The Checklist Manifesto (2009); Haynes et al., "A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population", NEJM (2009).