Survivorship bias and the WWII bomber armor story

Survivorship bias is the error of drawing conclusions from the people, companies or objects that made it through a selection process while overlooking those that did not, because the failures are no longer there to be counted. The best-known illustration comes from World War II, when the statistician Abraham Wald reasoned that bombers returning with damage showed where a plane could be hit and still fly. The popular version of the story is more dramatic than the documented record. [1]

Every set of examples has been filtered by something. Companies people have heard of are mostly ones that did not fail, funds in a performance table are the ones still open, and old buildings that still stand are the sturdy ones. If the filter is related to what you are trying to learn, the surviving cases mislead, and they do so with evidence that looks real. As one source puts it, the absent cases are missing because of the very outcome being studied. [1]

It is a form of selection bias in which selection happens on the outcome and the reasoner can see only one side of it. It is hard to notice because missing data do not announce themselves. Correcting it means asking who is not in the data, and why. [1]

The Statistical Research Group at Columbia University, which included Wald, Milton Friedman, Jacob Wolfowitz and Allen Wallis, worked on problems for the US military. One was how to estimate aircraft vulnerability to hits on different parts, which bears on where to add armor. The data were hit counts on planes that returned, plus the number of planes that did not. Armor is a trade-off, since extra weight cuts into bombs, fuel and ammunition. [1] [2]

In a series of memoranda in 1943, Wald started from the point that returning planes were a biased sample, since a fatal hit could not appear in the data. He built a method to estimate the chance that a hit on each region was fatal, using hits on survivors, the number of planes lost and an assumption that hits land in proportion to exposed area. A region with fewer hits on survivors than its area predicts is one where hits tend to be fatal. One source counts eight memoranda; the memoranda were reprinted in 1980 by the Center for Naval Analyses, and Mangel and Samaniego restated the method in 1984. [1] [3]

The popular version says that Wald was shown maps of bullet holes and told the military to armor the places with no holes. Several sources say this is a simplification. The documented work is a technical estimation method, not a single remark, and the sources do not document a particular hit map or an officer proposing to armor the holes. [1] [2]

Accounts differ on the outcome. One popular account says the military applied armor to the areas Wald identified, mainly engines, and that planes then survived more hits. Another source says the memoranda do not record what the military did with them, so the happy ending belongs to the retelling, though it notes the method was later used on Korean and Vietnam war data. The core reasoning, that damage on survivors marks where a plane can be hit and still return, is accepted across the sources. [2] [3]