Painting the Target

A marksman fires at a barn wall, then paints a bullseye around the tightest cluster of holes. He looks like a crack shot. The trick is that the target was drawn after the shots.

Data work the same way. Search a large set of numbers for any pattern and you will find something striking by luck alone. A cluster of illnesses near a factory, a streak of wins in a fund, a set of words in a poem can all look meaningful when the region, period or list was chosen after seeing them.

The protection is to state what you are looking for before you look. A pattern found by checking many possibilities needs a fresh test on new data before it should be trusted.

The lucky cluster

A town notices six cases of a rare illness on one street and blames a nearby pipe. Ira, an analyst, asks how the street was chosen. It was picked because the cases were there.

She draws the map with different boundaries and the cluster vanishes. She suggests testing the pipe idea on fresh data: other towns with similar pipes, and a rule set beforehand for what would count.

Try this today When you see a striking pattern, ask what was decided before the data came in.

More on this fallacy: Texas sharpshooter →