Base rate fallacy dooms TSA’s risk-based screening

by Sommer Gentry on May 28, 2012

The TSA has been peddling the concept of risk-based screening vigorously of late. PreCheck is one oft-touted example of risk-based screening; others include behavior detection and modified screenings for the oldest and youngest passengers. The TSA might feel the need to mollify an outraged public, but the public should be only more outraged that the TSA keeps wasting our money on screening methods that can’t possibly work.

Risk-based screening, if it describes any partitioning of airline passengers at checkpoints, is a fraud. It doesn’t matter what the criteria are for putting passengers into low-risk and high-risk categories, because the concept is mathematically unsound.

The base rate fallacy is the failure to account for underlying incidence of the condition being tested for. Imagine this hypothetical (and completely fantastical) scenario: that the TSA employs a near-perfect test for risk-based screening, a test that always flags terrorists if they are present and only flags innocent people 1 time out of 10,000. (The TSA’s actual success rate at flagging known terrorists is zero.)

Under these conditions, only one out of every five million people flagged would be a terrorist. This is the base rate fallacy in operation. In other words, for all intents and purposes, every flagged person is a false positive.

A screener doesn’t handle five million flagged passengers in an entire lifetime. That means that screeners and methods will be biased — even designed — to conclude that all positives are false positives.

If an actual terrorist got flagged in a system like that, he would be dismissed as some sort of false positive. The false positives aren’t just a waste of time and money — they undermine the entire system, because handling so many false positives will alter the thinking of the people operating the system.

It simply cannot work to use universal screening on any population where the incidence of what you’re screening for is less than one in a billion; it’s impossible. No demographic category of airline passengers has a large enough incidence of terrorism to be useful as a “risk-based-screening” subgroup.

The only group that can be reasonably targeted is the group of people who have joined or associated with extremist and violent organizations. Indeed, the only counterterrorism successes we’ve ever seen are those where intelligence and police resources are focused on finding extremists and targeting their plots at the planning stages. Every time a plotter has ever entered an airport, checkpoint screening failed to detect him.

Universal screening of airline passengers to find terrorists is mathematically senseless. You can use universal screening to find common conditions, for example, high blood pressure in people over 50. You can’t use universal or even “risk-based” screening to find one in one billion of anything; the TSA’s effort to do so is the very definition of counterproductive. They’ve managed to turn a huge proportion of the traveling public against them.

The TSA has earned every bit of hostility and uncooperativeness they get from travelers. Treating us as criminals, treating our belongings as expendable, and treating our bodies as meaningless objects has cost the agency dearly. We must continue to oppose the TSA’s dangerous and disrespectful abuses, whether the agency labels them “risk-based” or not.

The TSA knows that risk-based screening won’t make us safer. The term is cynically employed as just another smokescreen, a bit of propaganda to make people believe that the agency is actually doing something “to keep us safe,” when in fact it’s doing nothing of the kind.

(Editor’s Note: Sommer Gentry is a tenured professor of mathematics at the U.S. Naval Academy in Annapolis.)

(Photo: Flickr Creative Commons/Dylan Ng)