Organizational Pathology

Did you see the ACLU press release today about how the American terrorist watch list now has more than one million people on it?

“America’s new million record watch list is a perfect symbol for what’s wrong with this administration’s approach to security: it’s unfair, out-of-control, a waste of resources, treats the rights of the innocent as an afterthought, and is a very real impediment in the lives of millions of travelers in this country,” said Barry Steinhardt, director of the ACLU Technology and Liberty Program. “It must be fixed without delay.”

“Putting a million names on a watch list is a guarantee that the list will do more harm than good by interfering with the travel of innocent people and wasting huge amounts of our limited security resources on bureaucratic wheel-spinning,” said Steinhardt. “I doubt this thing would even be effective at catching a real terrorist.”

So, what does this mean?

Obviously this is actually detrimental to security. Consider the cost of the false positives: how many of these two million people might actually be terrorists, or have information about terrorist actions–maybe 10? 100? 1000? Certainly not a million. So when some of the two million people who fly every day in the US overlap with the million people on the list there will be false positives–and lots of them. Which means that all kinds of resources are wasted responding to non-threats (setting aside the inconvenience and denial of basic mobility rights for the people so affected) and that the security people are essentially trained to ignore the results–if the system ever does flag an actual terrorist, everyone who is supposed to respond will have already gone through the routine thousands of times for false positives, and will hardly be responding on high alert.

Think also about wheat and chaff. If you’re tracking information on 1000 people, you can keep on top of it, and see some patterns. If you’re tracking information on 1000000 people then it’s that much easier for a valid signal to be lost in the chaff. (I kind of wish that the TIA types would relax their jackboot dream of the panopticon and focus instead on a much more narrowly targetted approach.)

One interesting aspect of this whole thing is the question “Why is the list growing so fast?”, which might be better considered as “Why do so many people get added, and why is no one ever taken off?”

The answer is an obvious case of organizational pressure causing the opposite result from what is intended: the organization wants to identify terrorists and stop loss of life/property, but the way they are structured inevitably results in something like this which makes that goal less likely (and which, incidentally, results in massive resource waste and liberty violation). Think about it: what are the costs, incentives, and potential consequences, to a federal employee to put someone on the list, or to take someone off the list. The removing part is easy: there is no incentive for an employee to remove anyone, but there is a very high disincentive in that removing someone who later turns out to be a terrorist (no matter how infinitesmally small this possibility is) is both career ending, and will result in a highly undesirable outcome. The same thing in reverse plays into people getting on the list–if you believe that putting somone on the actually list lowers the chance of an attack, and has no real cost, but not putting someone on it might result in an attack, then you just keep adding everyone. Hell, if you believe that it makes sense to add your mom to the list.

The problem is manifold. Part of it is that people don’t understand the full costs of putting someone on the list–there’s both a training and a scale problem there. The other part though, the part that drives security professionals nuts, is that the human brain is hardwired with certain cognitive biases that cause us to be arational about risk. Loss aversion is the big one here, causing us to more easily be blind to the true costs of things we think are preventing a loss.

Still, all my blather aside, I hope there’s no one who is looking at this news and thinking that this list is in any way making them safer. It isn’t, and it can’t be. And even if it did to some small degree, it wouldn’t be worth the cost, in either resources, time, or liberty.

Creative Commons Attribution-NonCommercial-ShareAlike 2.5 Canada
This work by Chris McLaren is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 Canada.