Beyond Blame

Chapter 4. The Bad Apples

Bill finished the last of his beer, and carefully set the glass on the table. “I would love to live in a world where we could find root causes for everything. That would be so great. Unfortunately, I live in the real world where things are messy.”

Bill took out a car key from his pocket and placed it in the middle of the table. “Let’s ignore the electronic parts of this key for a second. Imagine it’s a key to an old-fashioned car. We can hold it, we can view it from all sides, and, most importantly, we can reason about the way that it works. We can hold a very simple mental model of the key in our heads.

“Now let’s think about a car. It has a lot more parts than just this key. It’s considerably more complicated than the key, but it’s still possible to know pretty much all there is to know about the car—especially if it’s an antique one, like a Model T. A good mechanic, after working with these cars for a while, could build a pretty realistic mental model of how they function and break down. It would be considerably more nuanced than the mental model for the key, because it might have to deal with external realities like the weather, or the type of fuel that could affect the functioning of the car. But it would match the complexity of the car itself. The way that the mechanic would think about a complicated antique car would be sufficiently complicated.”

“Wouldn’t you say,” Linda interrupted, “that Mike is like an experienced mechanic? He knew all there is to know about routers, right? And that’s what makes his actions so unforgivable.”

“If we only choose to think about routers, disconnected from anything else, then you’re probably right. But the outage we lived through was not just about routers. Our systems are like a whole bunch of Model Ts, and every other kind of vehicle running on a 12-lane highway on the hottest day of the year. Our ability to fully understand how this complex system functions is diminished, as is our ability to control the system.”

“Just because it’s difficult, do we give up trying to understand?” Linda asked. “Let’s say there’s an accident, which causes some traffic to back up on the highway...”

“That’s a great example!” Bill interrupted. “Why did this accident happen?”

“Maybe the driver had too much to drink, or maybe the driver was distracted?”

“Maybe. But why? Why was the driver drunk or distracted?”

“Well, who knows and who cares? All we care about is that we find and prosecute the person who caused it, if that person was breaking the law.”

“If we can establish beyond a shadow of a doubt,” Bill said, “that that person was breaking the law, and punish him—and that’s all we do—then we didn’t go deeply enough to understand the complex system that we’re part of every time we drive. Maybe we’ve found someone to blame for the accident. But we still don’t know how he was able to do it, what were the conditions that allowed the accident to occur. And even if we’ve removed him from the road, we’ve learned nothing that could make driving safer for anyone else who drives in the future.”

“I disagree.” Linda said. “The roads would be much safer for everyone if certain people weren’t allowed to drive.”

“What you’re saying,” Bill said, “is that the cause of the accident and the traffic jam thereafter is this driver. Remove him from the road, and you don’t have that accident and its aftermath. Remove dangerous drivers and our roads automatically get safer. This is such a comfortable story, the story of how there are a few bad apples that we need to prevent from causing trouble. But how do we identify them, especially before they do something bad? Is there a gene for recklessness? Do we take away the driver’s licenses of anyone who has ever been in an accident?”

“That doesn’t sound crazy for drunk drivers.”

“I think we do that already, but I’m not sure it does much to prevent the next drunk driver from getting behind the wheel. If someone decides to get behind the wheel drunk at 4 o’clock in the morning and there are no other cars on the road, would there be an accident? And, conversely, what about the sober, experienced driver who has an accident because of some other condition like poor visibility due to fog, or a car malfunction, or due to the way that an intersection is built?”

Bill continued. “Context is important, Linda. Imagine a man is caught speeding, and also talking on his cell phone. Clearly he’s breaking at least two laws.”

“Yes, and he should be fined for breaking them,” Linda said.

“Now imagine he’s speeding and talking on the cell phone. The policeman who stops him sees that the man’s wife is in the backseat, in labor. The man is rushing to the hospital and is on the phone with the doctor.”

“Well, in that case, he should get a police escort to the hospital.”

“That’s why we don’t have robots to blindly apply the law and dole out punishment. In this case, it would actually be reckless for the police officer to arrest the husband. In this context, the husband is doing something that is reasonable, even though he might be breaking the law. He’s not trying to cause an accident, he just wants to get his wife to the hospital as quickly and safely as possible.”

“I think I see the point you’re trying to make, Bill,” said Ollie. “It has to do with intention. Most people don’t set out to deliberately cause accidents, or for that matter, to take down networks. Most people come to work to do a good job. But what about people who seem to make more mistakes than others, or who repeatedly make the same mistake? And what if, despite their best intentions, they cause harm?”

“The fact that someone who’s drunk even has the ability to start a car,” Bill said, “has a lot to do with the harm that they can cause. What if cars wouldn’t start if their driver was impaired?”

Linda nodded. “Our roads would be a lot safer.”

“They would be,” Bill said. “But instead of learning from the accidents and from the people involved in them—instead of taking these opportunities to understand more fully these complex systems and to make them safer and more resilient—we jump to some quick conclusion, label somebody the root cause, and get rid of them or punish them.”