The failure of complex systems

On Monday, British Petroleum announced that Tony Hayward was going to get his life back, after being the whipping boy for the environmental disaster in the Gulf of Mexico. He visited Congress to say good-bye. And good riddance.

On Thursday, Howard Metzler got his ass chewed because Arlington Cemetery is in as good a shape as a lot of other government agencies.

Military contractors also came to Congress, sounding like they were winning the war by themselves.

But Congress is in a get-tough mood. Tough on Wall Street slime-balls, at least in appearance, or until they pay up. Tough on sloppy government, sucking up to pissed-off voters, dreadfully fearful of those nasty tea-people, just in case they really do have clout in November. Tough on latinos, that one is easy, they don't have voter cards, so screw 'em. Tough on contractors, the only real winners of the War on Terror, thanks George. Congress reacts to incompetence, malfeasance, and catastrophes, the only way they know how. They shoot the messenger. And they have a very good reason.

Listening to Howard Metzler, it only took a few minutes to realize that a complex system had been created to manage his little corner of the world. The OMB determined his budget. Contractors were not under his authority. The chain of command was non-existent. The system he inherited was already broken, and he was only barely in control of any attempt at a solution. Whether this was obvious to the Subcommittee on Contracting Over-site is not known. They were too busy sticking their pencils up Mr. Metzler's ass. It is an election year, and they all knew that this hearing would be front page when the shit hit the fan. It was the lead story on several channels. But Metzler was not an incompetent, and that was immediately clear. He had designed successful systems for the Veteran's Administration. He seemed to understand the problem.

Tony Hayward, banished to the gulags, and grateful for the opportunity, also faced his own inquisition. His only sin was that of wishing he could get back to his nice life, and not have to deal a hole in the ground, but to hear the questions you would have thought he dug that well himself. If pushing a fountain pen in England is a crime, you can hang the bastard. That is about the extent of his responsibility in this disaster.

But when the MMS questioned the people who were on Deepwater Horizon when the well blew out, it was clear that these people knew their jobs, and knew the system. These were not blue-collar dummies. They corrected the interviewers on several fine points, and although at times they seemed defensive, they did not seem incompetent. So who is to blame?

Finally, military contractors sat down for their kaffee-klatch, all full of patriotism, telling a commission on wartime contracting that their folks were doing God's work, even if they were doing the job at three times the amount we would have spent had we just let the DOD do the job. So we cannot find a few billion. It's a big job. Logistics is complicated.  Do you want sleep soundly in your beds, or count your money?

In each instance we have created complex systems that depend on every element working correctly. If it does not work exactly the way it is designed to, the system may or may not fail. That uncertainty is what encourages the users of the system to take chances. Every engineer and system analyst in the country laughed when idiots in the government demanded that this sort of thing never be allowed to happen again, as if we could design cars that never stop running, or space shuttles that never blow up, or airplanes that never crash, or a homeland security system that guarantees we will never be attacked again. There is always a 'mean time before failure', an expression used to estimate the time we have before a widget fails.

There is also a fundamental rule of complex systems that says we may be able to design a thing to make it as efficient as possible, but we cannot design a perfect thing, or a perfect system. The more complex a system becomes, the more we introduce the possibility of failure.

Most of these systems, military logistics, deep-ocean wells, and space shuttles, are complex within themselves. Other are complex because we have created systems to manage them, and those systems are too complex to manage efficiently.

Boring, isn't it? That's why we shoot messengers. It's a helluva lot easier than fixing systems.