Cop-outs or checklists?

[This entry has been amended 5/30/2008]

A seeming contradiction has been nagging me these past several months. I like checklists detailing standard operating procedures. Follow them and you win. So many of you ignore me, just certain you’re the one exception to the rule. Well, maybe you are. Or maybe not. Sort this out for me please.

By way of my first example is a recent story called The case of Mistaken Lemonade. Scott Simon reflects on the story of Professor of Archeology Chris Ratte, who inadvertently bought his young son a bottle of alcoholic hard lemonade at a baseball game. Authorities took his kids away from him for a couple of weeks. Any errors of transcription are mine; emphasis is also mine.

The police hated to do it but they had to follow procedure, put him in a foster home, county bureaucrats said they hated to do it but they had to follow procedure. Three days later, a county judge said the boy could go home but the professor had to move into a hotel. He said he hated to do it but he had to follow procedure. Case has now been dismissed after two weeks of anxiety separation and humiliation.

At each step, informed and responsible officials, police officers, social workers and judges said, they hated to do what they did but had to obey procedures. Procedures are what people […] cite when they fear to take responsibility for an independent decision. Decisions can be criticized and second guessed. Procedures may be dumb but they spare you from thinking or being criticized from the consequences of your own judgment. And to be fair, procedures are often imposed because previous officials have been lax and let a child go back to an abusive household. A story like this doesn’t usually get much coverage on NPR where, we try to analyze policy implications but you might want to remember what happened to the Ratta family the next time a poll discloses that the American people distrust bureaucracies, public or private whether they run schools, airlines or health care systems. They abide by procedures not people. They take lemons and just make a mess.


General reactions from all parties concerned say

Almost everyone Chris Ratte met the night they took Leo away conceded the state was probably overreacting.

The sympathetic cop who interviewed Ratte and his son at the hospital said she was convinced what happened had been an accident, but that her supervisor was insisting the matter be referred to Child Protective Services. And Ratte thought the two child protection workers who came to take Leo away seemed more annoyed with the police than with him. “This is so unnecessary,” one told Ratte before driving away with his son.

But there was really nothing any of them could do, they all said. They were just adhering to protocol, following orders. And so what had begun as an outing to the ballpark ended with Leo crying himself to sleep in front of a television inside the Child Protective Services building, and Ratte and his wife standing on the sidewalk outside, wondering when they’d see their little boy again.

It was still worse though.

But that doesn’t explain why CPS refused to release Leo to the custody of two aunts — one a social worker and licensed foster parent — who drove all night from New England to take custody of their nephew. Chris Ratte’s sisters, Catherine Miller and Felicity Ratte, left Massachusetts at 10:30 the night of the fateful lemonade purchase after the police officer who’d reluctantly requested a removal order told Ratte the state would likely jump at the chance to place Leo with responsible relatives. But when the two women arrived at the CPS office early Sunday, a caseworker explained they would not be allowed to see Leo until they had secured a hotel room. The sisters quickly complied. But by the time they returned to CPS around 10:30 a.m., their nephew had been taken to an undisclosed foster home, where he would remain until a preliminary court hearing the following afternoon.

I think you get the picture. As Scott Simon said, procedures can be used as ways to avoid taking “responsibility for an independent decision. Decisions can be criticized and second guessed. Procedures may be dumb but they spare you from thinking or being criticized from the consequences of your own judgment”.

But here’s the other side of procedures and checklists. In The Checklist (New Yorker magazine, so well written it was the singular impetus for my subscription), who can say whether the Allies would have won WW2 without them?

A small crowd of Army brass and manufacturing executives watched as the Model 299 test plane taxied onto the runway. It was sleek and impressive, with a hundred-and-three-foot wingspan and four engines jutting out from the wings, rather than the usual two. The plane roared down the tarmac, lifted off smoothly, and climbed sharply to three hundred feet. Then it stalled, turned on one wing, and crashed in a fiery explosion. Two of the five crew members died, including the pilot, Major Ployer P. Hill.

An investigation revealed that nothing mechanical had gone wrong. The crash had been due to “pilot error,” the report said. Substantially more complex than previous aircraft, the new plane required the pilot to attend to the four engines, a retractable landing gear, new wing flaps, electric trim tabs that needed adjustment to maintain control at different airspeeds, and constant-speed propellers whose pitch had to be regulated with hydraulic controls, among other features. While doing all this, Hill had forgotten to release a new locking mechanism on the elevator and rudder controls. The Boeing model was deemed, as a newspaper put it, “too much airplane for one man to fly.” The Army Air Corps declared Douglas’s smaller design the winner. Boeing nearly went bankrupt.

Still, the Army purchased a few aircraft from Boeing as test planes, and some insiders remained convinced that the aircraft was flyable. So a group of test pilots got together and considered what to do.

…the pilots went on to fly the Model 299 a total of 1.8 million miles without one accident. The Army ultimately ordered almost thirteen thousand of the aircraft, which it dubbed the B-17. And, because flying the behemoth was now possible, the Army gained a decisive air advantage in the Second World War which enabled its devastating bombing campaign across Nazi Germany.

So what salvaged the B-17 and possibly the war? It was the development of the preflight checklist. It was a checklist that allowed pilots to manage this feat of technology. The article says flying in those days was

…a culture defined by how unbelievably dangerous the job was. Test pilots strapped themselves into machines of barely controlled power and complexity, and a quarter of them were killed on the job. The pilots had to have focus, daring, wits, and an ability to improvise—the right stuff. But as knowledge of how to control the risks of flying accumulated—as checklists and flight simulators became more prevalent and sophisticated—the danger diminished, values of safety and conscientiousness prevailed, and the rock-star status of the test pilots was gone.

Ahhh Rock Stars. Finally we get to designers. The idea that designers are rock star pilots should die quickly too. Making it in design is only dangerous if you make it so, swinging on a trapeze without a net. Going by the numbers, it’s safe and pragmatic. Success is not an affect of bravery but of prudence. Prudence = Rational + Predictable. That’s how your suppliers, retailers and contractors pick their partners.

From what I can tell from a cursory examination -as I said, I’m still trying to resolve these contradictions- the only difference between the case of bad lemonade and the fighter pilot’s checklist, is one is mandatory. The other are guidelines that can be arbitrarily ignored or employed depending on whether one chooses (or not) to make an independent decision. One set is a definitive, quantified, proven must-do. The other is arbitrary, subject to the vagaries of questionable independent motivations. Does this boil down to differences between hard and soft science? Is it a simple matter of technology vs. human?

Here’s another contradiction. From an article in the NYT (Can You Become a Creature of New Habits?) Dawna Markova, an “executive change consultant” has discovered the power of Kaizen, making small incremental change to dramatically improve operations. Yet if you go to her blog, this is what she says (it annoyed me):

America’s greatness was derived from our capacity to think imaginatively. We have drawn on our capacity for ingenuity and innovation. Standardization does not produce ingenuity or innovation. If you think about it, in fact, it can only result in the disintegration of imagination. I am remembering Lego’s, the small red and white plastic blocks that David used to play with for hours on end, creating creatures and crafts, large and small. There were no diagrams or instruction for Lego’s back then. All that was needed was an endless supply of those blocks and a child’s imagination. A friend told me recently that Lego’s hadn’t been selling too well, because children were used to kits that told them what to create and gave them specific direction on how to create it. The company decided that to keep up with the competition, they’d have to follow the trend. Now, you child can buy a different kit for each craft he or she decides to build, complete with instructions for where to put each block. Same little red and white plastic Lego’s. All that’s left out is the imagination.

I’m far from being the first to argue that standardization provides parameters -a preflight checklist- to prevent crisis. Standards are not the end all, these are constantly being revised -but they give you a platform from whence you can make changes if everyone’s starting from the same mark.

Returning to the New Yorker magazine article that inspired my subscription, the subject matter wasn’t preflight checklists for fighter pilots. It was about a man named Pronovost, who with a checklist, has saved more lives than any scientist in the past decade. As the author Atul Gawande explains, we all want brave heroes to save us. I wrote three previous entries on how factories (and DEs) reward heroism, circumventing root cause analysis and resolution. Guwande says:

But the prospect pushes against the traditional culture of medicine, with its central belief that in situations of high risk and complexity what you want is a kind of expert audacity—the right stuff, again. Checklists and standard operating procedures feel like exactly the opposite, and that’s what rankles many people.

Maybe we need a new kind of hero? The boring kind? How about someone who created the impetus for this:

In the U.S., hospital-acquired infections affect 1 in 10 patients, killing 90,000 of them and costing as much as $11 billion each year. Pronovost, 43, began investigating this alarming trend at Johns Hopkins’ hospital in 2001 and concluded that arming physicians with a chart reminding them of each step in routine procedures drastically reduces the medical errors that lead to such infections. After he published his results in several prominent journals, the medical community started listening. Michigan hospitals began implementing Pronovost’s checklists in ICUs in 2003. Within three months, hospital-acquired infections at typical ICUs in the state dropped from 2.7 per 1,000 patients to zero. More than 1,500 lives were saved in the first 18 months.

Hospitals around the world have begun to implement his simple checklist. In recognition, Time Magazine has honored Dr. Peter Pronovost as one of the top 100 people of 2008. Hey, he’s even young and good looking. I bet he’d look good in a cape and tights.

In the end, I find the evidence in favor of procedures but what of the case of bad lemonade? Where do procedures go wrong? Regardless of which side of the fence you fall, the lack of any procedural practices are the biggest failure of all. And sadly, that’s where too many DEs are. Why do so many insist on winging it or doing it “intuitively”? Why do so many feel they have to be at a certain level before they start doing things right? If bad practices kill large enterprises, they kill smaller ones faster and more effectively. In the end, I think it’s the brave who end up adopting standard practices however boring and staid they may be.

[Amended 5/30/2008]
It would be comedic if it weren’t so tragic but apparently, Dr. Pronovost’s checklist, responsible for saving millions of dollars and thousands of lives has been banned. Yes! Banned! In comments, janarshaw left a comment explaining the gory details. Here’s an excerpt, emphasis is mine:

…last month, the Office for Human Research Protections shut the program down. The agency issued notice … that, by introducing a checklist and tracking the results without written, informed consent from each patient and health-care provider, they had violated scientific ethics regulations. Johns Hopkins had to halt not only the program in Michigan but also its plans to extend it to hospitals in New Jersey and Rhode Island.

The government’s decision was bizarre and dangerous. But there was a certain blinkered logic to it, which went like this: A checklist is an alteration in medical care no less than an experimental drug is. Studying an experimental drug in people without federal monitoring and explicit written permission from each patient is unethical and illegal… Indeed, a checklist may require even more stringent oversight, because the data gathered in testing it could put not only the patients but also the doctors at riskby exposing how poorly some of them follow basic infection-prevention procedures.

A large body of evidence gathered in recent years has revealed a profound failure by health-care professionals to follow basic steps proven to stop infection and other major complications. We now know that hundreds of thousands of Americans suffer serious complications or die as a result.

Scientific research regulations had previously exempted efforts to improve medical quality and public health — because they hadn’t been scientific. Now that the work is becoming more systematic (and effective), the authorities have stepped in. And they’re in danger of putting ethics bureaucracy in the way of actual ethical medical care.

As in the example of the bad lemonade, we have high level bureaucratic butt covering to hide the errors of doctors who fail to follow standard operating procedures. Wow! Any guesses as to who lobbied for this ruling? It is just so ironic that bureaucracy finds it more important to cover the malfeasance of doctors than it finds the need to use checklists to protect patients. It’s a good thing the Office for Human Research Protections doesn’t run OSHA, labor regulations, or airline safety. The OHRP is run under the auspices of the US Department of Health and Human Services. Just who is in charge of this circus?

Get New Posts by Email

17 comments

  1. Todd Hudson says:

    Legos are standardized! All Lego parts have standard connectors that allow you to reconfigure them into new designs made from Legos. It’s kinda like the advantages gained from using blocks and pattern numbers. Mix them up for a new style and they still sew up together. However, the pattern maker should still use a checklist to make sure it will work.

  2. dosfashionistas says:

    This is one of those “WoW!” entries that I will be thinking about for days.

    It seems to me that procedures are essential, especially in dealing with anyone who contracts to do a job for you. How else can you expect them to do the job the way you want it done? How you are going to get your procedures across to people who may or may not read or even speak English is another question.

    I can’t wait to read more comments. But my daily procedure(or routine) says that I need to be writing right now or I will have nothing ready to post on auction tonight.

  3. Eric H says:

    Regarding the victory in WWII, someone* once noted that “War is chaos. The Americans practice chaos every day. That is why they will win.”

    Regarding the bad lemonade, I would challenge everyone at every step to show me where the “procedures” required what was done. Too often, I find that people hide behind procedures they have never read and have begun to interpret for themselves. When you actually pull out the text, you frequently find that they are inventing procedures, and usually much more conservatively than what is written.

    It occasionally happens that something arises which falls through the cracks of the most carefully written procedures. Those procedures must be rewritten. The alternative is to start ignoring the procedures, a slippery slope that is best left untried. It is a constant struggle to make the actual procedures and the written procedures match; this is transparency.

    Bottom line: any organization, once it has grown large enough, becomes a faceless bureaucracy in which written procedures must become a substitute for the nexus of responsibility and authority that we find in individuals acting on their own behalf. In the bureaucracy, the authority is held high in the hierarchy, while the responsibility falls lower. When you separate the two, you find powerlessness, fear of mistakes and reprisals, and mindless standardization. Standards replace judgement and must be applied carefully when dealing with moral issues.

    But we are for the most part talking here about small to medium organizations doing (what should be) precision work. Precision work divided among many people doing similar things should be easily standardizable. Pattern-making is the process of turning the designer’s judgement into standards. No moral questions are involved; the judgement may be removed ruthlessly.

    * That someone was German Chief of Staff Gen. Jodl.

  4. Carrie says:

    That lemonade story is so bizzare. It reminds me how many times people have gone the other way and ignored blatant problems as well due to “following procedures”.

  5. Rachel says:

    Incredible post today. I hung onto every word.

    I feel that you can “stick to procedure” and be smart about business and still be able to be creative within that. Finding the perfect balance in the practical and creative is very difficult.

  6. mc says:

    I think bureaucracy can be highly inefficient. But sometimes rules can help accomplish things and enforce accountability as well.

    I personally do not mind regulations. I also might be more inclined to trust an automated machine-man rather than some floating artist (just an exaggerated example). I am all for rituals, which may entail stability/order too… usually only thoroughly appreciated if I am the one who initiated and established the procedures. At least I know what to expect when I face another Drac.

  7. mc says:

    One more thing, preparation–in this case, checklists, may sequester the enemy: waste, error, mismanagement, etc. You could multi-task and run a Six Sigma-DMAIC (Define, Measure, Analyze, Improve, Control) simultaneously. Those people love charts. Graphs and procedures galore. Good times.

  8. Marie says:

    This could go on forever. I’ve worked in the semiconductor industry for 27 years, building chips (integrated circuits) in a wafer fab. In case your wondering what the heck a wafer fab is, it’s the clean room in the Intel commercials, where people are in coverals from head to toe, with only their eyes showing. That’s not to protect the people-it’s to protect the product from the people as we are one big ball of contamination.

    Everything is done by strict procedures written in reams of specifications. The product (silicon wafers) might go through 100-150 different procedures in which the chips are built, in layers, on the surface and some elements are driven into the surface. My long winded point is that even with all of this precision, there are still judgement calls made, at least where I’ve worked there were.

    Sometimes small malfuctions occur that don’t necessarily ruin the product and the process people have to decide whether to scrap it or send it on. Sometimes it is left up to an individual, sometimes decided by a board of engineers. Extra tests might be done. When the decision is made by individuals, we used to ask ourselves “would I want this in my car?” (This company-not Intel-made microcontrollers for cars among other things.) The process is constantly changing and evolving too, as is the product. Constant improvement and statistical process control are religion in semiconductor. All changes have to be approved by a board, or the customer for some changes, and written into the specs, but there will always be those judgement calls in there too.

    The person making the call has to take responsibility for it. Surely someone could have stepped up and made a call on the lemonade case. That is getting a little totalitarian if you ask me. Probably no one wanted to be responsible.

  9. ioanna says:

    I’m all for check-lists. For one they help me organize my stuff to do. And after reading the [very impressive] article about the hospitals I think all hospitals should implement them. However I also think that when human interactions are involved (as in the case of law enforcement and social workers) and emotions and personal judgment are in play a follow-the-rules/check-list procedure might be inadequate by itself. I think better training in decision-making should be incorporated in the system. I mean, just read this article on boing boing about airport security confiscating a 2 inch gun necklace as a security risk. As many check-lists as they had to help them out there was just no accounting for lack of personal judgment! :)

  10. janarshaw says:

    A bit of a follow up on Dr. Pronovost. After publication of the New Yorker article, the Office of Human Research Protection recently shut down the Johns Hopkins University project.

    See Atul Gawande’s New York Times op ed piece

  11. Kathleen says:

    Ohmigawd!

    It’s bureaucracy to the rescue, exactly the point of my entry. Excerpted from the link that janarshaw left, emphasis mine:

    Re: Dr. Pronovost’s checklist responsible for saving millions of dollars and thousands of lives:

    …last month, the Office for Human Research Protections shut the program down. The agency issued notice … that, by introducing a checklist and tracking the results without written, informed consent from each patient and health-care provider, they had violated scientific ethics regulations. Johns Hopkins had to halt not only the program in Michigan but also its plans to extend it to hospitals in New Jersey and Rhode Island.

    The government’s decision was bizarre and dangerous. But there was a certain blinkered logic to it, which went like this: A checklist is an alteration in medical care no less than an experimental drug is. Studying an experimental drug in people without federal monitoring and explicit written permission from each patient is unethical and illegal… Indeed, a checklist may require even more stringent oversight, because the data gathered in testing it could put not only the patients but also the doctors at riskby exposing how poorly some of them follow basic infection-prevention procedures.

    A large body of evidence gathered in recent years has revealed a profound failure by health-care professionals to follow basic steps proven to stop infection and other major complications. We now know that hundreds of thousands of Americans suffer serious complications or die as a result.

    Scientific research regulations had previously exempted efforts to improve medical quality and public health — because they hadn’t been scientific. Now that the work is becoming more systematic (and effective), the authorities have stepped in. And they’re in danger of putting ethics bureaucracy in the way of actual ethical medical care.

    As in the example of the bad lemonade, we have high level bureaucratic butt covering to hide the errors of doctors who fail to follow standard operating procedures. Wow! Any guesses as to who lobbied for this ruling?

  12. Karen C says:

    I’m all for standard operating procedures, but what has happened to common sense and gut instincts? I mean, why are we now forced to put our carry-on liquids (soap, shampoo, etc.) in ziplock baggies? I can still open the baggie once on the plane, so what is this preventing??? Or having to take off our shoes to get through security because of what one man did. What happens if someone pulls the same stunt with explosives in, say, his underwear? Will we all have to travel without our panties??? Maybe we can legislate to have common sense, but then that would put all of the politicians out of a job.

  13. Matt says:

    Kudos on a great article.

    Coming from a software development background, checklists remind me a lot of the thinking behind some of the more current development methodologies. The issue, in software, is not having or not having a checklist, but making sure that checklist is constantly reviewed and updated so that it’s the best possible checklist given the information at the time.

    As you’re pointing out, it’s also important that people are participants in the checklist rather than following it wrote. The lemonade example is a perfect example of wrote checklist following, while the b-299 and the medical checklists are great examples of actively participated checklists.

    Keep up the great work!

  14. Mark Graban says:

    Thankfully the Federal Government reversed their ruling, as I wrote about on my blog.

    The government problem was pretty bureaucratic — it wasn’t the checklists that were a problem, it was the collection of data for “research” related to the checklists that was the issue.

    Ironically, the Feds were just following procedure. And they managed to eventually make the right decision.

    LEAN BLOG LINK

    Policies are not bad. People who cowardly hide behind policies are bad.

    Toyota calls it “Standardized Work” not “Identical Work” for a reason.

  15. Holly says:

    Checklists are great. They free up your mind from the dreary details, so you have the brain-space to recognize when a moment of innovation comes along. When you make a habit of following the procedure, you never have to stop and wonder, “Wait, did I do that?”

    I have no comment on the lemonade thing, but I enjoyed this post.

Leave a Reply

Your email address will not be published. Required fields are marked *