What mission-driven workers can learn from a plane crash
At 2:10 UTC on June 1, 2009, an Airbus A330 was cruising at 35,000 feet over the Atlantic. Air France 447 had taken off from Rio de Janeiro four hours earlier, bound for Paris. There were 216 passengers and 12 crew on board. Captain Marc Dubois had stepped out of the cockpit twelve minutes earlier for a scheduled rest break, leaving two copilots in the seats: Pierre-Cédric Bonin in the right seat as the pilot flying, David Robert in the left seat as the pilot monitoring.
The plane flew into a band of ice crystals. The crystals clogged the three pitot tubes: small forward-facing probes that measure airspeed by sensing the pressure of oncoming air. With the airspeed signal lost, the flight computer did what Airbus had designed it to do: it disconnected the autopilot, sounded a chime in the cockpit, and handed control of the airplane back to the humans. It reconfigured the flight controls from "normal law" to "alternate law,” a degraded mode the pilots had trained on but rarely flown.
With the airplane rolling slightly to the right and the airspeed indications jumping, Bonin appears to have read the moment as a loss of altitude that needed correcting. He pulled back on his sidestick and held it there. Sidesticks on the A330 sit beside each pilot's leg and do not move in sync; Robert, eight feet away on the other side of the cockpit, could not see or feel Bonin’s input. In normal law, Bonin's input would have been limited by the autopilot, because pulling too far up can cause the plane to stall.
In alternate law, the protections were gone. The nose came up. The airplane stalled.
The stall warning sounded 75 times during the descent. At one point it was continuous for 54 seconds: “STALL. STALL. STALL.” The pilots did not respond to it. Bonin held the stick back, thinking he was helping the plane climb up. Robert thought the airplane was descending uncontrollably and could not figure out why. Neither pilot understood that they were in a stall.
When Captain Dubois returned to the cockpit, he walked into a cabin where the airplane was falling at 10,000 feet per minute, the stall warning was sounding, the airspeed displays had returned to normal but the airplane itself was barely flying.
Captain Dubois:"Hey, what the hell are you doing?" ("Eh qu'est-ce que vous foutez?")
Bonin: "We've lost control of the plane!"
Robert: "We've totally lost control of the plane. We don't understand at all… We've tried everything."
About 2 minutes later, in response to Robert mumbling “Climb, climb, climb,” Bonin finally said, "But I've been at max nose up for a while" and Robert and Dubois both realized, seeminlgy at once, that the entire descent had been caused by an input neither of them had seen. Robert took the controls and pushed the nose down; exactly what you want to do to regain control in a stall. But the airplane was just 6,000 feet above the ocean, and there was not enough altitude left to recover.
Four minutes and twenty-three seconds after the autopilot disconnected, the plane hit the ocean nose-up at over 100 miles per hour. Everyone on board was killed. The black boxes were not recovered until 2011. The French Bureau of Enquiry and Analysis released its final report in July 2012.
Qualified pilots flew an intact wide-body airliner into the ocean because the automation they had relied on for years stopped flying the airplane, and the skills it had replaced were not there to take over. The instruments came back online within a minute of the initial problem. There was no fire, no explosion, no structural failure. The airplane was flyable and responding directly to pilot inputs. The crash is studied today not because the automation failed (automation fails all the time without consequence) but because the interface between the humans and the machine hid the information that would have let them recover.
Four concepts from human factors and human-computer interaction explain this crash. Each has a parallel in mission-driven organizations adopting AI.
The gulfs of execution and evaluation
In his book that’s now called “the Design of Everyday Things,” Don Norman names two distances that any operator has to bridge to use a system. The gulf of execution is the distance between what you want to do and what the system lets you do. In this case, how do I make it climb? The gulf of evaluation is the distance between the system's actual state and your beliefs about that state: what is it doing right now? An interface bridges these gulfs by making the available actions clear and the current state visible.
When the A330 switched to alternate law, both gulfs widened at the same time.
The execution gulf widened because the same control input now did something different. In normal law, pulling the stick all the way back was a safe input; the autopilot corrected for it. In alternate law, the protection was gone, and the same input would stall the airplane.
The evaluation gulf widened because the system's state was difficult to interpret. The cue that the airplane had switched modes was one line of text on a display and a brief chime. The airplane's actual flight state— stalled, falling, nose-up— was not displayed in any direct form. The A330 measures angle of attack but does not show it to the pilots. Bonin’s “max nose up” input was not shown to the other pilot like it is in some airplanes that move the side sticks together. Robert spent much of the descent trying to make sense of contradictory airspeed readings, with no instrument that would have told him plainly what the airplane was doing.
Bonin's mental model kept saying: we need to go up. So he did what he would need to do under normal law: pulling the side stick up. But system was not in normal law. Nothing in the cockpit interface bridged the gap, and certainly not with sufficient salience to overcome the panic induced by high-speed descent and the repeated warnings: STALL. STALL. STALL.
The non-mirrored sidesticks
Older Boeing cockpits use mechanically linked control yokes. If the right-seat pilot pulls back, the left-seat yoke moves backward too. The non-flying pilot can see, and feel through the column, what the flying pilot is doing. The yoke is a shared artifact: it carries information between the two human operators without either having to articulate it.
The Airbus A330's sidesticks do not move in sync by design. There is an indicator that reports "DUAL INPUT" when both pilots move their sticks at once, but no continuous feedback otherwise.
This is a problem of distributed cognition, a framework from anthropologist Edwin Hutchins (from UCSD; go Tritons!), who studied real cockpits for years and argued that flying an airplane is not something individual pilots do but something a system does, where the system includes pilots, instruments, checklists, and shared artifacts that pass information between people. No one person or computer is doing all the thinking: cognition is distributed across the cockpit. (Hutchins, Cognition in the Wild, MIT Press, 1995)
Remove a channel of information that the system was relying on, and the cognition gets trapped. For much of the four minutes, Robert thought the airplane was descending because the airspeed indication was wrong; he did not know that Bonin was holding the stick all the way back, commanding the airplane’s nose upward and causing the stall. Bonin never said so. He may not have realized what he was doing, because in normal law that input would have been harmless.
Deskilling
The two concepts above explain how the situation arose. Deskilling explains why it could not be fixed.
The aviation joke that runs through this literature: the cockpit of the future will have two crew members, a pilot and a dog. The pilot is there to feed the dog. The dog is there to bite the pilot if the pilot touches anything.
Deskilling is the gradual loss of a manual skill that automation has taken over. Highly automated airliners fly themselves for almost the entire trip. Bonin's logbook recorded the routine numbers, but very little manual flight at high altitude, in degraded modes, or in turbulence. The BEA report found that Air France had identified internally, before the crash, that some of its long-haul pilots had weak basic flying skills and difficulty making sense of equipment failures.
If the plane had stayed in normal law, his stick input would not have caused a stall, and his rusty manual flying would not have been tested. The system had carried him for years. It stopped carrying him at 2:10 UTC, and many people paid with their lives.
The handoff
When Captain Dubois walked back into the cockpit, he inherited the controls in a degraded mode he had not been flying in, with two operators whose models did not match each other or reality, and with no shared external artifact that captured what had been happening. The cockpit voice recorder shows him reconstructing the situation from scratch by asking questions.
A handoff is a transfer of control between operators that requires the receiving operator to inherit not just the controls but the state: what's been done, what's been tried, what the system is currently in. The handoff failed because there was nothing to hand off with. By the time Dubois pieced the picture together, it was too late to save the plane.
Why this matters on the ground
Fortunately for everyone, I do not make decisions with 228 lives at stake at my job. But, I (and hopefully you) can learn a lot from this extreme example.
When an organization deploys an AI tool, staff build a mental model of what the tool will do, based on their training and the tool's behavior in normal use. Changes like model upgrades, context-window overflows, and routing can be hidden and widen the gulf of evaluation: the tool's actual state diverges from the user's picture of it, and nothing in the interface signals the change. The execution gulf follows: the prompts that used to work no longer do.
When two people both touch the same AI-assisted work, the tool does not pass information between them about what it did, what it left out, or what it made up. There is no linked yoke. We need to communicate to one another about what we know, including how we used AI, so that our colleagues can evaluate it accurately.
When staff use an AI tool every day for a task they used to do by hand, the manual skill can atrophy. When the tool fails or returns a bad output, the human who has to step in may not have done the task in months or even years.
When work passes from an AI agent back to a human because the agent got stuck, the human inherits a state— actions taken, sources used, decisions made— that they may not be able to interpret
When you have the opportunity to help design an AI implementation or the work practices around it, be aware of these risks. Make the system's mode visible to its users, make its actions visible to people downstream, keep humans in the loop on the underlying skill, and design interfaces that surface disagreements between the tool and the user instead of hiding them.
—
I used Claude to turn my notes from the manuscript and a lecture I’d given for a class I recently taught to put this together :)
