Automation complacency

This is an unpublished draft. Please keep it to yourself.

In May of 2009, approximately 35,000 feet above the Atlantic Ocean, an Airbus A330’s airspeed sensors iced over and failed. As a result, the autopilot disengaged, placing the aircraft under manual control. Five minutes later the aircraft struck the water, destroying it and killing all 228 passengers and crew on board.

Pilots are all trained to fly without autopilot, but their awareness, skills, and mental models of how an aircraft responds are transformed and atrophied by years of autopilot use. When autopilot goes away, and when circumstances are unusual, pilots can make fatal mistakes like the ones that resulted in the crash of Air France Flight 447.

Using AI coding assistants is strikingly similar to flying with autopilot. Fortunately, we are early in their adoption, so it’s not too late to do what the aviation industry required hundreds of deaths to precipitate. Let’s not miss our chance.

• • •

The crash of Air France Flight 447 is a notorious event in modern aviation history. It is the deadliest in the entire history of Air France, and the deadliest for the Airbus A330 on the whole. It took years of investigation before the French aviation safety bureau (the BEA) published their final report.

The investigation produced one key finding: the pilots’ skills to take over full control of the aircraft in those unusual circumstances were not correctly maintained. The crash was the result of a persistent manual attempt to pull the nose up, even while stall warnings were sounding.12

The better automation gets, the more dangerous the handback becomes.3 The aviation industry call this “automation complacency.”4 As a result of Air France Flight 447 and other, similar events, the FAA tightened up their enforcement of mandatory manual flying during routine operations.5

So what, who cares?

III. The Cockpit We’re Building

  • AI coding assistants are autopilot for software development — and the near-term pitch is largely true
  • The act of construction encodes differently than the act of review6
  • Offshoring research established the precedent: the knowledge that disappears is the knowledge that was never written down7

IV. The Lag

  • The costs and benefits of automation don’t arrive on the same timeline
  • VC funds velocity, not five-year consequences — this is structural, not conspiratorial
  • Offshoring delivered the same short-term metrics and the same deferred costs8

V. What the FAA Did

  • The FAA didn’t ban autopilot — it required deliberate, scheduled friction9
  • Core principle: you cannot maintain the ability to take over a system you never operate[^10]

VI. What You Can Do Today

  • Maintain currency — build components by hand on a regular cadence; the skill expires without practice[^11]
  • Own the architecture — AI executes decisions, it doesn’t make them; that knowledge must live somewhere human
  • Name the handback scenario — if the tooling disappeared tomorrow, what would you be unable to do?

Closing

  • Return to the accident report: the assumption of readiness was never tested until the moment it couldn’t be
  • We are making the same assumption — we just haven’t hit the water yet

  1. Bureau d’Enquêtes et d’Analyses (BEA), Final Report on the Accident on 1st June 2009 to the Airbus A330-203 Flight AF 447, July 2012 (PDF). ↩︎

  2. IEEE Spectrum, “Air France Flight 447 Crash Caused by a Combination of Factors”: “A review of pilot training did not provide convincing evidence that the associated skills had been correctly developed and maintained.” ↩︎

  3. Oliver, Calvard & Potočnik, “The Tragic Crash of Flight AF447 Shows the Unlikely but Catastrophic Consequences of Automation”, Harvard Business Review, September 2017. ↩︎

  4. Casner, Geven & Schooler, “The Retention of Manual Flying Skills in the Automated Cockpit”, Human Factors, 2014. Pilots’ cognitive skills—reasoning about position, diagnosing failures—decay faster than motor skills under sustained automation use. ↩︎

  5. Flight Global, “New FAA Pilot Training Guidelines Seek to Address Manual Flying Deficiency”, November 2022. ↩︎

  6. van der Meer & van der Weel, “Handwriting but Not Typewriting Leads to Widespread Brain Connectivity”, Frontiers in Psychology, 2023. Active construction activates broader neural networks than passive transcription or review. ↩︎

  7. Williams, C., “A Steep Learning Curve”, BCS, The Chartered Institute for IT, summarizing findings from Information Systems Journal (2010). Tacit knowledge—gained through experience and repeated interaction in the task environment—transfers only when offshore engineers are embedded directly within the client organization. Documentation and formal training capture explicit knowledge only; the rest requires proximity and sustained contact to survive the handoff. ↩︎

  8. FAA Safety Alert for Operators (SAFO) 13002, “Manual Flight Operations”, January 2013. ↩︎

  9. Flight Safety Foundation, “Use It or Lose It”: cognitive skills required for manual flight decay measurably under sustained automation use, even when motor skills remain intact. ↩︎

Lead image by Gemini Nano Banana

Comments