This is an unpublished draft. Please keep it to yourself.

Automation complacency

I. The Accident Report Nobody Read

  • The story of AF447: a routine flight, a 40-second sensor failure, a functional aircraft, 228 dead1
  • The investigation’s core finding: the skills had not been correctly developed and maintained2

II. The Automation Paradox

  • The better automation gets, the more dangerous the handback becomes3
  • Aviation researchers named this dynamic: automation complacency4
  • The FAA eventually required airlines to reintroduce manual flying into routine operations5

III. The Cockpit We’re Building

  • AI coding assistants are autopilot for software development — and the near-term pitch is largely true
  • The act of construction encodes differently than the act of review6
  • Offshoring research established the precedent: the knowledge that disappears is the knowledge that was never written down7

IV. The Lag

  • The costs and benefits of automation don’t arrive on the same timeline
  • VC funds velocity, not five-year consequences — this is structural, not conspiratorial
  • Offshoring delivered the same short-term metrics and the same deferred costs8

V. What the FAA Did

  • The FAA didn’t ban autopilot — it required deliberate, scheduled friction9
  • Core principle: you cannot maintain the ability to take over a system you never operate[^10]

VI. What You Can Do Today

  • Maintain currency — build components by hand on a regular cadence; the skill expires without practice[^11]
  • Own the architecture — AI executes decisions, it doesn’t make them; that knowledge must live somewhere human
  • Name the handback scenario — if the tooling disappeared tomorrow, what would you be unable to do?

Closing

  • Return to the accident report: the assumption of readiness was never tested until the moment it couldn’t be
  • We are making the same assumption — we just haven’t hit the water yet

  1. Bureau d’Enquêtes et d’Analyses (BEA), Final Report on the Accident on 1st June 2009 to the Airbus A330-203 Flight AF 447, July 2012 (PDF). ↩︎

  2. IEEE Spectrum, “Air France Flight 447 Crash Caused by a Combination of Factors”: “A review of pilot training did not provide convincing evidence that the associated skills had been correctly developed and maintained.” ↩︎

  3. Oliver, Calvard & Potočnik, “The Tragic Crash of Flight AF447 Shows the Unlikely but Catastrophic Consequences of Automation”, Harvard Business Review, September 2017. ↩︎

  4. Casner, Geven & Schooler, “The Retention of Manual Flying Skills in the Automated Cockpit”, Human Factors, 2014. Pilots’ cognitive skills—reasoning about position, diagnosing failures—decay faster than motor skills under sustained automation use. ↩︎

  5. Flight Global, “New FAA Pilot Training Guidelines Seek to Address Manual Flying Deficiency”, November 2022. ↩︎

  6. van der Meer & van der Weel, “Handwriting but Not Typewriting Leads to Widespread Brain Connectivity”, Frontiers in Psychology, 2023. Active construction activates broader neural networks than passive transcription or review. ↩︎

  7. Williams, C., “A Steep Learning Curve”, BCS, The Chartered Institute for IT, summarizing findings from Information Systems Journal (2010). Tacit knowledge—gained through experience and repeated interaction in the task environment—transfers only when offshore engineers are embedded directly within the client organization. Documentation and formal training capture explicit knowledge only; the rest requires proximity and sustained contact to survive the handoff. ↩︎

  8. FAA Safety Alert for Operators (SAFO) 13002, “Manual Flight Operations”, January 2013. ↩︎

  9. Flight Safety Foundation, “Use It or Lose It”: cognitive skills required for manual flight decay measurably under sustained automation use, even when motor skills remain intact. ↩︎

Lead image by Gemini Nano Banana

Comments