AI Handles Incidents, Engineers Lose Touch With Their Systems

Sylvain Kalache's September 2026 post starts from his own history. As an SRE at LinkedIn in 2012 he designed a system meant to heal itself and learn from earlier incidents; it stayed a prototype because the AI of the time could not do it. Tools that do now exist: they inspect alerts, form hypotheses, query telemetry, correlate recent deployments and apply the fix. He likes them, dislikes their marketing name "AI SREs", and worries that engineers are losing touch with their systems. He now works at Rootly, an incident-management company, which matters because the remedy he describes is a product his employer built.

The argument

Routine incidents are how responders safely build an intuition for how their systems behave and fail. A tool good at routine incidents removes that practice, and when a hard, never-seen-before incident arrives that the tool cannot solve, the humans take over with less experience than they would have had.

He attributes the idea to Lisanne Bainbridge's 1983 paper (bainbridge-ironies-of-automation; he calls it "The Ironies of Automation", the published title has no article). His paraphrase holds up against the text: automation reduces operators' chances to practise routine work while leaving them responsible for abnormal situations, so they need more skill and more training than before.

From that he makes a prediction worth writing down because it can be checked later: average MTTR will fall for most incidents thanks to AI-assisted response, while resolution time for complex incidents will shoot up because responders have lost touch with their systems. He offers no data for it yet.

Aviation as the model

Pilots are his example of a profession that already lives with this. Automation flies most of the time, and pilots stay responsible for engine failures, unreliable instruments, rejected takeoffs and stalls. He cites fewer than one in-flight shutdown per 100,000 engine flight hours for modern turbine engines, rare enough that a commercial pilot can finish a career without meeting one outside a simulator. When one does happen, reaction has to be fast and right, and his counter-example is TransAsia Airways Flight 235: the right engine's propeller autofeathered after takeoff, the crew misidentified the problem, and the aircraft stalled and crashed 117 seconds after the first warning. Under FAA rules, he writes, captains complete recurrent training or a proficiency check every six months, including engine failure on takeoff.

The remedy

Rootly and Uptime Labs run realistic incident simulations. An engineer takes the incident commander's seat during a simulated e-commerce outage, investigates with observability tools, and coordinates with LLM-played stakeholders in Slack, including a CEO and customer support. What gets practised is sense-making from incomplete information, clear communication, coordinating people and running the response.

He considers the obvious alternative, asking the agent to explain the steps, signals and evidence behind its diagnosis, and rejects it as a substitute: watching Serena Williams teaches you a few things, but you learn tennis on the court. He backs this with his own background. He spent more than five years building a software engineering school around learning by doing, with no teachers and projects instead of lectures, and when Dropbox told him the graduates it hired were weak at troubleshooting, he wrote projects that handed students broken infrastructure to diagnose and repair.

His name for the risk is comprehension debt: the growing gap between how a team's systems work and how well its responders understand them. The prescription is regular hands-on contact with the system, exposure to unfamiliar failures, practice under pressure, and rehearsed coordination for a SEV0. Tabletop exercises and chaos engineering are not new, he concedes, but they matter more now.

Reading it against Bainbridge

The post takes two of the paper's remedies, hands-on control and simulation, and leaves others out that would strengthen it. Bainbridge argued that simulators cannot reproduce unknown faults, so training has to teach general strategies rather than specific responses. That is a good description of what an incident-commander exercise trains, and a reason to prefer it over rehearsing yesterday's outages. She also warned that automatic control can camouflage a failure by compensating for it until it is beyond recovery. An AI responder that quietly resolves the symptoms of a slow degradation at 3am is a plausible software form of that, and it would push the complex incidents he worries about further from anyone's view.

His "comprehension debt" is a team-level cousin of cognitive-debt, which describes the same erosion inside one person's head. The coding-agent version of his argument is skill-atrophy-supervision-paradox; the general pattern is at ironies-of-automation. The school anecdote puts him on the same side as dont-outsource-learning, where the fix is keeping the struggle in the loop rather than reading an explanation of someone else's.