# Ironies of Automation (Bainbridge, 1983)

Lisanne Bainbridge, then in the Department of Psychology at University College London, published this five-page brief paper in *Automatica* (vol. 19, no. 6, pp. 775-779) in 1983. It started as a talk at an IFAC/IFIP/IFORS/IEA conference on man-machine systems in Baden-Baden in September 1982. Her subject is process plants, with examples borrowed from flight decks, and her thesis fits in one clause: "the more advanced a control system is, so the more crucial may be the contribution of the human operator." The pattern the paper names is covered on its own page at [[ironies-of-automation]]; this page follows the paper itself.

She opens by defining her terms, since the title depends on them. An irony is a combination of circumstances whose result is the direct opposite of what might be expected. A paradox is a seemingly absurd but perhaps well-founded statement. The paper uses both.

## The designer's two ironies

The classic aim of automation is to replace human manual control, planning and problem solving with machines, and the designer's reason is often that the operator is unreliable and inefficient. Bainbridge finds two ironies in that attitude. Designer errors are themselves a major source of operating problems, although the people who collect such data are reluctant to publish it. And the designer who tries to eliminate the operator still leaves the operator whatever the designer could not think how to automate. The human ends up with an arbitrary collection of tasks that nobody designed as a job, and little thought goes into supporting them.

## What the operator is left with

Two jobs remain: watch that the automatic system is working, and take over when it is not.

Taking over needs manual control skill, and that skill decays. Experienced operators make a step change with the minimum number of actions and the process settles smoothly, while inexperienced ones set it oscillating. Physical skills deteriorate when unused, the refinements of gain and timing first, so an operator who has spent years monitoring may take over as an inexperienced one. Worse, takeover happens when something is already wrong, which calls for unusual actions. The operator needs to be more skilled and less loaded than average at exactly the moment automation has made them less skilled and more loaded.

Diagnosis needs cognitive skills, and Bainbridge splits them in two. Long-term knowledge of how the process behaves is retrieved efficiently only if it is used often, and it develops only through use and feedback, so classroom instruction without practice leaves little behind. Her sharpest sentence here is about succession: the current generation of automated systems is monitored by former manual operators who "are riding on their skills, which later generations of operators cannot be expected to have."

The second cognitive skill is what she calls working storage. An operator in control does not carry raw readings in their head but predictions and decisions about where the process is going, and that picture takes time to build. Manual operators arrive a quarter to half an hour before their shift to get a feel for the plant. Someone taking over an automated plant in a hurry has none of it and acts on minimum information.

## Monitoring

Monitoring looks like the simple job, and Bainbridge takes it apart. Vigilance studies going back to Mackworth in 1950 show that nobody, however motivated, keeps effective visual attention on a source where very little happens for more than about half an hour. Watching for rare abnormalities is therefore humanly impossible and gets handed to alarms, which raises the question of who notices when the alarm system fails. The traditional fix of making operators keep a log does not help much: "people can write down numbers without noticing what they are."

The more serious irony is that the automatic system was installed because it does the job better than the operator, and the operator is then asked to check that it is doing it well. If its decisions could be fully specified, a computer makes them faster, over more dimensions and against sharper criteria than any person. The human can only judge at some meta-level whether its decisions look acceptable, and if the computer was brought in because human judgement was inadequate, there is no basis for that either. "The human monitor has been given an impossible task."

There is also the matter of status. She knew of one automated plant where management had to be present on the night shift or the operators switched the process to manual. Skill is part of a worker's standing, a job reduced to monitoring is "very boring but very responsible", and deskilled workers insisting on high pay are holding on to the last symbol of a status the job no longer supports.

## Approaches to solutions

She restates the problem as a paradox before offering remedies: by automating the process, the operator is given a task that is only possible for someone who is in on-line control.

For monitoring, a low-probability event that must be noticed quickly needs artificial assistance, "if necessary even alarms on alarms", though a proliferation of flashing red lights confuses rather than helps. Automatic control can camouflage a failure by compensating for it until the trend is beyond control, so the automatics should also watch for unusual movement, and they should fail obviously. Graceful degradation is listed as a human advantage and is not something to aim for in computers. If a person must follow the computer's decisions, the computer has to make them by methods and at a rate the person can follow, even when that is not the most efficient method. Raising the failure rate artificially to keep operators alert would destroy their trust, and one study found performance was worse with computer aiding because the operator made the decisions anyway and checking the computer added to the load.

For working storage, the answer depends on how fast failures develop. If shutting down is simple and cheap, shut down automatically. If a failure unfolds within seconds with no warning, a reliable automatic response is necessary whatever it costs, "and if this is not possible then the process should not be built if the costs of failure are unacceptable." Slower failures can be bought time with overlearned manual responses, practised often on a high-fidelity simulator.

For long-term knowledge, let operators use hands-on control for a short period each shift, and "if this suggestion is laughable then simulator practice must be provided." Simulators cannot teach unknown faults, so training has to be about general strategies rather than specific responses, and procedures cannot cover everything either. "It is ironic to train operators in following instructions and then put them in the system to provide intelligence." The section closes on the line the paper is best known for: "Perhaps the final irony is that it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training."

## Human-computer collaboration

The last section surveys ways computers could support operators rather than replace them, and finds an irony in almost each one. Its opening sentence is the paper's compressed thesis: "By taking away the easy parts of his task, automation can make the difficult parts of the human operator's task more difficult." The Fitts-list approach of giving each of human and machine what it does best ignores how the two are integrated and how the human's skills and motivation are kept alive.

Advice from the computer makes the operator slower and less integrated than generating the actions themselves, and gives no practice in being "intelligent". Error checks work better on the effects of actions than on the sequence, since checking effects assumes nothing about the operator's strategy. Software displays tuned for normal operation can camouflage abnormal conditions, displays that switch with the operator's supposed skill level may confuse under time pressure, and processed information may teach less than information the operator has to think about to take in: "It certainly would be ironic if we find that the most compatible display is not the best display to give to the operator after all!" For shared workload, the human must know which tasks the computer is handling and how, or the result looks like a human team with no clear allocation of responsibility.

The conclusion is modest. People working without time pressure are impressive problem solvers and are less effective under it. Automating does not necessarily remove the difficulties, and resolving them may take more technological ingenuity than classic automation did.

## Why it keeps getting cited

The paper's examples are control rooms of the kind shown in [[soviet-control-rooms]], but its structure is general, which is why software writers keep finding it. [[ai-sre-losing-touch]] applies it to AI incident response and borrows its remedies of hands-on practice and simulators. [[skill-atrophy-supervision-paradox]] is the coding-agent version of the monitoring irony, reached without citing Bainbridge. [[human-in-the-loop]] approval gates put a person in exactly the monitor's seat she describes. [[ironies-of-automation]] collects these and notes where the process-plant argument transfers to software and where it does not.