Plenty of AI deployments promise a human in the loop: one person looks over every decision and signs off, so responsibility stays with a human. That falls apart when the queue outruns anyone’s reading speed. The review fades while the signature remains.
The numbers explain why. Split an eight-hour shift (28,800 seconds) by the caseload and the result is the time a reviewer can spend on each case. Reference points run from 72 seconds a case down to 20 seconds and, in the fastest insurance example, 1.2 seconds. Each step down leaves less room to read anything.
What automation bias research shows
Automation bias is well documented. A 2016 systematic review of 40 papers, six from health care, found it in 25 of 31 studies on missed problems and in 21 of 23 on following wrong advice. In one study the review cites, doctors failed to correct a machine’s misdiagnosis for 24% of patients.
Rarity makes it worse. A 2005 search study by Wolfe, Horowitz and Kenner showed it: participants missed 7% of targets when half the images held one, but 30% when only 1 in 100 did, mostly by missing them.
That has a direct lesson for AI. A model that is usually right makes its mistakes rare, and those are the ones reviewers overlook.
Why human in the loop checks fail on speed
Article 14 of the EU AI Act tells human overseers to stay aware that they may lean too heavily on a system’s output. With 1.2 seconds a case, that awareness has almost no time to act.
Who takes the blame
Two writers have named the outcome. Dan Davies, in The Unaccountability Machine, describes an accountability sink, where a structure strips away ownership of a decision. Madeleine Clare Elish calls the human a moral crumple zone, absorbing the blame like the part of a car built to take the crash.
Under both labels the reviewer’s name ends up attached to a decision they had no real chance to examine, and every error traces back to them.
What aviation learned from Flight 173
Aviation learned this the hard way. In 1978, United Airlines Flight 173 ran out of fuel over Portland while the crew dealt with a landing gear problem. Two crew members raised the low fuel but neither pushed the issue, and the captain kept circling. The crash led the industry to adopt crew resource management.
That approach has three parts that translate to software:
- Checks are spoken aloud one item at a time, so inspection gets dedicated time in the flight.
- Training teaches junior crew to voice a worry and captains to listen to it.
- A cockpit recorder preserves each person’s words, so a warning cannot quietly disappear.
Applying the fixes to an AI queue
A human in the loop needs time, authority and a record, and the airline fixes supply all three.
Each practice has a counterpart in a review queue. Limit cases per reviewer to a number that leaves time to read. Give reviewers an override with no penalty and no quota. Record every override and warning along with who raised it.
None of this requires new technology. It requires deciding that review time is a resource to protect, and that a human in the loop who cannot object is only a signature on a form.
What organizations should check first
The fixes above cost little compared with the damage a rubber stamp can hide. A reviewer who approves hundreds of AI decisions a day gives the organization a paper trail, but not protection from bad calls.
A quick audit is possible with the formula above. Divide the length of a reviewer’s shift by the number of cases they handle. If the answer is a few seconds, the review is not happening, whatever the process document says. Teams that find a number like that have two honest options: slow the queue down, or stop describing the oversight as human.
Regulators are already pointing in this direction. If the EU AI Act expects overseers to guard against over-reliance, a queue designed for one-second approvals is hard to defend.
Source: moai.studio