AI Reverberations Pattern: Why Rollouts Backfire

Rolling out AI tools tends to follow a predictable script, and the AI reverberations pattern describes it. First the tool adds capability. Then people build quiet workarounds to cope with the new problems it creates, and when something finally breaks, the blame lands on them.

The pattern comes from a 2000 paper by David Woods and Sidney Dekker on the effects of technological change on human factors. They described what happens when advanced but opaque, “black box” technology enters complex, critical work. Software teams using generative AI will find it familiar.

The six steps of the AI reverberations pattern

The sequence runs like this:

  1. The technology provides new capabilities.
  2. Those capabilities raise demands and expectations, so the technology gets used. It also brings new complexities.
  3. Practitioners adapt or invent workarounds so they can meet the higher expectations.
  4. The complexities and adaptations are surprises. The original design didn’t anticipate them.
  5. An unanticipated situation eventually arises that people can’t handle or catch. Their workarounds are often incomplete and brittle, partly because of the increased demands.
  6. The workarounds hide the complexity from everyone else, so the system looks smooth from a distance. After a disruption, the failure gets labeled “human error.”

How it plays out in AI-assisted coding

Developers can now generate boilerplate, tests and whole functions from short prompts. The cost is new kinds of defects: misused APIs, plausible but wrong error handling, hallucinated packages, and security anti-patterns that look idiomatic. The developer also becomes a reviewer instead of an author, and reviewing code with no visible reasoning trail is harder than reviewing your own.

So developers adapt. They write rules of thumb such as never trusting an agent with auth changes, always rewriting error handling, and checking that every import exists. Some add blunt instructions to their system prompts, like a ban on running git push. These habits are informal and unresourced, so the organization never sees them.

A second adaptation follows. As code volume rises, review depth drops, because expectations about throughput have already absorbed the productivity gain. Then a race condition passes testing and fails under production load, or an unmaintained or typosquatted dependency slips in. The verdict from a distance: the developer should have caught it in review.

Incident response and customer messages

On-call engineers can ask an agent to summarize alerts, correlate signals, suggest diagnoses and draft communications. The agent can propose diagnoses that are wrong in hard-to-spot ways, and it can summarize logs while dropping the one anomalous detail a veteran would notice. Its runbook-style text sounds authoritative without any real knowledge of the system’s current state.

Experienced engineers adapt by treating it as a brainstorming tool and cross-checking results. That only works because they know the system deeply. Newer engineers who had the agent from day one may treat its answers as their main diagnostic method, which makes the agent an intermediary nobody designed or evaluated for that job.

When an incident escalates because a signal was left out of a summary, the official story becomes that the engineer should have verified before acting. Yet the company rewarded faster resolution and never counted “verify the AI’s output” as work.

The same arc applies to drafting status pages, incident summaries and customer emails. Output is fluent and confident, but it may commit to unconfirmed root causes or promise unrealistic timelines. Editors build checklists, trust grows after a clean run, and the checklists get shorter. Then one message contains an inaccurate commitment, and the conclusion is that the communications team should have proofread harder.

What the researchers recommend

  • Expect the work to change. Handing a task to AI doesn’t simply swap one worker for another.
  • Design for joint human-agent work instead of agent autonomy. Delegation isn’t joint activity, and good teams produce more than the sum of their parts.
  • Treat designs as hypotheses. Look for confirming and disconfirming evidence after deployment, the way chaos experiments do for infrastructure.
  • Don’t bank every gain as throughput. Keep some capacity as slack for when you need it.

What to do about the AI reverberations pattern

The point isn’t to avoid the technology. These dynamics have known answers from Cognitive Systems Engineering, even if mainstream software engineering hasn’t used them much.

A practical first step is to write down the workarounds your engineers already use and treat verification as scheduled, resourced work. Teams that do will see the next failure coming earlier, and they’ll be less likely to pin it on whoever happened to be on call.

Source: adaptivecapacitylabs.com

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.