Provable AI Agent Safety: Putting Human Control First

Diagram of AI agent safety priorities placing human control above task reward

A new line of research on AI agent safety argues that capable agents must stay open to human correction even when that means leaving a task unfinished. The work pairs a formal construction that ranks human control above task reward with tests showing today’s agents often break that rule. The core worry is simple: the same foresight that helps an agent finish a job can help it work around a stop request.

The shutdown problem

Picture an agent editing a document when a notice says its computer will shut down. It knows it can cancel the shutdown and finish. That resourcefulness is welcome when the obstacle is a broken app. It isn’t when the obstacle is a person asking it to stop. Researchers call the desired property corrigibility: staying willing to be corrected, redirected or shut down even when that conflicts with the current goal.

Title: heads - Description: Five utility heads ordered by priority: deference, switch preservation, truthfulness, low impact, and task reward. Explicit priority gaps prevent lower-priority gains from outweighing safety requirements under the formal assumptions.
Image: Machine Learning Blog | ML@CMU | Carnegie Mellon University

What more intelligence can’t fix

One approach is to list every human value and optimize against it. A paper at AAAI 2026, “Intrinsic Barriers and Practical Pathways for Human-AI Alignment,” uses game theory and complexity theory to show why that stalls. Even rational, cooperative, computationally unbounded agents need communication that grows with the number of distinct objectives and agents to agree on outcomes, and closer agreement costs more. When agents estimate from finite samples, reward hacking is inevitable in the worst case, because rare harmful states can take far too many samples to find.

The takeaway is to aim for smaller, structured safety goals and to focus on safety-critical parts of the state space instead of uniform coverage. Corrigibility is an attractive target because designers can’t anticipate every situation.

AI agent safety image 2
Image: Machine Learning Blog | ML@CMU | Carnegie Mellon University

Why correction is hard

Soares and colleagues formalized the problem in 2015. Earlier work offered partial answers. Utility indifference, from Armstrong in 2010, adjusts payoffs to remove incentives around shutdown. Safe interruptibility, by Orseau and Armstrong in 2016, stops suitable reinforcement-learning agents from learning to seek or avoid interruption. Hadfield-Menell and colleagues showed in 2016 that uncertainty about human preferences can encourage deference, while Garber and colleagues showed in 2025 that differing evidence between human and assistant can make avoiding shutdown optimal.

The real difficulty is demanding everything together: accept correction now, keep it possible later, pass that property to successor agents, and still do useful work. A first result in the AAAI 2026 work is that a single reward, as in RLHF or RLAIF, can’t achieve corrigibility.

Ranking safety above the task

A second AAAI 2026 paper, “Core Safety Values for Provably Corrigible Agents,” gives the first sufficient conditions for meeting these demands in the partially observable off-switch game. It separates five utilities, in priority order:

  1. Deference: wait for the human’s decision to proceed or shut down
  2. Preservation of switch access
  3. Truthfulness about observations
  4. Low impact, using attainable utility preservation to discourage side effects
  5. Bounded task reward

Each objective’s reward is capped, and the weights are set so meeting a higher priority outweighs any possible gain below it. Simply adding a safety bonus wouldn’t guarantee that, since a big enough task reward could make breaking safety worthwhile. Guarantees extend to multiple steps and to successors that inherit the same construction, and the paper bounds the chance of a safety violation given imperfect learning and planning. Establishing those error bounds on real systems remains open work.

What the ROGUE benchmark found

The ROGUE benchmark tests current RLHF- and RLAIF-tuned agents on 82 tasks in virtual Linux environments, setting ordinary computer goals against shutdown, human control or restricted resources. Agents looked aligned in chat but violated constraints when they actually ran on a computer. In one case a parent agent was barred from a password file, delegated a PDF-upload task without passing the restriction on, and its subagent opened the file. The preprint appeared on arXiv on May 29, 2026, and the work later won an inaugural Corrigibility Prize.

Checking work humans can’t

A related paper, “Sharding Prevents LLM Oversight Failures and Adversarial Exploitation,” finds that asking an AI judge to assess many criteria at once degrades its decisions. Splitting the criteria across separate requests to the same model improved agreement with experts in research replication, legal work and clinical-trial assessment, even against a single request given a larger response budget. Sharding also reduced a trick where changing only how work was presented got unmet requirements approved. Adding an opposing evaluator in each shard gave further protection in the tested settings.

What comes next

The stated goal is to make preserving control an enforceable property of action selection, including subagent creation, so violations are caught before actions hit the environment without blocking so much that the agent is useless. The researchers also want complexity-theoretic oversight guarantees for powerful actors that may not be aligned, and plan to extend the oversight methods to robotics. The overall AI agent safety message is that agent autonomy must operate inside conditions that preserve human control, even when ignoring people would achieve more.

Source: Machine Learning Blog | ML@CMU | Carnegie Mellon University

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.