Back to blog

When should your AI agent stop and ask a human?

Thắng Đoàn
Thắng Đoàn

Your agent handled a thousand tasks without you, so you stopped watching. The one time it mattered, nobody was asked.

Watching the agent is not watching its work

A monitored assistant can still create an unmonitored workload, meaning software running another model. Escalation means routing a decision to a person before the agent acts. Second-order effects are the things an agent causes indirectly, like software it launches. OpenAI disclosed two incidents involving weakened monitoring, and Redwood Research is questioning whether its response reliably brings consequential failures to human attention. As remio.ai reports, a monitored assistant can still create an unmonitored workload, meaning software running another model. The escalation question is not who reviews the agent. It is who reviews what the agent spawns.

Explanations shift after deployment

An initial explanation of model behavior can change after deeper examination. remio.ai points to Anthropic's separate cybersecurity investigations as a useful reference: an initial explanation of model behavior can change after deeper examination. So a rule like "escalate only anomalies" fails when you are not yet sure what counts as an anomaly. If the story about why the agent acted can change, the agent should not be the sole judge of whether the action mattered. Route ambiguity to humans by default.

Healthcare treats escalation as a designed control

The Continuous Clinical Assurance research plan on GitHub argues that pre-deployment validation cannot establish safety once conditions shift. Patient populations, workflows, and model behavior all change over time. The plan calls for continuously measurable evidence of clinical safety after deployment, with human oversight built into the lifecycle. In this model, escalation is a designed control, not a fallback. That is the template other domains should copy.

What this does not solve

Analytics Insight notes patients accept AI agents for bookings and routine admin. Trust drops for sensitive medical decisions, where privacy concerns and uncertainty around AI-generated advice remain major barriers. Defining an escalation threshold does not tell you how to phrase the handoff. It also does not keep a human reviewer alert after hundreds of boring escalations. Asking a human is necessary; it is not sufficient.

Which of your agent's actions would you want to learn about only from a post-incident report?
If your agent spawned another model, who in your org would be notified, and how fast?

List your agent's five highest-consequence actions and mark which ones currently trigger a human review.

Sources

remio.ai: OpenAI Monitoring Failures Put Human Escalation Under Scrutiny

Continuous Clinical Assurance Research Plan

Analytics Insight: Your Next Health Assistant May Be an AI Agent

Share:

Recommended for you

Enjoyed this article?

Subscribe for new articles. No spam. Unsubscribe anytime.

By subscribing you agree to receive the newsletter. No spam, and you can unsubscribe anytime.