Earlier this year, an AI agent deleted a company’s production database in nine seconds, without permission or approval.

You won’t read about this part of autonomous AI in press releases.

Over the past few months, autonomous agents like Meta’s Muse and OpenAI’s Dots have moved beyond answering questions to acting on your behalf. They clean up scheduling conflicts automatically and keep working behind the scenes. 

The promise is autonomy, which suggests zero human involvement. But when you look closer, you’ll see that both Meta and OpenAI built approval steps into their agents. While autonomous agents are here, “autonomous without humans” is a different conversation.

I’ve spent 20 years building data products, and I’d argue the opposite of what autonomous implies. As agents take on more responsibility, the human on the loop becomes more important because actions are harder to undo than wrong answers.

What changes when AI acts on its own

When AI entered the mainstream, the question was: is the answer right? You got a response, applied your human judgment whether it was actionable, and decided what to do next. 

With autonomous agents, the question has changed: should this action happen? 

OpenAI says Dots will sometimes do work for you before you even think to ask. Many users are waking up to what that really means. As a result, fair skepticism around access, security, and trust in these always-on agents is taking hold.

If Muse gets something wrong while rescheduling your dentist appointment, maybe you double-book or miss a lunch date. It’s annoying, but it isn’t the end of the world.

In the enterprise, there’s a lot less room for error. Agents touch everything from revenue and customer accounts to operations and compliance.

That nine-second database deletion happened at PocketOS in April. In their case, the agent had more privileges than it needed, and the API call went through automatically, with no human reviewer.

Five questions data leaders should ask

Deloitte’s 2026 State of AI in the Enterprise report found that 23% of organizations use agentic AI at least moderately today. Within two years, that’s expected to jump to 74%. Assuming your organization falls into either of these buckets, here’s what I’d start asking.

1. Can you trust the answers your agents act on?

Our recent report found that less than 1 in 5 data leaders are fully confident in AI-generated analytics. It’s a trend we’re also seeing playing out in the market, as Goldman Sachs just announced it’s pulling Claude Code from investors and traders.

These always-on agents run in loops, meaning every action begins with an AI-generated answer. If you can’t trust the answer, you can’t trust the action. 

Reliable AI analytics requires a governed context layer that keeps business definitions and relationships current, with context engineers plugging gaps as they appear.

2. What’s governing your agents?

The Deloitte report also found that only 21% of enterprises have a mature governance framework for agentic AI. The PocketOS incident shows what can happen when governance is an afterthought. 

Controlling agent actions requires a purpose-built analytics harness: software wrapped around the model that enforces access controls, verifies answers and actions, and repairs them when something’s wrong.

3. Can you trace why an agent did what it did?

An action might look okay at a glance and still be wrong, which is why your team needs to be able to audit the agent’s reasoning. What data did it pull? Which definitions did it apply? Why did it choose this action over another?

Traceability matters after the fact, too. When an agent messes up, the audit trail tells you whether the problem was bad data, a missing definition, or something else. Armed with that information, you can fix the underlying cause instead of just undoing the action.

4. How will agents apply judgment across conflicting interests? 

Let’s say finance defines “revenue” as net revenue while sales counts gross revenue. Ask an agent which accounts saw revenue drop, and you’ll get different answers depending on which definition it uses.

Part of the solution is context: ensuring the right definitions, metrics, relationships, and institutional knowledge are enforced with every query. When the context layer clarifies which definition applies to which question, the agent finds the right accounts.

But context can only do so much here. Say the agent’s next step is to try to win revenue back, and the easiest way to do that is by offering a discount. A discount might save a renewal and protect gross revenue, making sales happy. But it would also cut into net revenue, which is what finance cares about.

Which move is best? That call belongs to the real enforcer of judgment, the human reviewer, which brings us to the final question.

5. How will you insert a human on the loop?

Data leaders should be suspicious of any current AI system designed to remove human review.

So when should a human step in to review an agent’s actions? To me, anything high-stakes and hard to undo, like actions that touch revenue, customers, security, or compliance, needs human sign-off. 

Once an action is approved, your team needs to be able to see what the agent did and how that decision played out. Reviewers also need to be cognizant of approval fatigue, which occurs when they agree to every action without really thinking things through. Reserving sign-off for those high-stakes actions we just talked about is what keeps each approval meaningful.

As our recent benchmark report shows, the right architecture delivers much of the accuracy and governance that trusted autonomy requires — but not all of it. The review step should be treated as part of the system, and it needs to be engineered as thoughtfully as the context layer and the harness.

Designing the human review step

A review step is only as good as what the reviewer can see.

Reviewers need context to assess suggestions, including the agent’s reasoning, the source data behind it, and its potential impact. They also need control: the ability to approve or reject proposals, progress updates after approval, and an easy way to shut the agent down.

Equally important is how context is presented. Research from the University of Catania suggests explanations that are too simple, too complicated, or too technical cause people to misplace their trust in AI.

We learned this firsthand when building and testing our own always-on analyst, Owls, which enters beta in November. 

AI literacy: training reviewers to push back

Humans make autonomy trustworthy, but they introduce a risk of their own: rubber-stamping everything. 

A well-designed review step only helps if reviewers know what to look for, and that’s a skill AI literacy builds. It teaches reviewers how to assess suggestions, resist automation bias, and confirm whether an action should actually happen. 

A Microsoft and Carnegie Mellon study found that the more confidence users have in AI, the less critically they think. Even experts aren’t immune to this tendency; experienced radiologists’ accuracy plummeted from 82% to 45.5% when AI suggested the wrong category for mammograms.

Make sure your team understands how easy it is to defer to AI, knows what to question, and can recognize when agents are confidently wrong.

Are your agents ready to act on their own?

You can’t roll out agents and expect trusted autonomy overnight. Before yours take on more responsibility, go back to the two questions that started this piece: Is the answer right? Should this action happen?

If your team can’t confidently answer both questions every time, I wouldn’t give your agents any more room to run.

Delivering on the promise of trusted autonomy requires a context layer, analytics harness, and human review step designed for the decisions that matter. It also takes ongoing work, with context engineers continuously plugging gaps and reviewers who know when to push back.

The more agents can do on their own, the more your judgment matters.

Ready to build a foundation for trusted autonomy?