(This article originally appeared in CDO Magazine’s Trend Report, Generative + Agentic AI: Moving From Experimentation to Enterprise Scale.) 

There is a phrase enterprise leaders reach for whenever an AI conversation gets uncomfortable, the way a nervous magician reaches for a rabbit in the hat: human-in-the-loop. Someone will be watching. Someone will approve. Someone, somewhere, will save us from ourselves. It is the corporate equivalent of “a friend will drive.” It sounds responsible. Increasingly, it is a fiction.

In a zero-click world, decisions increasingly happen before a person ever asks, clicks, or compares. Generative and agentic AI do not merely accelerate enterprise workflows; they quietly shape which options surface, which risks get flagged, which cases escalate, and which actions execute without anyone in the room. For the chief data officer (CDO) and the enterprise data leaders who sit next to them, this is not a philosophical shift; it is a job-description shift. The data, the models, the lineage, and the guardrails that determine whether any of those hidden decisions are defensible often report, directly or dotted-line, to them. The interesting question is no longer “How do we use AI?” It is “Where, exactly, does AI sit in the chain of judgment — and is the human in the loop actually load-bearing, or just decorative?”

The numbers say: mostly decorative. McKinsey’s most recent State of AI survey of nearly 2,000 respondents reports that 88 percent of organizations now use AI in at least one business function, and 62 percent are experimenting with or scaling AI agents — but only 23 percent have begun scaling those agents in even one function. Deloitte’s State of AI in the Enterprise puts a finer point on it: just one in five companies has a mature governance model for autonomous agents. And Capgemini finds that 71 percent of organizations do not fully trust AI agents for enterprise use — while cheerfully deploying them anyway.

This is the enterprise AI paradox in one line: we have crossed the adoption threshold and skipped the accountability one. It is also, quietly, the CDO’s problem to close. Adoption is a business decision; accountability is a data-and-governance decision, and the office that owns data quality, lineage, model inventory, and policy is the office that will be asked, under oath or under a headline, what the loop actually contained. Or, as Figure 1 shows with the diplomatic restraint of a bar chart:

Figure 1: Enterprise AI adoption vastly outpaces mature agent governance. Sources: McKinsey; Deloitte.

It would be one thing if this gap were merely embarrassing at conferences. It is worse than that. The gap is where lawsuits, resignations, and Royal Commissions are born.

A Brief Museum of Loops That Weren’t

History has been kind enough to hand us three case studies of what happens when “human oversight” is really just human proximity to a machine no one understands. They are worth revisiting because every enterprise AI deployment quietly rhymes with at least one of them.

The Apple Card, launched in 2019 by Apple and underwritten by Goldman Sachs, is the polite one. Customers, including tech entrepreneur David Heinemeier Hansson and Apple co-founder Steve Wozniak, alleged in a viral Twitter thread that spouses with shared finances were receiving wildly different credit limits. The New York State Department of Financial Services launched an investigation, analyzed underwriting data for nearly 400,000 New York applicants, and, in March 2021, concluded that there was no unlawful discrimination. Vindication! Sort of. The same report noted “deficiencies in customer service and transparency” — that frontline staff could not explain the algorithmic decisions to the humans holding the plastic. The model was defensible in a regulator’s conference room. It was inexplicable at a kitchen table. That is a governance failure even when it is not a legal one — and it is exactly the kind of failure the CDO function is built to prevent, because the fix lives upstream in feature documentation, model cards, and the data-to-decision lineage that lets a service rep explain why a limit came back the way it did.

Australia’s Robodebt scheme, which ran from July 2015 to November 2019, is the tragic one. Services Australia used income averaging — taking annual tax data and dividing it by twenty-six fortnights — to infer welfare overpayments, then automatically issued debt notices demanding repayment. The Royal Commission’s 990-page 2023 final report concluded that the scheme was unlawful and, in its most-quoted finding, that there was “no meaningful human intervention in the calculation and notification of debts.” The scheme raised hundreds of thousands of debts totaling roughly A$1.76 billion, was linked to suicides, and produced a class-action settlement in excess of A$1.8 billion. It also cost several senior careers, though those recovered faster than the families did. The methodology behind that averaging — the choice of input, the granularity, the fitness of the data for its inferred purpose — is precisely the terrain a Chief Data Officer is expected to interrogate before a model becomes policy. Robodebt is what happens when no one with a data seat at the table is empowered to say the source data will not support the conclusion the system is about to enforce.

The Dutch toeslagenaffaire is the darkest. Beginning in 2013, the Belastingdienst (Dutch tax authority) used a self-learning risk-classification model that treated non-Dutch nationality as a fraud risk factor. Roughly 26,000 families — mostly low-income, disproportionately of migrant background — were falsely accused of childcare-benefit fraud and ordered to repay tens of thousands of euros. More than a thousand children were placed in state custody as a downstream consequence. Civil servants were required to conduct “manual review,” but they were not told why the algorithm had flagged an applicant; they treated a black-box risk score as objective fact. Prime Minister Mark Rutte’s entire cabinet resigned in January 2021 over the scandal. Every step of that failure — the feature selection, the training data, the absence of intelligible explanations at the point of review — sits inside the modern CDO remit. It is a reminder that in agentic systems, the ethics of a decision are set in the data pipeline long before anyone gets to “review” anything.

Three cases, three different loops, three different ways the loop was a Potemkin village. Figure 2 catalogs the specific failure mode in each:

Figure 2: Three enterprise-scale automation failures, categorized by which control in the human-oversight chain actually collapsed.

The through-line is uncomfortable. In every case, an organization could point to a human somewhere in the process — a customer-service rep, a compliance officer, a civil servant — and claim the loop was intact. It was not. The loop had been quietly reduced to a person present but powerless: no context, no authority, no explanation, no appeal path. “Human-in-the-loop” had become theater staged for the auditor. In each case, the accountability vacuum tracks the absence of a senior data leader with the mandate — and the veto — to insist otherwise.

Redefining the Loop (Or: How to Keep Your Cabinet)

The enterprises that succeed with agentic AI will not remove humans from the loop. They will do something harder: redefine what the loop is. Governance, operational risk, and oversight have to move from after-the-fact controls into embedded systems of accountability. For CDOs and enterprise data leaders, this is the operating agenda — the point where the data function stops being asked to “ensure quality” after the fact and starts being asked to co-design the control system with legal, risk, and the business. That means four things (see Figure 3), and none of them can be bought as a product.

Figure 3: Four load-bearing controls that convert “human oversight” from a compliance checkbox into an operational control system.

1. Decision Rights

Every automated decision should have a named owner accountable for the outcome, not the code. If the agent approves a refund, denies a claim, or drafts an email to a customer, someone with a title and a phone number owns that outcome. If you cannot name the person, you do not have governance; you have plausible deniability with a dashboard. The CDO is now the natural steward of the register that maps agents to owners, use cases to policies, and models to the data they consume — because no one else has the full graph.

2. Explainability at the Edge

Explainability cannot live only with model owners or regulators. If the frontline employee talking to a customer cannot explain — or override — an automated decision, the model’s legal defensibility is cold comfort. Regulators found no evidence that Apple Card violated fair-lending law. It was also a customer-service disaster. Both things were true. Building explainability into the customer-facing edge — plain-language reasons, contestability paths, disclosure standards — is a data-leadership deliverable, not a UX afterthought, and it is one that CDOs should be actively shaping alongside product and compliance.

3. Escalation Thresholds

Agents should be built to stop, not just to act. Every autonomous system needs pre-committed triggers — confidence scores, novelty detectors, financial thresholds — that force human judgment back into the process before the action is irreversible. Robodebt’s true failure was not bad math. It was the absence of a checkpoint with authority to say, wait, this doesn’t smell right, before an enforceable debt shipped. Defining those thresholds — what “low confidence” means, when a novel population triggers a hold, what dollar amount routes to a human — is work that belongs on the CDO’s desk, jointly with risk and the business owner, before an agent is ever cleared for production.

4. Continuous Monitoring

Static risk assessments age like unrefrigerated fish. Agentic AI systems change behavior in production, integrate with new tools, and drift from their design intent in ways their designers cannot fully anticipate. Governance must be a running process — logs, drift metrics, sampled reviews, and red-teaming — not a launch-day artifact stapled to a compliance memo. In practice, that means the CDO helps own the AI model inventory, the drift telemetry, and the periodic re-attestation cadence, so that governance keeps pace with agents whose behavior can quietly shift in production. 

None of this removes humans from the loop. It restores their weight in it. The organizations pulling ahead in enterprise AI are not the ones with the most agents in production; they are the ones whose humans can still explain, override, and be accountable for what those agents do — and, increasingly, whose CDOs and data leaders are treated as co-authors of the control system rather than downstream custodians of its exhaust. Everyone else is running a very expensive magic trick — and eventually, someone in the audience notices the dove.