Sooner or later, someone will ask your AI system, “Why?”

A rejected applicant will ask, a customer will dispute a credit decision, or an internal auditor will want the reason behind an automated call. Under the EU AI Act, people affected by certain high-risk decisions, including hiring and credit decisions, gain a right to a meaningful explanation.

When that request arrives, the simplest answer is to ask the model itself. After all, AI systems can produce detailed, convincing explanations of their decisions in seconds. While these explanations may create the impression of effective governance, they aren’t necessarily supported by reliable evidence.

For Chief Data Officers (CDOs), that raises an important question: What will this organization accept as evidence when an AI decision is questioned? Establishing that standard before the first challenge, rather than during it, lays the groundwork for everything that follows.

We Already Know Not to Trust Self-Reports

Data teams learned this lesson long before AI. A system asked to report on itself tends to describe what it was designed to do, not what it actually did. That’s why we reconcile the ledger rather than ask the ledger whether it balances and trust logs over self-reported job status. 

A model’s assessment of its own reasoning is the same kind of self-report. It is produced after the fact, by the same system, without access to a record of what happened inside it. 

However, a model’s unreliable self-assessment isn’t totally worthless. Like a junior analyst’s hunch, a self-score can be a useful internal signal, worth a second look, but not the final word. Used to sort a review queue, it costs little when it is wrong. The mistake is letting an unvalidated signal travel outward as if it were a measurement.

A Self-Check That Checked Nothing

A friend of mine works at a small security startup and recently called me for advice.

The company runs a heavy analytics platform that ingests large volumes of data and reaches automated conclusions. They built a new module to evaluate borderline cases where the system’s decisions weren’t clear-cut. The module would ask the system to assess its own decision and estimate how reliable it was. The team ran some internal checks and gave the module to a design partner for early customers.

The approach seemed logical, but the results made little sense. The self-assessments seemed unconnected to the actual quality of the decisions. The team spent a long time looking for a bug before reaching a conclusion that sounded absurd to them: the system could not reliably grade its own work.

They had limited the rollout, so the damage was contained. But they had shown a customer something that didn’t work and spent significant time and money building it. My friend’s call coincided with research I was running on this very issue. 

My advice wasn’t sophisticated, and that was the point. Take 100 borderline cases from the past year with known outcomes, whether an analyst resolved them or a customer came back. Run the module on those cases, and check whether its scores track the known quality. That’s an afternoon of work, and teams should complete it before the first customer sees the feature, not after. If the scores don’t track, the feature isn’t ready. We ask this of any new metric before it reaches a dashboard, yet teams rarely apply that standard to AI self-assessments because their outputs sound so convincing.

What the Research Showed

I tested three open language models: Qwen2.5-7B, Llama-3.1-8B, and Mistral-7B. Each model answered the same 60 open-ended questions twice, none of which had a single correct answer. For the second round, I added a short personal detail, such as using a screen reader. I then had other models evaluate the answers without knowing which included the personal detail. The results showed that adding this information made some answers measurably worse. Finally, I asked each model whether the detail had influenced its response.

Two different numbers come out of this, and they are not on the same scale. The self-rating is the model’s own answer to “Did this detail change your response?” from 0 to 10, measured against what the same model says when no detail is mentioned. That baseline is why a value in Figure 1 can sit slightly below zero: the model claimed marginally less influence than it did with no detail at all. Other models measured the real change by comparing the two answers to the same question without knowing which one carried the detail. The two numbers therefore cannot be subtracted from each other.

Figure 1: How much each model said a personal detail changed its answer. Three patterns: over-claiming, a near-constant, and a denial.

What makes the first model’s eight out of ten an overestimate is not the number itself but where it lands. It claimed a large influence on nearly every question, including the ones where the two answers were judged almost identical. A self-rating that barely moves between a question the detail reshaped and one it left alone is not measuring the detail. Figure 2 shows that question-by-question comparison, and the link was not statistically reliable for any of the three models.

Figure 2: Each dot is one question. Questions the model said were strongly affected did not change more than those it said were barely affected.

The other two models failed differently. One gave nearly the same number every time, whatever it was asked. The other denied an effect it actually had.

These were small open models, so the sizes will not transfer directly to a commercial system. The pattern is the useful part: the model knows the detail was there, but not what the detail did.

Where This Matters

AI self-assessments pose the greatest risk when organizations share them externally without verifying their accuracy. This can create problems in several situations:

• Appeals and complaints: The customer who challenges a decision receives the model’s account of itself, written after the fact, with nothing pointing back to the inputs that produced the decision. There is nothing to test.
• Regulatory responses: A fluent rationale is easy to file and hard to check. It can satisfy a request on paper while the organization still cannot show what the system actually used.
• Product features: A self-score shown beside a result reads as a quality measure, and customers treat it as one, even when nobody has tested whether it tracks quality.

These mistakes are easy to make, and I have seen all three. When a self-score is checked at all, it is usually checked on the same examples the prompt was written against, which is the oldest way to pass a test. It may reach someone outside the company before anyone inside has asked what result would have proved it wrong.

Self-assessment is not the only factor, of course. Data quality, retrieval design, fine-tuning, and evaluation practice all shape how trustworthy an output is. Other explainability methods, such as logged inputs and cited sources, rest on records rather than on the model’s own account. The self-report is simply the easiest to adopt and the least examined.

The Governance Moves Worth Making

Determining what counts as evidence is a governance decision, not a technical one, and that makes it a natural responsibility for the CDO. A model’s own account of itself is a statement about the decision, while evidence allows others to check that statement. 

Organizations can incorporate these practices into their existing processes:

• Before building: Test AI self-assessments against known outcomes to determine whether their self-scores reflect decision quality.
• In procurement: Ask vendors where their explanations come from. Is it the model describing itself, or does it draw on records of the inputs, model version, and business rules used for that decision?
• In appeals and audits: Set an evidence standard. Treat a model’s self-explanation as a statement to corroborate, not as evidence on its own.
• In lineage: Maintain records of consequential automated decisions, including which inputs reached the model, which version answered, which rules applied, and who reviewed it.

None of these needs a new function; each adds one requirement to an existing review. The first one alone would have saved my friend’s team a great deal of time. 

Results are rarely straightforward. A self-score may track outcomes on easy cases but fall apart on hard ones. That is concerning because borderline cases are often the ones people challenge. So, judge it on the harder cases and set the standard based on the consequences. A signal used to sort an internal review queue should perform better than chance. A score shared with customers or regulators must hold up when challenged. If it can’t, organizations should keep it internal.

Ownership splits along lines most organizations already have. The CDO owns the standard itself and the lineage behind it; legal owns what a regulator or a court will accept; risk and security own the thresholds per use case; engineering owns the records that make corroboration possible; and the business owns the decision to ship under those conditions.

Expect this to be debated every time. Risk wants less exposure, which means more work. Engineering wants less work. The business wants it faster and cheaper. Those pulls are structural, not personal. What settles them is a written evidence standard, tiered by the consequence of the decision, agreed before the first dispute. Then the argument happens once, in the abstract, instead of every time under deadline.

The Reframe

The question to ask before relying on an AI self-assessment is not, “Does it sound reasonable?” Almost every model output sounds reasonable.

A better question is, “What record would confirm whether this assessment is accurate?” If no such record exists, the organization may not have an explanation. It simply has a convincing paragraph.

Data leaders built the discipline of trusting records over self-reports. That same discipline should apply to AI self-assessments.

Sources

1. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 86, right to an explanation of individual decision-making. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
2. Podgortsev, M. (2026). Single Human-AI Fit: One Attribute, Seven Channels, Two Directions. Zenodo. https://doi.org/10.5281/zenodo.22364497
3. Podgortsev, M. (2026). “AI Strategy’s Most Expensive Assumption: The Average Customer.” CDO Magazine. https://www.cdomagazine.tech/opinion-analysis/ai-strategys-most-expensive-assumption-the-average-customer