Complying with the AI Act alone won’t make you resilient. That’s the thesis of a new Oxford paper that separates two questions we tend to conflate: is my AI trustworthy? and does my business survive if that AI fails? These are different things, and “ethical AI” regulation doesn’t answer the second one.

Two regulatory logics that don’t talk to each other

  • The trustworthy-AI stack (EU AI Act, ISO/IEC 42001, NIST AI RMF, and the FCA/PRA principles) asks: is this system safe, fair, and explainable? Its unit of analysis is the model or AI system, and its notion of failure is harm, bias, or opacity. Translated into concrete process questions: is there human oversight over automated decisions? Is everything documented? How was the model built, what testing did it undergo, who is accountable for it? If you want to dig deeper into what building responsible AI actually means, we covered this coming out of MWC.
  • The operational resilience stack (the UK’s operational resilience regime, DORA in the EU, and the Critical Third Parties regime) asks something entirely different: if something fails, does the service survive or recover in time? Its unit of analysis is the important business service, and its notion of failure is unavailability or intolerable disruption. Translated into concrete process questions: if something breaks, can the customer still pay with their card? And if they can’t, how long does it take for the service to come back?

Yet the central problem the author raises is that these two frameworks almost never connect within organizations. AI governance and model risk teams typically operate in silos separate from operational resilience teams and the accountable individuals under regimes like the SMCR. The result: an AI dependency can be perfectly governed from an ethical and regulatory standpoint, without ever having been mapped to an important service, assigned an impact tolerance, or assessed for substitutability.

The “silent” failures traditional resilience doesn’t catch

So why do classic resilience controls fall short for AI? one of the paper’s most interesting contributions is identifying why classic resilience controls fall short for AI. Traditional systems fail in binary fashion: they’re either available or they’re not. AI, by contrast, introduces silent degradation: a model can keep responding normally, within expected timeframes, while its decisions become progressively less correct.

Because of this, monitoring built to detect service outages will show everything as healthy while the system deteriorates underneath. This is a resilience failure that presents as full availability, invisible to controls calibrated to catch outages.

Provider concentration as systemic risk

  • From the “trustworthy AI” lens, a third-party provider is essentially a source of paperwork: documentation, certifications, proof of compliance. It’s treated as a source of written assurances.
  • From the “resilience” lens, that same provider looks completely different: a potential weak point in the system. The question is no longer “are the papers in order?” but “what happens if this provider fails, and how many companies depend on it at the same time?”

The AI Resilience Framework as the central proposal

To close this gap, the author proposes a five-step framework, designed to plug into the resilience management firms already have, rather than creating a separate process just for AI:

  1. Dependency mapping: identifying which AI components underpin each important business service.
  2. Criticality-Substitutability Matrix: classifying each dependency by its criticality to the service and its substitutability (substitutable, degradable, or irreducible), producing four treatment tiers, from the “danger zone” with no viable alternative (Tier 1) to light-touch treatment (Tier 4).
  3. Extended impact tolerances: going beyond availability to incorporate correctness/degradation thresholds, with data drift monitoring and continuous output validation.
  4. Real fallback doctrine — the paper’s most forceful argument: when an AI system has replaced a manual process, and that process has been decommissioned or left to atrophy, the contingency plan is fictional: it exists on paper, but has none of the resources — people, practice, capacity — to actually be executed. Detecting these cases should be treated as a first-order resilience finding.
  5. Provider-level concentration management: it’s not enough to assess whether an AI provider is trustworthy or compliant. The question is also: “how easily could we stop depending on this provider if something goes wrong?” This means treating foundation model providers as critical suppliers, evaluating not just their assurances (security, certifications, compliance) but the firm’s own ability to substitute them if needed.

The only thing missing: mapping AI like any other dependency

Put simply, companies already have registers of important services, impact tolerances, and operational resilience teams. The only thing missing is to stop treating AI as an exclusive topic of “ethics and AI governance” and start mapping it into that same structure, as just another dependency — the same way a database or a cloud provider would be treated.

Ultimately: being trustworthy answers whether the system deserves confidence. Being resilient answers whether the business stays standing when that confidence isn’t enough. These are different questions — and only one of them saves you when the provider goes down at three in the morning.

*Cover image generated with Artificial Intelligence*