FDA devices and bank model risk: the two mature regimes

Lesson 4 of 5 in Sectoral Enforcement: FTC, EEOC, CFPB, FDA, and Financial Regulators.

Two US regimes were governing algorithms rigorously long before ‘AI governance’ had a name. Study them not as trivia but as previews: what every other sector is slowly reinventing, medicine and banking already run.

FDA. AI-enabled software that diagnoses, predicts, or treats is a medical device, and more than 1,200 AI/ML-enabled devices have been authorized — overwhelmingly through risk-based classification (most via the 510(k) and De Novo pathways; the riskiest via premarket approval). The regime’s hard problem is the one that defines AI: the product changes after clearance. FDA’s answer is the Predetermined Change Control Plan (PCCP): the manufacturer specifies in the marketing submission what the model will be allowed to learn, how updates will be validated, and the guardrails — and FDA authorizes the change process, not just a frozen artifact. Around it sit the Good Machine Learning Practice principles (2021, joint with UK and Canadian regulators) and the January 2025 draft guidances on AI-enabled device software lifecycle management and on AI in drug and biologic development — the whole package built on a total product lifecycle philosophy: authorization is the beginning of oversight, not the end.

Banking. In 2011 — years before the deep-learning boom — the Federal Reserve and OCC issued SR 11-7 / OCC 2011-12, Supervisory Guidance on Model Risk Management, after the financial crisis showed what unvalidated models do to a balance sheet. Its definition of ‘model’ (a quantitative approach processing inputs into estimates) comfortably captures ML systems, so when banks adopted AI, the framework was already waiting. Its architecture will look familiar — because the NIST AI RMF rhymes with it deliberately. Practitioners call it the RMF before the RMF:

• Effective challenge as the master principle: every model faces critical review by qualified people with authority and incentives independent of the model’s owners.
• Validation on three legs: conceptual soundness (is the design defensible?), ongoing monitoring (is it behaving in production?), and outcomes analysis (do predictions match reality — backtesting).
• Three lines of defense: the business that owns and uses the model; an independent model risk management function that validates it; internal audit checking the whole system.
• A model inventory — you cannot govern what you have not listed — and full applicability to vendor models, where banks must demand documentation and test what they cannot see. Sound familiar? It is every third-party clause in every modern AI framework, fifteen years early.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.