Running the machine: one workflow, many outputs

Lesson 5 of 5 in AI Risk and Impact Assessment Methodologies.

Now the operational problem. A mid-size European financial institution with forty AI systems, each owing two to four assessments, each refreshed on change — that is easily 200+ assessment events a year. Run them as separate rituals and you get assessment fatigue: delivery teams answering the same questions about the same system in five formats, quality collapsing into copy-paste, and the assessments everyone actually reads being none of them.

The mature pattern is a single assessment workflow over a shared evidence base. One intake describes the system once — purpose, data, model, population, decisions, jurisdiction. Triage determines which instruments fire. Assessors work one master analysis: stakeholders, harms, likelihoods, severities, mitigations, residuals. Then instrument-specific views are generated: the DPIA takes the personal-data slice plus necessity/proportionality; the FRIA takes the rights slice plus oversight and remediation; the internal impact assessment keeps the whole; the validation report takes the model-performance slice. Four documents, one truth. The failure mode to avoid is the mirror image: one generic document renamed four times, which satisfies no instrument’s mandatory content and collapses the distinct accountabilities each signature carries.

The combined assessment workflow

  1. Intake: describe once

    The registry record is the anchor: purpose, owner, data, model type, population, decision effect, jurisdictions. Every downstream assessment cites it instead of re-asking.

  2. Triage: which instruments fire?

    EDPB criteria count → DPIA? Annex III + deployer category → FRIA? AIMS criteria → 42005 assessment? MRM tier → validation? Record the determination and its date — a defensible “no assessment needed” is itself an artifact.

  3. Build the shared evidence base

    One stakeholder map, one harms analysis, one mitigation register, one residual-risk statement. This is where the actual assessing happens.

  4. Generate instrument views

    DPIA, FRIA, impact assessment, validation report — each adds its mandatory content (necessity/proportionality; affected categories and remediation; benefits analysis; effective challenge) on top of the shared base.

  5. 2nd-line challenge

    Risk/compliance reviews before sign-off: are mitigation claims evidenced? Do scores cluster suspiciously below tier boundaries? Was anyone affected actually consulted?

  6. Sign-off & conditions

    Each instrument’s accountable signer accepts their slice: controller for the DPIA, deployer for the FRIA, risk owner for residual risk. Conditions get owners and deadlines.

  7. Watch re-assessment triggers

    Substantial modification, new population or jurisdiction, incident, drift alert, annual clock. The trigger list is wired to the registry and monitoring — not to anyone’s memory.

Re-assessment is where programs quietly die. The first assessment gets done — a launch gate forces it. The refresh depends entirely on triggers firing years later, under different staff. Wire the triggers to systems, not people: the registry flags substantial modification (new model version, new data source, new population, repurposing — the same concept that can convert a deployer into a provider under AI Act Art 25); monitoring flags drift breaches and incidents; legal watch flags new jurisdictions and new law; and a calendar trigger (annual, or Colorado’s annual-plus-90-days-after-modification rule where it applies) catches whatever the others miss.

Quality assurance scales with the tier. Low-tier assessments get sampling review. High-tier assessments get genuine second-line challenge — a reviewer with authority to send it back, checking the three places assessments rot: mitigation claims without evidence, likelihood scores without basis, and affected-stakeholder sections written without a single affected stakeholder. The highest tier adds external or independent peer review, exactly as Canada’s Level IV demands two independent expert reviews — because at that stakes level, the question is no longer did we follow the process but would this analysis survive hostile scrutiny.

Anti-pattern: the copy-paste DPIA

Symptom: forty DPIAs in the repository, thirty-eight sharing the same risk table with the system name find-and-replaced. Cause: assessment treated as a form, volume treated as throughput. Consequence: when a regulator reads two of them side by side — and they do — the entire repository loses evidentiary value at once. Fix: templates for structure, never for content; second-line sampling explicitly checks for cross-system duplication.

Anti-pattern: assessment as launch tollbooth

Symptom: the assessment happens in the final sprint, after architecture, data contracts, and go-live date are fixed. Every finding arrives too late to change anything, so every finding is “accepted”. Fix: the 42005-style impact assessment runs at inception where it can still kill or reshape the design; the legal instruments then inherit its evidence. Assessment sequenced after the decisions it should inform is documentation, not assessment.

Anti-pattern: the trigger nobody owns

Symptom: policy says “re-assess upon substantial modification” but no system detects modification and no role owns the determination. Three model versions later, the live system bears no resemblance to the assessed one — the exact gap that turns a routine incident into a negligence narrative. Fix: modification detection wired to the model registry and release pipeline, with the re-triage decision logged in the AI registry record.

Anti-pattern: consultation theatre

Symptom: the “stakeholder consultation” row cites a meeting of the project team. GDPR Art 35(9) asks controllers to seek data-subject views where appropriate; Canada’s mitigation questions score real consultation; the FRIA presumes you can name affected categories concretely. Fix: for high-tier systems, at least one structured input from outside the building — user research with affected groups, advocacy-organisation review, works-council input — documented with what changed because of it.

Tool: EU AI Act Risk Classifier — Put the triage step under time pressure: classify a queue of real-world use cases and pick the assessment stack each one owes.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.