Explainability at governance altitude: SHAP, LIME, counterfactuals

Lesson 6 of 6 in Testing and Evaluation: Fairness Math, Red-Teaming, and Explainability.

Governance does not need you to derive Shapley values. It needs you to know what each explanation technique actually delivers, where it deceives, and which audience it serves — because “the model is explainable” is a claim you will have to defend to validators, regulators, and the person who was denied.

Three families dominate post-hoc explanation of black-box models. All three answer subtly different questions, and none of them opens the box — they describe behaviour from outside.

SHAP

What it does: attributes a prediction to each input feature using Shapley values from game theory — each feature’s fair share of the difference between this prediction and the average prediction. Local (this decision) and aggregatable to global (the model overall). Additive: contributions sum to the output.

Trade-offs to know: exact Shapley computation is exponential; TreeSHAP computes it efficiently and exactly for tree ensembles (why SHAP owns credit scoring, where gradient-boosted trees dominate), while KernelSHAP approximates for arbitrary models — slower, sampling-based, and less stable. Correlated features can split or misassign credit.

Governance role: the de facto engine behind adverse-action reason codes in US credit — top negative SHAP features become the “principal reasons” on the notice.

LIME

What it does: LIME fits a small interpretable surrogate model (usually sparse linear) to the black box’s behaviour in the neighbourhood of one input: “near this case, the model acts as if these features matter this much.”

Trade-offs to know: the result depends on how you perturb and how you define “near” — run LIME twice on the same case and you can get different explanations. That instability is a documented governance hazard: an unstable explanation cannot support a consistent legal justification.

Governance role: quick model-debugging and validator exploration; increasingly disfavoured as the basis for regulated explanations precisely because of instability. If LIME output goes to a customer, version and archive the exact configuration.

Counterfactuals

What it does: finds the smallest change to the input that flips the decision: “had your annual income been €5,200 higher, the loan would have been approved.” No feature-importance math at all — a statement about the decision boundary.

Trade-offs to know: the nearest counterfactual may be non-actionable (“be five years younger”) or rest on immutable or protected attributes; good implementations constrain the search to features the person can change. Multiple valid counterfactuals exist for one decision — choosing which to show is a policy decision.

Governance role: the format regulators and affected individuals consistently understand best — it doubles as recourse, telling the person what to do next, which is the practical spirit of GDPR’s “meaningful information” and of contestability rights.

Match the explanation to the audience — the most-failed exam question in practice
Audience	What they need to do	Right explanation artefact
Model validator (2nd line)	Challenge whether the model works for the right reasons and where it breaks	Global SHAP importance, partial-dependence behaviour, per-segment error analysis, faithfulness and stability tests of the explanation stack itself
Regulator / auditor	Verify documented logic, reproduce results, check compliance claims	Technical documentation with pinned model + explainer versions, the pre-registered metric choices, disaggregated results — reproducibility over readability
Affected individual	Understand the decision and act on it — or contest it	Plain-language principal reasons plus an actionable counterfactual (recourse); never raw SHAP numbers
Frontline overrider (loan officer, clinician)	Decide in seconds whether to trust or override this specific output	Local top-factors display with calibrated confidence and known-failure-mode warnings — designed against automation bias, not just for information

Where explanation is a legal duty

ECOA / Regulation B (US credit): adverse-action notices must state specific principal reasons for denial. CFPB Circular 2022-03 closed the black-box loophole: a creditor cannot claim its model is too complex to explain — if you cannot generate accurate reasons, you may not use the model. SHAP-derived reason codes are the industry’s standard answer.

GDPR Arts 13–15 & 22: “meaningful information about the logic involved” in automated decisions, plus the Art 22 safeguards (human intervention, the right to express a view and to contest). The CJEU’s SCHUFA line of cases pushed scoring itself into Art 22’s reach.

EU AI Act Art 13: high-risk systems must be transparent enough for deployers to interpret and use outputs appropriately — instructions for use carry the system’s capabilities, limitations, and performance characteristics. (Art 86 adds a right to explanation of individual decisions for affected persons.)

Key terms: shap, lime, counterfactual explanation, interpretable by design, explanation faithfulness, adverse action notice

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.