Fairness: pick your definition, because you cannot have them all

Lesson 3 of 5 in Core Ethics Principles and Trustworthy AI.

Every charter demands "fairness". This lesson shows you why that word, unqualified, is an unsigned cheque — and why the person who chooses the definition of fairness is making a policy decision, not a technical one.

Start with the tempting shortcut that does not work: fairness through unawareness — just delete the protected attribute. Remove race from the model and it cannot be racist, right? Wrong, and you already know why from the harms module: proxies. Postal code, school attended, shopping patterns — models reconstruct the deleted attribute from its correlates. Worse, deleting the attribute makes bias harder to measure, because now you cannot even test outcomes by group. Blindness is not fairness; it is unaccountability.

So the field defines fairness on outcomes, measured by group. Three definitions dominate every debate, every audit, and every lawsuit. Meet them through a single running example: a model scoring loan applicants from Group A and Group B.

Demographic parity

Equal approval rates. The share of Group A approved equals the share of Group B approved — full stop, regardless of anything else about the applicants.

Intuition: the model’s decisions should not redistribute opportunity along group lines.

The objection: if the groups genuinely differ on repayment-relevant factors (because of, say, historical inequities in wealth), forcing equal rates means holding the groups to different evidentiary standards — and ignoring qualification entirely. Demographic parity looks at decisions only, never at outcomes.

Equalized odds

Equal error rates. Among people who would repay, the same share of each group gets approved (equal true-positive rates — the "equal opportunity" variant); among people who would default, the same share of each group gets wrongly approved (equal false-positive rates).

Intuition: the model’s mistakes should not concentrate on one group. A qualified applicant’s chance of being wrongly denied shouldn’t depend on their group.

The objection: it requires knowing true outcomes (who actually repays), which you observe only for people you approved — the selective-labels problem auditors fight constantly.

Calibration

Scores mean the same thing for everyone. Among all applicants scored "70% likely to repay", about 70% actually repay — in both groups. A lender reading the score needs no group information to interpret it.

Intuition: the score is honest as a prediction, uniformly.

The objection: a model can be perfectly calibrated while its errors land unevenly — calibration says nothing about whose lives the false denials concentrate in. Which is exactly what happened in the most famous fairness fight ever.

Why couldn’t they? In 2016–2017, Jon Kleinberg’s group and Alexandra Chouldechova proved it, independently: when two groups have different base rates (different underlying frequencies of the outcome — reoffense, repayment, disease), no non-perfect model can be calibrated for both groups and have equal false-positive and false-negative rates across them. Not "it’s hard". Mathematically impossible — as impossible as a square triangle.

Sit with what that means, because it is the single most policy-relevant theorem in this field:

  1. "Make the model fair" is an incomplete instruction. Fair by which definition? Someone must choose, and every choice sacrifices another definition.
  2. The choice is normative, not technical. Equalized odds prioritises not concentrating mistakes on one group; calibration prioritises honest scores for decision-makers; demographic parity prioritises equal access. These are values in mathematical clothing.
  3. Vendors claiming "our model is unbiased" owe you one question: by which metric — and which metrics did you thereby give up? A vendor who cannot answer has not done the analysis.

And a warning against fatalism: the theorem licenses choice, not surrender. Impossibility of everything is not permission for nothing — you can still test, choose deliberately, document the choice, and reduce disparities within it. The theorem also says nothing against fixing the data and design problems (bad labels, proxies, unrepresentative sampling) that create avoidable unfairness under every definition.

Choose a fairness notion for your system

Interactive decision tree — outcomes:

  • Fairness through unawareness — the fallacy

    Proxies (postal code, school, spending patterns) reconstruct the deleted attribute, so discrimination survives — while your ability to measure it by group is destroyed. Auditors and regulators treat attribute-blindness as an anti-pattern: you generally need protected attributes at testing time precisely to prove outcomes are fair.

  • Calibration

    Ensure a score of X means the same outcome probability in every group, so downstream decision-makers can trust scores at face value. Document the trade: calibration under different base rates mathematically forces unequal error rates — monitor where the mistakes land, and say so in your fairness statement.

  • Fix your evidence problem first

    You cannot verify equalized odds without ground truth for rejected applicants. Options: audit studies, holdout approvals, natural experiments, or proxy outcomes — each with limits. Until then, error-rate fairness claims about your model are assertions, not measurements. This evidentiary gap is one of the most common findings in real bias audits.

  • Equal opportunity (equal true-positive rates)

    Qualified individuals get the same chance of a favourable decision regardless of group — the variant of equalized odds that guards the harm you named (wrong denials). Accept and document the consequence: with different base rates, your scores will not be perfectly calibrated across groups.

  • Equalized odds (both error rates equal)

    Neither wrong denials nor wrong approvals concentrate on one group. The strictest error-rate standard — and the one COMPAS failed in ProPublica’s analysis. Expect an accuracy or calibration cost when base rates differ, and put that trade-off in writing for your decision-makers.

  • Demographic parity

    Equal selection rates by group — coherent where equal access is the point and individual "qualification" is not well-defined (outreach, ad delivery, top-of-funnel screening). Less defensible for individualized risk predictions, where it ignores outcome-relevant differences entirely.

  • Parity alone won’t carry an outcome prediction

    For individualized predictions, pure demographic parity ignores qualification and outcome data — it can force approving unqualified applicants of one group while denying qualified applicants of another. Go back and weigh calibration or error-rate fairness; consider parity as a monitoring signal (large rate gaps warrant investigation — the "four-fifths rule" logic) rather than the target itself.

Key terms: demographic parity, equalized odds, calibration, base rate, impossibility theorem (fairness), proxy variable

Tool: Fairness Lab — See the impossibility theorem with your own eyes: toggle fairness criteria over a synthetic applicant pool and watch satisfying one break the others.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.