Bias: how discrimination gets automated

Lesson 3 of 6 in Why AI Needs Governance: A Taxonomy of Harms and Risks.

Bias is the most litigated, most regulated, and most misunderstood AI harm — misunderstood because people picture a bigoted programmer, when the mechanism is almost always structural. Bias enters through at least six doors, and none requires anyone to intend it:

  1. Historical data — the world’s past prejudice, recorded as ground truth. Amazon’s recruiting model learned from ten years of male-dominated hiring and penalised the word “women’s”.
  2. Labels — subjective human judgments frozen into targets. “Was this employee successful?” encodes whoever defined success.
  3. Proxies — neutral-looking features that stand in for protected ones. The Obermeyer study (Science, 2019) found a healthcare algorithm used cost as a proxy for need — and because less money had historically been spent on Black patients, equally sick Black patients scored as healthier. Fixing the label roughly tripled their eligibility for extra care.
  4. Feature choice — what the designers thought worth measuring, and what they left out.
  5. Feedback loops — the model’s decisions generating the data that retrains it (walk through the diagram below).
  6. Deployment context — a model accurate on one population applied to another.

The bias feedback loop — predictive policing edition

  1. Historically over-policed neighbourhoods

    Decades of enforcement concentrated in certain areas produce arrest records that reflect policing patterns, not just crime patterns.

  2. Arrest data becomes training data

    The records go in as ground truth. The dataset cannot distinguish “more crime” from “more police looking”.

  3. Model predicts high risk in the same areas

    The model faithfully learns the geographic pattern in its data.

  4. More patrols sent to predicted areas

    Resource-allocation decisions follow the scores.

  5. More arrests recorded there

    More officers looking means more recorded incidents — regardless of the underlying crime rate.

  6. Model retrains on the new data

    The loop closes: the model’s own effects now “confirm” its predictions. Intervention points: audit input data for enforcement bias, cap the model’s influence on allocation, or measure against victimisation surveys instead of arrests.

Anti-discrimination law gives you the second essential distinction. Disparate treatment means using the protected trait itself — the toeslagenaffaire feeding nationality into risk scores. Disparate impact means a facially neutral practice that lands unequally — no protected trait in the features, but the outcomes skew anyway, usually via proxies. Most AI bias is disparate impact, which is why “we removed race from the model” — fairness through unawareness — is the field’s most persistent fallacy. The model reconstructs the removed variable from postcode, shopping patterns, and a hundred other correlates.

Two cases anchor everything else you will read:

COMPAS (2016)

ProPublica analysed COMPAS, a recidivism-score tool used in US bail and sentencing decisions, and found Black defendants who did not reoffend were nearly twice as likely as white defendants to have been labeled high-risk — while white reoffenders were more often labeled low-risk. The vendor countered that the score was equally calibrated for both groups. Both claims were mathematically true: they used different fairness definitions, and (as you will learn in the ethics module) those definitions are provably impossible to satisfy simultaneously when base rates differ. COMPAS made algorithmic accountability a public issue and put a permanent question over AI in criminal justice.

Gender Shades (2018)

Joy Buolamwini and Timnit Gebru tested three commercial gender-classification systems and disaggregated the results: error rates below 1% for lighter-skinned men, up to 34.7% for darker-skinned women. The cause was representational — training and benchmark datasets dominated by lighter-skinned male faces. The study’s method mattered as much as its finding: aggregate accuracy hides subgroup failure, so testing must be disaggregated. That principle now sits inside every serious evaluation regime, and the study helped push IBM, Microsoft, and Amazon to pause or exit facial-recognition sales to police by 2020.

Key terms: disparate impact, proxy variable, allocative harm, representational harm, COMPAS, Gender Shades

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.