Predictive policing, emotion recognition, and the border

Lesson 2 of 5 in Biometric Surveillance, Law Enforcement AI, and Protecting Children.

Predictive policing comes in two flavours, and the law now treats them very differently.

Place-based systems (PredPol/Geolitica, HunchLab) forecast where crime is likely and send patrols there. The governance problem is the feedback loop: police patrol where the model points, find crime because they are looking, and the arrests feed back as training data confirming the prediction. Historically over-policed neighbourhoods get mathematically re-certified for more policing. Independent analyses found Geolitica’s predictions were both largely inaccurate and largely ignored by officers — Santa Cruz, its birthplace, became the first US city to ban predictive policing in 2020, and the vendor wound down in 2023.

Person-based systems score individuals. Chicago’s “Strategic Subject List” ranked roughly 400,000 people by supposed involvement risk; an inspector-general review found no evidence it reduced violence, and it was shut down in 2019. The Dutch “Top 600”/“Top 400” youth lists and the toeslagenaffaire risk-scoring you met in Foundations belong to the same family.

The EU AI Act drew the line at the person: Article 5(1)(d) prohibits risk assessments predicting whether a natural person will commit a crime based solely on profiling or on assessing personality traits and characteristics. Systems supporting a human assessment grounded in objective, verifiable facts directly linked to criminal activity stay outside the prohibition — and place-based or person-based tools used by law enforcement generally land in Annex III high-risk territory instead.

Emotion recognition claims to infer inner states from faces, voices, or keystrokes. The scientific foundation is contested — the largest review of the field (Barrett et al., 2019) concluded facial movements do not reliably map to emotions across people and cultures. The AI Act responded with a split verdict: Article 5(1)(f) prohibits emotion inference in workplaces and education institutions (except for medical or safety reasons), while emotion recognition used by law enforcement or at borders is high-risk, not banned — one of the compromises fought over hardest in trilogue.

The border is where the strictest-sounding rules meet the weakest oversight. Annex III point 7 classifies as high-risk: polygraph-style tools (the EU-funded iBorderCtrl lie-detection pilot became the cautionary tale), risk assessments of people entering, systems assisting examination of asylum, visa, and residence applications, and identification of persons in migration contexts. Critically, migrants and asylum seekers rarely have practical access to the remedies the Act assumes — which is why civil-society scrutiny (AccessNow, EDRi, the #ProtectNotSurveil coalition) concentrates here. In the US, DHS and CBP deploy facial comparison at airports and the CBP One app for asylum appointments with no equivalent statutory framework — oversight runs through DHS AI directives, the sectoral posture you saw in the US domain.

Interactive sorting exercise: Sort each law-enforcement or border practice into its EU AI Act treatment.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.