Fairness evaluation and policy alignment

Lesson 3 of 3 in Governance and Responsible AI.

Fairness for FM outputs starts by making it measurable: pre-defined metrics per cohort — refusal-rate parity, outcome parity, tone/sentiment scores — published to CloudWatch where they can trend and alarm. Then it needs machinery to improve the numbers, and the exam names the trio. Prompt Management holds candidate prompts as immutable, diffable versions; Prompt Flows (Amazon Bedrock Flows) wires variants into testable pipelines, so an A/B is two flow configurations over the same evaluation traffic — systematic, not two engineers eyeballing outputs. LLM-as-a-judge makes scoring scale: a judge model evaluates thousands of sampled outputs against your fairness rubric and explains each score — Bedrock evaluations runs these as managed jobs. Judges have biases too; calibrate against a human-labeled slice before trusting the trend (evaluation depth).

The AIP-C01 exam guide then closes the loop from measuring fairness to enforcing policy: guardrails derived from policy requirements (the written rules become denied topics, filters, and an Automated Reasoning policy compiled from the policy document itself), model cards documenting limitations (the honest boundary of where the system should be used), and Lambda automated compliance checks — pipeline gates that run the evaluation suite and adversarial regression before any variant promotes. Organizational governance is the frame around all of it: one framework mapping each written policy line to a named control, owned and audited.

The fairness A/B loop — systematic, scored, gated

  1. Prompt v12 vs v13

    Candidate prompts as immutable Prompt Management versions — diffable, attributable, rollback-able.

  2. Flows route the A/B

    Two flow configurations over the same sampled evaluation traffic — the systematic A/B the guide asks for.

  3. LLM-as-a-judge scores

    A judge model applies the fairness rubric per cohort — refusal parity, tone, outcome — explaining each score.

  4. Metrics to CloudWatch

    Per-variant, per-cohort fairness metrics land as named metrics with alarms.

  5. Parity within threshold?

    A Lambda compliance check reads the numbers — the automated compliance gate.

  6. Promote the variant

    The winning version ships; the card and changelog record why. Monitoring continues — bias drift re-opens the loop.

  7. Revise prompt / guardrail

    Failed parity goes back to authoring — a new version, never an edit in place.

Interactive sorting exercise: Each governance requirement has one named mechanism. Place it.

Interactive checkpoint quiz (3 questions) — open this page in a browser to take it.