GenAI data integrity: prompt–response pairs and safety screens

Lesson 5 of 5 in Data Quality, Labeling, and Bias.

Fine-tuning corpora are training data, and MLA-C02 says so out loud: prompt–response pair validation and content safety screening are named under validating data quality, right beside outlier detection and deduplication. The failure modes are new costumes on old problems — a malformed JSONL line is a missing value, a repeated Q&A pair is a duplicate, an eval question hiding in the training set is leakage wearing a lanyard.

Format & schema

Every line parses; every record carries the required fields (prompt and completion, correct keys for the target service); lengths fit the model's context. Mechanical, scriptable, skipped surprisingly often.

Pair quality

Does the completion actually answer the prompt? Spot-check a sample by hand; filter empty, truncated, or "as an AI model" responses; watch the length distribution for degenerate pairs.

Duplicates & near-duplicates

Exact and fuzzy dedupe. Repeated pairs overweight their pattern and, if they straddle train/eval, inflate every metric you report.

Eval-set contamination

Remove training examples that overlap your evaluation set (exact or near-match). A contaminated eval measures memory, not capability — leakage, GenAI edition.

Safety screen

Run the corpus through PII detection and toxicity screening (Amazon Comprehend offers both for text), filter or rewrite flagged examples, and keep provenance/licensing records for what remains.

The pair gauntlet

  1. Raw prompt–response pairs
  2. Format & schema check

    Every JSONL line parses; prompt and completion are present under the right keys; lengths fit the model's context.

  3. Dedup & decontaminate

    Exact and fuzzy dedupe, then remove overlap with the evaluation set — a contaminated eval measures memory, not capability.

  4. Safety screen

    Comprehend PII and toxicity detection over the text; filter or rewrite flagged pairs and keep provenance records.

  5. Accepted fine-tuning set

    Validated, unique, decontaminated, screened — ready to train on.

  6. Rejected pairs

    Logged with the reason each pair fell out — an audit trail and a repair queue.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.