Pillar tour I: GENOPS and GENSEC

GENOPS01 — how do you achieve and verify consistent model output quality?

The canonical best practice of the whole lens is GENOPS01-BP01: periodically evaluate functional performance — flagged High risk if unmet, and effectively the engineering restatement of every accuracy obligation you have met in law.

The mechanics matter. Build a ground-truth dataset of prompts with known-good responses, covering your real user personas and use cases. Apply stratified sampling — divide the ground truth into relevant categories and sample from each, so the evaluation set reflects the diversity of production traffic instead of over-weighting the easy majority case (the same disaggregation logic that bias audits use, aimed at quality). Run the model against it on a schedule, score the outputs with custom metrics tailored to your business — the open-source fmeval library gives you a framework for defining them, ragas covers RAG-specific ones, and Amazon Bedrock’s built-in model evaluation handles common cases — and track the scores over time to catch drift.

Two governance details hide in the guidance: your organisation’s AI policy should define minimum performance levels (the threshold is a policy decision, not an engineer’s preference), and evaluations re-run whenever a new candidate model appears or customisation is applied — a fine-tuned model faces the same evaluation bar as the original. GENOPS01-BP02 adds the other half: collect and monitor user feedback, because ground truth goes stale and users notice first.

GENOPS02 — how do you monitor and manage operational health?

Three best practices: monitor all application layers (a GenAI app is a stack — retrieval, orchestration, model, guardrails — and a latency spike can live in any of them), monitor foundation-model metrics (invocation counts, token throughput, error and throttle rates — the vocabulary CloudWatch already emits for Bedrock), and mitigate the risk of system overload with rate limits and back-pressure before a traffic spike becomes an outage.

The lens’s framing is worth quoting in spirit: foundation models interact with software and data differently from traditional systems, so classic monitoring practice applies plus a model-shaped layer on top. If you built the monitoring lesson’s dashboards, GENOPS02 is largely a checklist of what you already run.

GENOPS05 — when do you customise the model?

The lens prescribes an escalation ladder: start with prompt engineering, then retrieval-augmented generation, then fine-tuning, and only then custom model building — each rung more capable, more expensive, and heavier to operate and re-evaluate than the last. The best practice is knowing when a rung is justified: task specificity, data availability, and resource constraints drive the decision, not enthusiasm.

Governance reads this as a proportionality principle for capability: every step up the ladder adds evaluation debt (GENOPS01 applies in full to the customised model), security surface (GENSEC06’s poisoning risks arrive with training pipelines), and cost. The cheapest model change is the one you did not make.

GENSEC01 — how do you manage access to generative AI endpoints?

Foundation models are reached through managed, serverless, or self-hosted endpoints, and each paradigm has its own exposure. The best practices are classic discipline aimed at new targets: least privilege access to the model endpoints themselves, private network communication between applications and models (no model traffic over the public internet), least-privilege permissions for what the model can read — the data stores behind your RAG pipeline — and access monitoring over all of it.

That third practice is the quietly radical one: the model is a principal now. If the retrieval role can read the entire document lake, then every user of the chatbot effectively can too, one well-crafted question at a time.

GENSEC02 — how do you stop harmful, biased, or incorrect responses?

One best practice carries the question: implement guardrails to mitigate harmful or factually incorrect model responses. You already know the machinery in vendor-specific depth — content filters, denied topics, contextual grounding checks, the ApplyGuardrail API — from the cloud-guardrails module, so the lens adds the review-shaped question on top: are they implemented, on every path to every model, with thresholds someone can defend?

In review terms this is where your guardrail configuration stops being an engineering artifact and becomes an answer: the reviewer ticks the BP only if the guardrails demonstrably run in production, which makes the intervention metrics from your gateway the natural evidence.

GENSEC04 — how do you secure system and user prompts?

Two best practices. First, a secure prompt catalog: system prompts are operational assets — engineered, tested, versioned — and they deserve the same protection as code, because an attacker who can read your system prompt learns your defenses and one who can write it owns your application. Second, sanitise and validate user inputs before they reach the model — the lens’s answer to direct prompt injection, and the input half of the guardrail architecture you built in the previous module.

GENSEC05 — how do you avoid excessive agency?

Excessive agency — straight from the OWASP Top 10 for LLM applications — is the risk that an agent, empowered to act on a user’s behalf, takes actions beyond its intended purpose. The lens is precise about why: the agent has little knowledge beyond its prompt of what is permitted, so the boundary must live outside the model.

GENSEC05-BP01 makes the boundary concrete: agents in Amazon Bedrock run under execution roles (Flows under service roles), and those roles get least-privilege policies scoped to named resources; permission boundaries cap the maximum permissions a role can ever hold; separation of duties splits the prompt engineer who builds the workflow from the security engineer who writes its role; and user confirmation gates consequential actions. Read that list again with your governance hat on: it is the autonomy-spectrum lesson from module one, compiled into IAM. The agent’s authority is decided in a policy document a human reviews — not negotiated with the model at runtime.