AIP question anatomy and the professional traps
Lesson 3 of 4 in Troubleshooting GenAI — and Exam Readiness.
The machine itself: 75 questions — 65 scored, 10 unlabeled unscored pilots — in 180 minutes, as multiple choice (one of four) and multiple response (two or more of five or more, graded as an exact set, no partial credit). Results land on a scaled score of 100–1,000 with the pass line at 750 — the Professional-tier line, above the associate 720 — and scoring is compensatory: one overall score, no per-domain minimums. Unanswered scores as wrong; guessing is free.
What "Professional" changes is not the format but the options. An associate item usually has one defensible answer and three that collapse on contact. An AIP item routinely offers two or three architecturally valid solutions — all real, all buildable — and the key is the one that satisfies a single qualifier or constraint buried in the stem: with per-score explanations, without re-indexing, while keeping responses synchronous. Read the requirement sentence first, mark the qualifier, and make the options come to it.
Specimen A — multiple choice
A company compares two foundation models for a legal-summarization feature. The evaluation must produce a quality score with a written explanation for every scored response, run on thousands of responses per week, and must not require assembling a human workforce. Which solution meets these requirements?
A. Run Amazon Bedrock programmatic evaluation jobs using BERTScore and F1 against reference summaries B. Run Amazon Bedrock evaluation jobs that use a judge model (LLM-as-a-judge) over the prompt dataset C. Run Amazon Bedrock human-based evaluation jobs with an AWS-managed work team D. Enable Amazon CloudWatch generative AI observability and review invocation metrics weekly
The dissection
- Three of the four are real evaluation approaches — A, B, and C all exist as managed Bedrock job types. At Professional level, "it exists and works" eliminates nothing. That is the design.
- The qualifiers decide: explanation for every scored response kills A (programmatic metrics emit numbers, not reasons). No human workforce kills C by its own definition — managed or not, a work team is a workforce. Thousands per week confirms the automated path.
- B is the only option satisfying all three stated requirements: judge jobs score each response against rubric metrics and write a natural-language explanation per score, at pipeline scale.
- D is the out-of-domain decoy — observability reports usage and traces, not response quality. One option in most sets is doing a different job than the stem asks; name the job each option does and that option self-deletes.
The protocol, compressed: requirement sentence → qualifiers underlined → each option labeled with the job it actually does → survivors judged only on the qualifiers.
Specimen B — Select TWO
A production assistant on an on-demand Amazon Bedrock model intermittently returns ThrottlingException (HTTP 429) errors during business-hours peaks. The team must eliminate the user-facing failures while keeping responses synchronous. Select TWO actions that together meet the requirements.
A. Implement exponential backoff with jitter in the client's retry logic B. Purchase provisioned throughput sized for the business-hours baseline C. Move the workload to Amazon Bedrock batch inference D. Increase the model's maximum output token parameter E. Provision AWS Direct Connect between the office and AWS
The dissection
- A + B cooperate — the Select-TWO signature. Provisioned throughput absorbs the known peak; backoff-with-jitter absorbs the residual bursts above it. Each alone leaves a gap; together they meet "eliminate the failures".
- C solves throttling by abandoning the second requirement: batch is asynchronous by design. Right problem, killed by the constraint — the Professional move is noticing which requirement each option sacrifices.
- D is a plausible-sounding knob with no mechanism: output length does not change request or token rates. If you cannot say why an option fixes the stated failure, it does not.
- E is the out-of-scope decoy: Direct Connect is private network connectivity — it is on the AIP exam guide's out-of-scope services list, and bandwidth was never the bottleneck. Options reaching for out-of-scope services to solve GenAI problems are near-certain distractors.
- Grading: exact set. A alone, B alone, or A+C — all score zero. Complete the set with your best second pick; a blank guarantees the zero a guess merely risks.
| Trap | How it looks | The counter |
|---|---|---|
1 · The one-qualifier split | Two or three options are genuinely valid architectures; they differ on a single stem phrase — "with explanations per score", "without re-indexing", "while keeping responses synchronous" | Underline the qualifier before reading options; judge survivors on it alone |
2 · The out-of-scope decoy | An option hinges on adopting a service the exam guide lists as out of scope — Amazon Redshift, Amazon CodeGuru, AWS Savings Plans, AWS Direct Connect are the classic four (also Forecast, Fraud Detector, Trusted Advisor, Elastic Beanstalk) | Know the out-of-scope list as a distractor signal: these services may dress a scenario, but an answer that depends on one is almost never the key |
3 · The associate reflex | The answer that was right at associate depth: "add RAG to reduce hallucination", "enable Guardrails", "use a bigger model" — correct category, no diagnosis | Professional stems ask which failure and which lever: name the failing half (retrieval vs generation), the specific metric, the specific remediation |
4 · The needless fine-tune | "Fine-tune the model" offered where retrieval or prompting meets the need — fresh knowledge, company documents, format fixes, truncation | Fine-tuning buys style, format discipline, and task specialization; it does not add fresh or private knowledge (RAG does), and it never fixes an input the model did not receive |
Interactive sorting exercise: Each phrase is an answer option from an AIP-style item. Sort it onto the trap it embodies.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.