Domain IV: Manage AI Model Development and Evaluation — Study Game
How to Play
Pick a game mode and test yourself. Cover answers and try to recall before peeking.
The Two Gates (IV.5 + IV.6)
Card 1 — Front: What's the IV.5 gate question?
Answer: Is the prepared data quality sufficient to train on? End of Phase III.
Card 2 — Front: What's the IV.6 gate question?
Answer: Is the model ready to operate in production? End of Phase V.
Card 3 — Front: Distinguish III.8 from IV.5.
Answer: III.8 = end of Phase II ("do we have what we need?"). IV.5 = end of Phase III ("is prepared data sufficient to train?"). Different artifacts, different boundaries.
Card 3b — Front: When are you ready to move past Phase III (the IV.5 gate)?
Answer: All data preparation issues addressed and the plan executed — Phase III is about actually doing the prep. Workbook steps complete with a well-defined answer, set of steps, and execution. No roadblocks. "You are not ready to build your model until the data is exactly how you need it, in the right shape, and in the right format."
Card 3c — Front: What does PMI ask to decide model readiness at the IV.6 gate?
Answer: Have you developed a model that aligns with the goals and objectives you set out to achieve? Does it perform well on the data that was carefully selected and prepared? Have you completed all the necessary steps of model evaluation — testing, validation, metrics analysis? And do you have adequate responses to the Phase V workbook questions?
Card 4 — Front: Outcomes at the IV.5 and IV.6 gates?
Answer: GO or NO-GO. NO-GO means pause and iterate back to the phase that owns the root cause — for IV.6, PMI names Phase I, II, III, or IV. Do not progress with roadblocks or warning signs.
Card 5 — Front: What does IV.6 GO authorize?
Answer: Domain V (deployment) work to begin.
Technique Selection (IV.1)
Card 6 — Front: What's the PM's role in IV.1?
Answer: Oversee — ensure technique is documented, justified against AI pattern + success criteria, aligned with operational constraints. PM doesn't pick the technique.
Card 7 — Front: Three ML categories?
Answer: Supervised, Unsupervised, Reinforcement.
Card 8 — Front: Difference between algorithm and model?
Answer: Algorithm = procedure. Model = trained artifact. You train an algorithm to produce a model.
Card 9 — Front: Three patterns of pretrained AI?
Answer: Pretrained model (adapt for task), Foundation model (very large pretrained), GenAI (generates new content).
Card 10 — Front: What's transfer learning?
Answer: Pretrained + fine-tune on your task data.
Card 11 — Front: What's RAG?
Answer: Retrieval-Augmented Generation — retrieve relevant context + generate from foundation model.
Training (IV.3)
Card 11b — Front: Four root-cause areas to review when training overruns?
Answer: Data, Technique, Hardware, Results — review when training overruns.
Card 12 — Front: Overfit vs underfit?
Answer: Overfit = memorizes training data, fails on new. Underfit = doesn't learn even on training.
Card 13 — Front: Typical train/validation/test split?
Answer: ~70%/15%/15%.
Card 14 — Front: What does generalization mean?
Answer: Model performs well on data it hasn't seen — the goal of training.
Data Preparation (IV.4)
Card 14b — Front: Four core data preparation activities (IV.4)?
Answer: Transform formats, Reconcile inconsistencies, Impute missing values, Map fields.
Card 15 — Front: How much of an AI project is data engineering?
Answer: About 80% — data identification and data aggregation. Over 80% of project time goes to data wrangling. Model training, tuning, and implementation are the small remainder.
QA/QC (IV.2)
Card 16 — Front: What does IV.2 QA/QC cover?
Answer: Configuration management + performance verification + bias measurement + documentation throughout development.
Card 17 — Front: Three transparency dimensions?
Answer: Systemic (how built), Decision (why this prediction), Algorithmic (algorithm-level).
Card 18 — Front: XAI vs Interpretability?
Answer: XAI = post-hoc explain any model. Interpretability = inherently understandable models. High-stakes prefers interpretability.
GAME MODE 2: Scenario Showdown — What Should the PM Do?
Scenario 1: The Training Overrun
- Training planned for 2 days
- Already running 5 days
- Data scientist says "one more day should do it"
Reveal
Pause training. Conduct (Data/Technique/Hardware/Results) root-cause review. Document decision: continue, change approach, or escalate. 2.5x overrun = project event, not technical hiccup.
Scenario 2: The Black-Box Healthcare Decision
- Data scientist proposes deep learning for high-stakes medical-imaging classification
- Healthcare client requires AI decisions be explainable
Reveal
Document technique selection; ensure trade-off between performance and explainability is presented to stakeholders for decision; consider interpretable-by-design alternatives. IV.1 + Domain I.2 cross-pull.
Scenario 3: The Operational Mismatch
- IV.6 review: model performance meets success criteria
- Chosen technique requires GPU compute
- Production environment is CPU-only
Reveal
NO-GO — a model that can't run in its destination isn't ready to operationalize. Inference efficiency and operational efficiency are part of Phase V evaluation. Loop back to V.1 (infrastructure) or IV.1 (technique change) with stakeholder decision.
Scenario 4: The Bias Discovery During QA
- QA/QC reveals demographic bias in recommendation model
- Data scientist suggests fairness post-processing layer
Reveal
Treat as IV.2 + I.3 issue: document, escalate per accountability, engage stakeholders for remediation, do not authorize IV.6 GO until bias within tolerance.
Scenario 5: The Parallel Work Request
- Phase IV complete
- Data scientist asks to begin Domain V work in parallel with IV.6 gate
Reveal
Confirm IV.6 must complete before Domain V work begins — sequential, not parallel. Gate authorizes the transition.
Scenario 6: The Plateau Concern
- Training completed; loss curve plateaued early
- Data scientist reports "acceptable" final accuracy
Reveal
Coordinate investigation of early plateau (data quality, technique fit, hyperparameter tuning). Engage IV.2 QA/QC. Don't blindly accept "acceptable" without root-cause.
Scenario 7: The Iteration Trap
- 4 training runs over 2 weeks
- Each improving slightly but not meeting criteria
- Data scientist suggests 5th iteration
Reveal
Pause and review iteration trajectory: are improvements converging or plateauing? Is the technique a fit? Is data sufficient? Document decision: continue, change technique, rescope, or escalate.
Scenario 8: The Quality Gate with Bias
- IV.5: data quality measures met
- Bias measurement reveals demographic disparity
- Data scientist suggests proceeding and addressing at training time
Reveal
NO-GO — an unresolved bias finding is a roadblock, and the data isn't "exactly how you need it." Loop back to IV.4 to remediate or III.1 to redefine. Trustworthy AI (I.3) cross-pull.
GAME MODE 3: Pattern Match Challenge
| # | Scenario | ECO Task |
|---|---|---|
| 1 | Overseeing model technique selection | IV.1 |
| 2 | Overseeing model QA/QC | IV.2 |
| 3 | Managing model training execution | IV.3 |
| 4 | Managing data transformation | IV.4 |
| 5 | Verifying data quality (gate) | IV.5 |
| 6 | Verifying model ready for ops (gate) | IV.6 |
| 7 | Confirming the model can run where it's going before authorizing deployment | IV.6 |
| 8 | Confirming the prepared data is in the right shape and format before training | IV.5 |
| 9 | Root-cause triage when training overruns the planned window | IV.3 |
| 10 | Coordinating the categories of data prep work (transform, reconcile, impute, map) | IV.4 |
Scoring: 9-10 = Expert | 7-8 = Solid | Below 7 = Review
GAME MODE 4: Fill-in-the-Blank Speed Round
- At IV.5 you are not ready to build your model until the data is exactly how you need it, in the right ________ and the right ________.
- At IV.6, PMI asks whether the model aligns with the ________ and objectives you set out to achieve.
- III.8 = "do we have what we need?" IV.5 = "is the ________ data sufficient to train?"
- AutoML automates the technical decision but doesn't replace ________ documentation (IV.1).
- The PM doesn't pick the technique — the ________ does.
- Training overrun by 2.5x = project event. Apply review: Data, Technique, ________, Results.
- About ________% of an AI project is data engineering work.
- IV.6 GO authorizes ________ ________ work to begin.
- RAG = ________-Augmented Generation.
- Reproducibility means same data + same pipeline = ________ output.
Reveal answers
- shape / format
- goals
- prepared
- governance
- data scientist
- Hardware
- 80
- Domain V
- Retrieval
- same
GAME MODE 5: True or False Lightning Round
| # | Statement | Correct |
|---|---|---|
| 1 | III.8 and IV.5 are the same gate | FALSE — different gates at adjacent boundaries |
| 2 | The PM picks the model technique | FALSE — data scientist picks; PM oversees governance |
| 3 | At IV.6 you can proceed as long as the model meets its requirements, even if key Phase V workbook questions are still unanswered | FALSE — PMI requires adequate workbook responses too; do not progress with no answers, roadblocks, or warning signs |
| 4 | Domain V work can begin in parallel with IV.6 gate | FALSE — sequential |
| 5 | AutoML bypasses IV.1 documentation requirement | FALSE — automation ≠ governance |
| 6 | Whether the model can actually run in its destination is part of Phase V evaluation | TRUE — model size, inference efficiency, operational efficiency, and edge-device demands are all evaluated |
| 7 | Production validation substitutes for IV.6 evaluation | FALSE — gate is pre-deployment |
| 8 | Reproducibility means inference reproducibility only | FALSE — training reproducibility too |
| 9 | The PM declares IV.6 GO unilaterally | FALSE — multi-stakeholder sign-off |
| 10 | A failed contingency test still satisfies V.7 if documented | FALSE — V.7 requires tested plans |
| 11 | A model that hits its technical metrics but misses the business KPIs still passes evaluation | FALSE — iterate, refine, or start over so it aligns with the business objectives |
| 12 | "Reflecting real-world differences" excuses bias | FALSE — amplification/perpetuation matter |
Scoring: 11-12 = Exam ready | 9-10 = Almost | <9 = Review
Scoring Summary
| Mode | Score | Max |
|---|---|---|
| Flashcards | ___/22 | 22 |
| Scenarios | ___/8 | 8 |
| Pattern Match | ___/10 | 10 |
| Fill-in | ___/10 | 10 |
| True/False | ___/12 | 12 |
| TOTAL | ___/62 | 62 |
Rating: 53+ = mastered · 40-52 = strong · 27-39 = review · <27 = re-study.