Domain IV: Manage AI Model Development and Evaluation — Comprehensive Study Guide

Exam weight: 16% of PMI-CPMAI exam (~16 scored questions, of ~19 total)

Score-report framing: ❌ Below Target — PRIORITY 3 for rebuild

Maps to CPMAI methodology phases: Phase III (Data Preparation), Phase IV (Model Development), Phase V (Model Evaluation)

Number of ECO tasks: 6 (IV.1 through IV.6) — 2 of which are go/no-go gates (IV.5 + IV.6)

Estimated study time: 13 hours

Note from docs/ECO_TASK_REFERENCE.md: the score report flagged Task IV.2 (Oversee AI/ML model QA/QC) as having no questions on his form. Cover it anyway — the retake form is randomized.

>

Two of six tasks are explicit go/no-go gates — that's 33% of the domain by task count. The ECO names three go/no-go tasks in total (III.8, IV.5, IV.6) and publishes weights by domain only, not by topic — but two of this domain's six tasks being gates makes the gate decision pattern the highest-leverage thing to master here.

Overview

Domain IV is the most procedurally complex of the three weak domains. It spans three CPMAI methodology phases (Data Preparation, Model Development, Model Evaluation) and contains two of the three explicit go/no-go gates in the entire ECO. Every task begins with an oversight verb: oversee, manage, verify. The PM is responsible for ensuring that data preparation produces sufficient quality, model technique selection is sound, training is managed, QA/QC standards are upheld, and the model is verified ready before it crosses into Operationalization (Domain V).

The unifying pattern: Domain IV tests whether the project manager can hold the gate. The data scientist wants to keep iterating. The ML engineer wants to keep tuning. The business stakeholder wants to ship. The PM is the one who facilitates the documented decision against documented criteria — and who is willing to call NO-GO and iterate back when the criteria aren't met.

Most wrong-answer traps in Domain IV are technically-correct moves that bypass either a gate, an iteration trigger, or a stakeholder decision. The same oversight-verb framing from Domain III applies — and it applies more sharply because Domain IV's two gates have very specific decision criteria.

Table of Contents

• Module 1: Data Preparation — Pipelines, Quality, and the Prep Gate (Lessons 1-7)
• Module 2: Model Technique and Selection (Lessons 8-13)
• Module 3: Model Development and Training (Lessons 14-20)
• Module 4: Model QA/QC, Evaluation, and Iteration (Lessons 21-31)
• Module 5: The Operationalization Gate and Phase V Closeout (Lessons 32-36)
• Quick Reference: The Two Gates (IV.5 + IV.6) Cheat
• Quick Reference: Model Evaluation Checklist
• Cross-Domain Links
• Knowledge Check

Module 1: Data Preparation — Pipelines, Quality, and the Prep Gate

Lessons 1-7 | What data preparation requires, and the gate that decides whether to begin it.

☐ Lesson 1: ECO Task IV.4 — Manage Data Transformation to Conduct Data Preparation

After the data is gathered (III.5) and the Phase II gate (III.8) has decided GO, the team enters Phase III — Data Preparation. The PM's job is to manage the transformation effort: ensure pipelines are built, transformations are documented, quality is preserved, and the prep work feeds the success criteria from Domain II.

The PM does not write transformation code. The PM coordinates the data engineering team, tracks pipeline development, and ensures the prepared dataset is usable before model training begins.

KEY TAKEAWAYS

• Data transformation = Phase III work. Begins after III.8 GO.
• PM manages transformation; data engineers execute it.

💡 Memory Aid — Data Prep

Transform formats, Reconcile inconsistencies, Impute missing values, Map fields. Four core categories of data prep work the PM coordinates.

PM Oversight Angle

• PM owns: Coordinating data preparation execution; tracking pipeline development; ensuring transformations are documented; verifying prepared dataset is usable for training.
• Deliverable: Data Preparation Plan + status tracker; documented pipelines; prepared dataset ready for IV.5 verification.
• Iteration trigger: Transformation reveals data issues that III.7/III.8 missed → loop back to III.7 (re-evaluate) or III.1 (re-define requirements).
• Escalation trigger: Transformation cost or timeline exceeds project tolerance; data quality remediation requires new data sourcing.
• Wrong-answer trap: "Have the data scientist start training on the partially-prepared data while the engineer finishes." Training before prep is complete inserts data quality issues into the model.
• Question pattern signal: Stems mentioning "the team is preparing data," "data transformation is in progress," "the data engineer is building pipelines."
• ECO task tag: Domain IV, Task 4 — Manage data transformation to conduct data preparation

☐ Lesson 2: Data Preparation Concepts

Data preparation is the work of making raw data usable for training. It includes:

• Cleaning — removing errors, duplicates, outliers, inconsistencies.
• Transforming — changing format, structure, or representation (e.g., normalization, encoding, aggregation).
• Imputing — filling missing values (or flagging and excluding).
• Augmenting — generating additional training examples (rotation/cropping for images, paraphrasing for text).
• Splitting — dividing into training, validation, and test sets.
• Labeling — for supervised learning (often expensive and time-consuming).

PMI puts the bulk of an AI project — about 80% — in data engineering work such as data identification and data aggregation, not in model training, tuning, and implementation. Over 80% of project time is spent on data wrangling. Underestimating this is a top reason projects miss deadlines.

KEY TAKEAWAYS

• Data prep includes clean, transform, impute, augment, split, label.
• About 80% of an AI project is data engineering — model training, tuning, and implementation are the small remainder.

☐ Lesson 3: Data Engineering and Pipelines

Data engineering builds the pipelines that move data from sources through preparation and into the model's training environment. Key pipeline concepts:

• ETL (Extract, Transform, Load) — extract from source, transform to target schema, load to destination.
• ELT (Extract, Load, Transform) — load raw, transform in destination (modern cloud pattern).
• Streaming pipelines — continuous data flow vs scheduled batch.
• Data lakes vs data warehouses — lakes hold raw data flexibly; warehouses hold structured, cleaned data.

The PM coordinates pipeline ownership and ensures the deployment plan (V.1) accounts for pipeline maintenance in production.

KEY TAKEAWAYS

• ETL = transform before load. ELT = transform after load.
• Lakes = raw flexibility. Warehouses = structured precision.
• Pipeline ownership = PM coordination concern.

☐ Lesson 4: Data Collection and Ingestion

Collection brings data into the pipeline; ingestion is the technical implementation. Both happen in Phase III but are heavily informed by Phase II decisions (III.3 sources, III.5 gathered data). Common ingestion patterns: API pulls, file drops, database replication, streaming connectors, batch uploads.

PM concerns: ingestion reliability, error handling, data validation at entry, source-side rate limits or licensing.

KEY TAKEAWAYS

• Collection + ingestion = bringing data into the pipeline.
• Ingestion is informed by Phase II source decisions.

☐ Lesson 5: Data Preparation Pipelines

Phase III's signature deliverable is a data preparation pipeline that:

1. Ingests from identified sources
2. Validates input format and content
3. Cleans (remove errors/duplicates)
4. Transforms (format/schema)
5. Imputes missing values
6. Augments if needed
7. Splits into training/validation/test
8. Outputs to model training environment

The pipeline is reusable and reproducible — same input + same pipeline = same output. Reproducibility is a governance requirement (V.3).

KEY TAKEAWAYS

• Pipeline = reproducible end-to-end transformation of raw data into training-ready form.
• Reproducibility is a governance requirement.

☐ Lesson 6: Pipeline Complexity

Real-world pipelines are complex. PMI flags this as a project risk:

• Multiple data sources with different formats and refresh cadences.
• Dependencies between pipeline stages (one step's output is another's input).
• Failure handling — what happens when a source is unavailable or input is malformed?
• Versioning — pipelines themselves change as data and requirements evolve.
• Monitoring — pipelines need observability or failures go undetected.

The PM doesn't design the pipeline but tracks complexity as a project risk and ensures observability is built in.

KEY TAKEAWAYS

• Real pipelines are complex; complexity = risk.
• Failure handling, versioning, observability = required, not optional.

☐ Lesson 7: ECO Task IV.5 — Verify data quality for go/no-go decision to conduct data preparation

The first gate in Domain IV. After data preparation pipelines are built and run, the PM facilitates a verification gate: is the prepared data quality sufficient to proceed with model training?

This is distinct from III.8, which asked "do we have the data and understanding?" — a Phase II close-out gate. IV.5 asks "now that we've prepared the data, is the prepared output of sufficient quality to train on?" — a Phase III close-out gate.

PMI's readiness test for moving past Phase III:

• All data preparation issues have been addressed — and the plan was not just written but executed. Phase III is about actually doing the data preparation.
• Workbook steps complete and checked off — a well-defined answer, a well-defined set of steps, and a well-defined execution for the critical questions.
• No roadblocks. You know what you know and what you do not know, and the unknowns are not things you need to resolve before moving on.
• The data is exactly how you need it, in the right shape, and in the right format. Until then, you are not ready to build the model.

The decision is GO or NO-GO. On NO-GO you pause and iterate back to the phase that can actually fix the issue — PMI's own example: if you are unsure what data you need, go back to Phase II to figure that out. Iterating back carries no penalty in CPMAI; pushing forward with unresolved issues does.

KEY TAKEAWAYS

• IV.5 ≠ III.8. III.8 = Phase II close (data understanding). IV.5 = Phase III close (data prep complete).
• Two outcomes: GO / NO-GO. NO-GO = pause and iterate back to the phase the issue demands.
• Readiness test: prep issues addressed and executed, workbook steps complete, no roadblocks, data in the right shape and format.

💡 Memory Aid — Gate Criteria

Issues addressed and the plan executed, workbook steps checked off, no roadblocks, data exactly as needed in the right shape and format. The four things PMI says make you ready to leave Phase III.

PM Oversight Angle

• PM owns: Facilitating the documented data-quality verification with stakeholders. Compiling pipeline output evaluation into a gate decision package.
• Deliverable: Phase III Go/No-Go Decision — evidence that prep issues were addressed and executed, completed workbook steps, roadblock assessment, confirmation the data is in the required shape and format, decision (GO/NO-GO), stakeholders engaged.
• Iteration trigger: Prepared data is not in the shape the model needs, or a roadblock remains → NO-GO and iterate back to IV.4 (more transformation work), III.7 (re-evaluate raw data), or III.1 (re-define requirements).
• Escalation trigger: A NO-GO that requires Phase II or Phase I changes the team cannot make alone, or that materially changes project value.
• Wrong-answer trap: "Begin model training and improve data quality in parallel." Bypasses the gate. Quality issues compound into model defects.
• Question pattern signal: Stems mentioning "data preparation is complete," "the team is ready to train," "data quality is being verified," "the data engineer says the pipelines are done."
• ECO task tag: Domain IV, Task 5 — Verify data quality for go/no-go decision to conduct data preparation

Module 1 Summary

The one thing: Data preparation begins only after the III.8 GO, and it closes at its own gate — IV.5 is not a repeat of III.8.

PM's job in this module: Manage the transformation work and verify prepared-data quality against documented criteria before Phase IV opens.

Exam signals: Stems that put a data-quality question at a gate. Decide first which gate: III.8 closes data understanding, IV.5 closes data preparation.

Trap to watch: Answering a IV.5 scenario with III.8 reasoning, or letting prepared data through because the raw data already passed a gate.

ECO tasks: IV.4, IV.5

Module 2: Model Technique and Selection

Lessons 8-13 | What technique, algorithm, and model approach the project will use.

☐ Lesson 8: ECO Task IV.1 — Oversee AI/ML model technique(s) (e.g., algorithm, selection)

The PM oversees the team's selection of model technique(s) — the algorithmic approach (supervised/unsupervised/reinforcement learning), the model family (linear, tree-based, neural network, transformer), and any pretrained-model decisions. The PM doesn't pick the technique; the data scientist does. The PM ensures the choice is documented, tied to the AI pattern from Phase I, and aligned with the project's success criteria.

A common exam scenario: the data scientist proposes a complex deep-learning model. The right PM response is rarely "approve" — it's "ensure the choice is documented and justified against the AI pattern, success criteria, and operational constraints (cost, latency, explainability)."

KEY TAKEAWAYS

• Technique selection = data scientist's call, PM-overseen.
• Documentation + justification against pattern + criteria + operational constraints.

PM Oversight Angle

• PM owns: Overseeing technique selection; ensuring the choice is documented, justified, and aligned with Phase I AI pattern + success criteria.
• Deliverable: Model Technique Justification — section of the CPMAI workbook documenting algorithm/family/pretrained choices, rationale, alignment with pattern, operational implications.
• Iteration trigger: Selected technique reveals operational constraint mismatch (e.g., real-time latency required but technique can't deliver) → loop back to V.1 deployment plan or IV.1 re-selection.
• Escalation trigger: Technique requires resources, cost, or vendor relationships beyond project authority.
• Wrong-answer trap: "Approve the data scientist's choice and proceed." Approval without documentation is a governance gap.
• Question pattern signal: Stems mentioning "the data scientist proposes [model]," "the team is choosing between [techniques]," "an algorithm has been selected."
• ECO task tag: Domain IV, Task 1 — Oversee AI/ML model technique(s)

☐ Lesson 9: Machine Learning Fundamentals — Algorithm vs Model

Two terms commonly confused on the exam:

• Machine learning algorithm — the procedure for learning patterns from data (e.g., gradient descent, decision tree induction).
• Machine learning model — the result of running an algorithm on data (the trained artifact that makes predictions).

You train an algorithm on data to produce a model. The model is what gets deployed.

KEY TAKEAWAYS

• Algorithm = procedure. Model = trained artifact.
• You train an algorithm to produce a model.

☐ Lesson 10: ML Algorithm Basics

ML lets computers learn patterns from data and make predictions. Three high-level categories:

• Supervised learning — learn from labeled examples (input → known output).
• Unsupervised learning — find structure in unlabeled data (clustering, dimensionality reduction).
• Reinforcement learning — learn by trial-and-error in an environment with rewards.

The choice of category depends on the AI pattern and the data available:

• Recognition / Classification → typically supervised.
• Pattern discovery → unsupervised.
• Sequential decision-making → reinforcement.

KEY TAKEAWAYS

• 3 categories: supervised, unsupervised, reinforcement.
• Category choice depends on AI pattern + data availability.

☐ Lesson 11: Pretrained Models, Foundation Models, and GenAI

Modern AI rarely trains from scratch. Three patterns:

• Pretrained model — a model already trained on a generic task that you adapt for your specific use.
• Foundation model — a very large pretrained model (e.g., GPT-4, Claude, LLaMA) that can be specialized via prompting or fine-tuning.
• GenAI — generative AI that produces new content (text, images, audio, code) — typically built on foundation models.

Using pretrained / foundation / GenAI models reduces the data needed for training (Phase II decisions reflect this — see III.8 questions about "can you use pretrained models?").

KEY TAKEAWAYS

• 3 patterns: pretrained, foundation, GenAI.
• All reduce required training data — feedback into Phase II gate (III.8).

☐ Lesson 12: Transfer Learning and Third-Party Models

Transfer learning = taking a pretrained model and fine-tuning it on your task-specific data. Saves training time and works with less data than training from scratch.

Third-party models — sourced from vendors, open-source repositories, or model marketplaces. Brings a governance question: is the model's training data license-compatible? Is bias measurement available? Is provenance documented?

KEY TAKEAWAYS

• Transfer learning = pretrained + fine-tune on your task.
• Third-party models bring governance and provenance questions to V.3.

☐ Lesson 13: Automated Machine Learning (AutoML)

AutoML automates parts of model development — algorithm selection, hyperparameter tuning, feature engineering, model selection. Reduces the data-science skill barrier.

For the PM: AutoML doesn't remove the need for documented justification (IV.1). The output of AutoML is a chosen technique; it still needs to be documented, evaluated, and gated through IV.5/IV.6.

KEY TAKEAWAYS

• AutoML = automated technique selection, hyperparameter tuning, feature engineering.
• Doesn't bypass IV.1 documentation or gates.

Module 2 Summary

The one thing: Technique selection is the data scientist's call and the PM's to oversee — the distinction is the whole domain in miniature.

PM's job in this module: Ensure the technique choice is documented and justified against the problem, and that third-party or pretrained model use is a recorded decision.

Exam signals: Any stem where the PM is asked which algorithm, which model, or whether to use a pretrained one.

Trap to watch: Picking the algorithm. Even when the technically correct choice is obvious, selecting it is not the PM's move.

ECO tasks: IV.1

Module 3: Model Development and Training

Lessons 14-20 | The development phase — actually building and training the model.

☐ Lesson 14: ECO Task IV.3 — Manage AI/ML Model Training

Once technique is selected (IV.1) and prepared data is gated (IV.5), training begins. The PM manages training — coordinates the team's effort, tracks progress, monitors for issues (training time overruns, loss curves not converging, resource exhaustion), and surfaces blockers.

A worked example of that management responsibility: model training has been running 5 days against a planned 2-day window. The data scientist says "one more day should do it." What does the PM do? The right answer is to pause and conduct a structured root-cause review (data, technique, resources, hyperparameters), reassess against project plan, and make a documented decision. NOT "let them keep going" and NOT "switch to a smaller model."

KEY TAKEAWAYS

• Training = data scientist executes, PM manages.
• 2.5x time overrun = project event, not technical hiccup. Pause, review, decide.

💡 Memory Aid — Training Triage

When training overruns: review Data (quality, volume, distribution), Technique (algorithm fit), Hardware/resources, Results so far. Four categories to root-cause before proceeding.

PM Oversight Angle

• PM owns: Managing training execution; tracking progress; surfacing blockers; coordinating root-cause review when training overruns or fails.
• Deliverable: Training Status Tracker; root-cause documentation when training events occur; documented training-completion declaration.
• Iteration trigger: Training reveals technique mismatch → loop back to IV.1. Training reveals data issues → loop back to IV.4 or III.7.
• Escalation trigger: Training cost overrun beyond budget; resource constraints requiring infrastructure decisions.
• Wrong-answer trap: "Let the data scientist switch to a simpler model immediately" — bypasses root-cause review. The technique choice is documented in IV.1; changing it without review is governance bypass.
• Question pattern signal: Stems mentioning "training is taking longer than planned," "the data scientist requests more time," "training has failed," "model performance is below expected."
• ECO task tag: Domain IV, Task 3 — Manage AI/ML model training

☐ Lesson 15: AI Model Development Phase Overview

Phase IV — Model Development — is where the team applies the chosen technique to the prepared data to produce a model. The phase is iterative: train, evaluate, adjust, retrain. Multiple iterations are normal; a "one-shot training run" is rare.

The PM ensures iterations are tracked, lessons are captured per iteration, and the cumulative time/resource cost stays within budget.

KEY TAKEAWAYS

• Phase IV = train + evaluate + adjust + retrain, iteratively.
• Iterations are normal; cumulative cost is the PM's tracking concern.

☐ Lesson 16: Model Validation

Validation is the practice of testing the model on data it didn't see during training. Common approach: split the prepared dataset into training (~70%), validation (~15%, used to tune hyperparameters), and test (~15%, used for final unbiased evaluation).

The PM ensures validation is performed and results are documented before declaring training complete.

KEY TAKEAWAYS

• Validation = test on unseen data. Typical split: 70/15/15 train/validation/test.
• Required before training is declared complete.

☐ Lesson 17: Generalizing to New Data

The goal of training is generalization — performing well on data the model hasn't seen. Two failure modes:

• Overfitting — model memorizes training data, fails on new data. Symptom: high training accuracy, low validation accuracy.
• Underfitting — model fails to learn patterns even on training data. Symptom: low accuracy on both training and validation.

Both are technical problems the data scientist addresses, but the PM tracks them as project risks and ensures evaluation reports include them.

KEY TAKEAWAYS

• Overfit = memorizes, fails on new data.
• Underfit = doesn't learn even on training.
• Both are PM-tracked project risks.

☐ Lesson 18: Building GenAI Systems

GenAI systems differ from traditional ML in development:

• Foundation model is given — you don't train from scratch.
• Customization via prompting, RAG, or fine-tuning — not retraining the foundation.
• Output evaluation is harder — generative outputs are subjective; traditional accuracy metrics don't apply directly.

The PM coordinates GenAI development against the same technique-selection (IV.1), training-management (IV.3), and gate (IV.6) framework — but recognizes the work is more about prompt engineering, retrieval design, and evaluation criteria than traditional model building.

KEY TAKEAWAYS

• GenAI = customize a foundation model via prompting / RAG / fine-tuning.
• Same ECO framework, different specifics.

☐ Lesson 19: Retrieval-Augmented Generation (RAG)

RAG enhances a foundation model by retrieving relevant context at inference time and feeding it into the prompt. The model's output is grounded in retrieved documents rather than pure parametric memory.

When to use: when the foundation model needs domain-specific or current information that wasn't in its training data. Example: answering customer questions from your product documentation.

KEY TAKEAWAYS

• RAG = retrieve relevant context + generate from foundation model.
• Use when grounding in current/domain-specific information matters.

☐ Lesson 20: Fine-Tuning LLMs

Fine-tuning adjusts a foundation model's weights using task-specific data, producing a custom model that performs better on your task than the base model.

When to use: when prompting and RAG aren't sufficient; when task-specific patterns need to be learned; when the volume of task data is sufficient (typically thousands of examples minimum).

When NOT to use: small data, generic tasks, when prompting suffices, when RAG suffices, when latency is critical.

KEY TAKEAWAYS

• Fine-tuning = adjust foundation model weights with task-specific data.
• Use when prompt + RAG aren't enough AND task data is sufficient.

Module 3 Summary

The one thing: Phase IV is iterative by design: train, evaluate, adjust, retrain — and overfitting is the failure it exists to catch.

PM's job in this module: Manage the training effort — schedule, resources, documentation, and the split discipline that makes validation meaningful.

Exam signals: Stems about performance that looks excellent on training data, or a model that will not generalize.

Trap to watch: Treating a retrain cycle as slippage. Iteration inside Phase IV is the method working, not the schedule failing.

ECO tasks: IV.3

Module 4: Model QA/QC, Evaluation, and Iteration

Lessons 21-31 | The QA/QC and evaluation discipline that catches model defects before deployment.

☐ Lesson 21: ECO Task IV.2 — Oversee AI/ML model QA/QC (e.g., configuration management, model performance)

QA/QC = configuration management + model performance verification. The PM oversees quality assurance practices throughout development:

• Configuration management — versioning of code, data, model artifacts, hyperparameters, and environment.
• Performance verification — measuring against documented success criteria.
• Bias measurement — informational bias across user segments.
• Documentation — what was tested, with what data, what the result was.

Asterisked task: the first attempt form had no IV.2 questions. The retake form may differ. Cover it.

KEY TAKEAWAYS

• QA/QC = config management + performance verification + bias measurement + documentation.
• The PM oversees the QA/QC regime; the data scientist + ML engineer execute.

PM Oversight Angle

• PM owns: Overseeing the QA/QC program — ensuring config management, performance verification, bias measurement, and documentation are happening throughout development.
• Deliverable: QA/QC reports per training iteration; consolidated quality summary feeding into IV.6 gate.
• Iteration trigger: QA/QC reveals quality issues that exceed tolerance → iterate back to address them (more training, different technique, more data).
• Escalation trigger: QA/QC reveals systemic issues that require resource or scope decisions.
• Wrong-answer trap: "Skip QA/QC since the data scientist is confident in the model." QA/QC is regime-driven, not confidence-driven.
• Question pattern signal: Stems mentioning "the team is testing the model," "model performance is being measured," "QA hasn't started yet."
• ECO task tag: Domain IV, Task 2 — Oversee AI/ML model QA/QC

☐ Lesson 22: Why Model Evaluation Matters

Model evaluation answers "is the model good enough to ship?" Without evaluation, you have no objective basis for the IV.6 gate decision. PMI's framing: model evaluation is a discipline, not a step — done continuously during development, not just at the end.

KEY TAKEAWAYS

• Evaluation = objective basis for IV.6 gate.
• Evaluation is continuous, not a one-time end-of-development step.

☐ Lesson 23: When Model Evaluation Falls Short

Consequences of inadequate evaluation:

• Production failures — model performs differently on real data than test data.
• Compliance violations — bias or fairness issues surface post-deployment.
• Business impact — predictions drive bad decisions; revenue or trust erodes.
• Reputational damage — public AI failure (incorrect denials, biased outputs).

PMI's exam frequently tests recognition of "evaluation gap" scenarios — the wrong answer is usually "deploy and observe; we'll catch issues in production."

KEY TAKEAWAYS

• Inadequate evaluation = production failures, compliance violations, business impact, reputation risk.
• Wrong-answer trap: "deploy and observe" — evaluation is pre-deployment.

☐ Lesson 24: How to Evaluate a Model Effectively

PMI frames effective evaluation as three key questions answered in the context of the business requirements:

• Have we met our expected levels of fit? Does the model deliver the expected performance against its generalization requirements? What tuning approaches are being used so it generalizes well with minimal errors? How are decisions about hyperparameter configurations and settings recorded for future reference?
• Is the model meeting business KPIs? These are non-technical measures — ROI, time saved, and similar. A model that performs well technically but misses the business KPIs still requires action: iterate, refine, or even start over so it aligns with the business objectives.
• Have we thoroughly evaluated external and generative models? Foundation and pretrained models can accelerate the project but are harder to adjust and fine-tune, so their performance has to be confirmed rather than assumed. GenAI systems have to be checked against performance requirements, including poor text generation and undesired image outputs, and confirmed to align with your standards.

Evaluation does not stop at the gate. As data evolves, so must the model: if data drift or model drift occurs, retraining — or changing how you use a third-party model — may be needed to keep performance where it is required.

KEY TAKEAWAYS

• Three evaluation questions: expected levels of fit, business KPIs, external and generative models thoroughly evaluated.
• Technically strong but failing business KPIs is still a failure — iterate, refine, or restart.

☐ Lesson 25: Model Iteration — Why and When

Model iteration is the practice of repeatedly training, evaluating, adjusting, and retraining. Reasons to iterate:

• Performance below success criteria.
• Bias detected.
• New training data becomes available.
• Failure modes identified.
• Hyperparameters need tuning.

The PM tracks iteration count, cumulative time, and remaining budget. Iteration is normal; runaway iteration without convergence is a project risk.

KEY TAKEAWAYS

• Iteration is normal — train + evaluate + adjust + retrain.
• Runaway iteration without convergence = project risk; PM-tracked.

☐ Lesson 26: When to Retrain the Model

When to retrain depends on the model's performance trends, data stability, and the dynamics of its operating environment. PMI groups the triggers three ways:

• Performance triggers — retrain when performance metrics fall outside acceptable ranges: data distribution mismatch (new data no longer matches the distribution of the training data), unacceptable model performance (accuracy or error rates become unacceptable), and imbalance changes (the balance of classes in the data shifts significantly).
• Regular retraining — retraining on a schedule even while the model is still performing well. PMI's example: if data starts to drift after 2 to 3 months, plan to retrain every 6 weeks to stay ahead of the problem.
• Specific scenarios — driven by how stable the domain is. A handwriting-recognition model on stable data may only need retraining when a new handwriting style emerges; in dynamic environments where data and usage patterns change frequently, regular retraining is essential.

The retraining decision is PM-coordinated with stakeholders, not data-scientist-unilateral.

KEY TAKEAWAYS

• Three trigger categories: performance triggers, regular (scheduled) retraining, specific scenarios.
• Performance triggers = data distribution mismatch, unacceptable performance, imbalance changes.
• Retraining is stakeholder-coordinated, not unilateral.

☐ Lesson 27: Data Drift and Model Drift

(Same concepts that surface in Domain V monitoring — Domain IV is where the response capability is built.)

• Data drift — the gradual change in data characteristics over time that can lead to degraded model performance if not addressed.
• Model drift — the degradation in a model's performance over time as the underlying data distribution changes.

The two are linked, not independent: drift in the data is the cause, degraded model performance is the effect. Retraining addresses both. Both are detected through monitoring (V.4), but the response (retrain, recalibrate, replace) is built into the model life cycle plan from Phase V.

KEY TAKEAWAYS

• Data drift = the data changes. Model drift = performance degrades because the underlying data distribution changed.
• Data drift and model drift = inevitable post-deployment.
• Detection = V.4 (monitoring). Response = built in Phase V planning.

☐ Lesson 28: KPIs — Business and Technical

Two KPI tiers must align:

• Business KPIs — what the project was supposed to deliver in business terms (revenue, cost saving, customer satisfaction, decision quality).
• Technical KPIs — model performance metrics (accuracy, F1, recall, latency, throughput).

Technical KPIs that don't translate to business KPIs are vanity metrics. Business KPIs without technical KPI underpinning are unmeasurable. Both must be defined, tied to Domain II success criteria, and tracked throughout development and operations.

KEY TAKEAWAYS

• Business KPIs = business outcomes. Technical KPIs = model performance.
• Both required. Either alone is insufficient.

☐ Lesson 29: Audit Trails and Auditability

AI audit trails document the full path from data collection through model training to deployment to inference outputs. Why they matter:

• Compliance — regulators may inquire about specific decisions.
• Liability — when AI causes harm, audit trail enables accountability.
• Debugging — when production issues occur, audit trail enables root-cause analysis.
• Trust — stakeholders trust AI more when its behavior is auditable.

Audit trails should capture: input data, model version, prediction, timestamp, decision rationale, human-in-the-loop overrides.

KEY TAKEAWAYS

• Audit trails span data → training → deployment → inference.
• Required for compliance, liability, debugging, trust.

☐ Lesson 30: AI Transparency

Two distinct transparency concepts:

• Systemic transparency — visibility into all components and ingredients of the model: data sources, preprocessing, architecture, training parameters.
• Decision transparency — visibility into why a specific prediction was made.

Systemic transparency is achievable for most models. Decision transparency is hard — many modern models (especially deep learning) are "black boxes."

KEY TAKEAWAYS

• Systemic transparency = how the model was built. Achievable.
• Decision transparency = why this specific prediction. Often hard.

☐ Lesson 31: Explainability vs Interpretability

Often used interchangeably, technically distinct:

• Explainability (XAI) — methods to make decisions of any model understandable (post-hoc explanations).
• Interpretability — building inherently understandable models (interpretable by design).

For high-stakes decisions (healthcare, finance, legal), interpretability is preferred. For low-stakes decisions (recommendations), explainability post-hoc may suffice.

Not all algorithms can be fully explained — deep learning is famously a black box. The trade-off between performance and explainability is a project decision tied to V.3 (governance) and Domain I (Trustworthy AI).

KEY TAKEAWAYS

• XAI = post-hoc explanations of any model.
• Interpretability = inherently understandable models.
• High-stakes → prefer interpretability. Low-stakes → XAI post-hoc may work.

Module 4 Summary

The one thing: Evaluation is the objective basis for the IV.6 gate, and drift is why evaluation never finishes at deployment.

PM's job in this module: Oversee QA/QC — configuration management, performance verification, bias measurement, documentation — and keep business KPIs alongside technical ones.

Exam signals: Retraining triggers, degrading production accuracy, and any stem separating data drift (inputs changed) from model drift (performance fell as a result).

Trap to watch: Answering with a technical metric when the question asks whether the business criteria from II.8 are met.

ECO tasks: IV.2, with pulls to II.8 and V.4

Module 5: The Operationalization Gate and Phase V Closeout

Lessons 32-36 | The IV.6 gate and the closeout of Phase V (Model Evaluation) before transition to Domain V.

☐ Lesson 32: ECO Task IV.6 — Verify model ready for operationalization go/no-go decision

The second gate in Domain IV. After model is trained (IV.3), QA/QC'd (IV.2), and evaluated, the PM facilitates the operationalization-readiness gate. This is the gate that authorizes the project to enter Domain V.

PMI's model-readiness test asks three questions to determine whether the model is ready to move to the next CPMAI phase:

• Have you developed an AI model that aligns with the specific goals and objectives you set out to achieve?
• Does your AI model perform well on the data that has been carefully selected and prepared?
• Have you completed all the necessary steps of model evaluation — thorough testing, validation, and performance metrics analysis — to help ensure the model is robust, accurate, and reliable?

You proceed if the model meets all your requirements and you have adequate responses to the Phase V workbook questions. An adequate response is not necessarily a final answer — it is something you can work with, because without something to work with you cannot iterate. You should have clear, well-defined answers so you understand what you know and what you do not know.

Readiness is evaluated across the model's performance measures (levels of fit, generalization, accuracy and false positive/negative rates, and recorded hyperparameter changes), GenAI evaluation where GenAI is used (objective rather than subjective measures; no hallucinations or unwanted artifacts), and the plan for iteration and improvement. Resource and efficiency questions belong here too — model size, training and inference efficiency, operational efficiency, and edge-device resource demands all determine whether the model can actually run where it is going.

The decision is GO or NO-GO. Do not move to Phase VI until the model meets the requirements — and PMI is explicit that this may mean going back to Phase I, II, III, or IV to identify and address the issue. Better to step back than to move forward with a flawed model.

KEY TAKEAWAYS

• IV.6 = the gate authorizing entry to Domain V (operationalization).
• Readiness test: aligns with goals, performs on the prepared data, evaluation steps complete, adequate Phase V workbook responses.
• Two outcomes: GO / NO-GO. NO-GO = do not proceed; iterate back to Phase I, II, III, or IV depending on the root cause.

💡 Memory Aid — Gate Criteria

Aligned with the goals you set, performs on the carefully prepared data, evaluation fully completed (testing, validation, metrics), workbook responses adequate. The readiness test before the model crosses into production.

PM Oversight Angle

• PM owns: Facilitating the documented operationalization-readiness decision with stakeholders. Compiling QA/QC + evaluation findings into a gate decision package.
• Deliverable: Phase V Go/No-Go Decision Document — goal alignment, performance on the prepared data, completed testing/validation/metrics analysis, resource and efficiency findings, adequate Phase V workbook responses; decision (GO/NO-GO); stakeholder sign-off.
• Iteration trigger: Requirements not met → NO-GO, then iterate to the phase that owns the root cause. PMI's mapping: poor model performance → Phase IV; performance issues rooted in data quality → Phase III; fundamentally flawed or inadequate data → Phase II; consistent failure to meet KPIs → Phase I.
• Escalation trigger: A NO-GO that requires Phase I/II rework or materially changes project value; trustworthy-AI breach requiring legal or regulatory engagement.
• Wrong-answer trap: "Deploy to production and validate against business KPIs there." Bypasses the gate. Production validation isn't the gate — pre-deployment evaluation is.
• Question pattern signal: Stems mentioning "the model is ready to deploy," "the data scientist says training is complete," "the team wants to move to operationalization," "the model is being evaluated for production."
• ECO task tag: Domain IV, Task 6 — Verify model ready for operationalization go/no-go decision

☐ Lesson 33: Phase V — Preparing for Deployment / Model Readiness

Once IV.6 = GO, the project transitions to Domain V (operationalization). The "deployment readiness" deliverables include: trained model, audit trail, reproducible pipeline, monitoring plan, deployment plan (which V.1 builds), governance plan (which V.3 builds), contingency plan (which V.7 builds).

KEY TAKEAWAYS

• Post-IV.6 GO = project transitions to Domain V.
• Readiness deliverables = trained model + audit trail + pipelines + plans (deployment, governance, contingency, monitoring).

☐ Lesson 34: Phase V — Planning for Improvement (Iteration Plan)

Even after deployment, the model will need to improve. The iteration plan (built before deployment, executed throughout production) covers:

• Retraining cadence — scheduled or trigger-based.
• New data integration — how new data feeds back into retraining.
• Performance benchmarks — when does the model need to be improved vs replaced?
• Sunset criteria — when does the model retire?

The iteration plan is part of the deployment plan (V.1) and is monitored through V.4.

KEY TAKEAWAYS

• Iteration plan = retraining cadence + new data integration + benchmarks + sunset.
• Built before deployment, executed in production.

☐ Lesson 35: Iterating Back to Previous CPMAI Phases

Phase V findings can trigger iteration back to earlier phases (same pattern as Domain III's 12 iteration triggers). Common triggers from Phase V:

• Evaluation reveals data quality issues missed earlier → loop to Phase III.
• Evaluation reveals technique mismatch → loop to Phase IV (technique selection).
• Evaluation reveals scope mismatch with business needs → loop to Phase I.
• Any KPI unmet → "you cannot move past Phase V" — go back to the phase that owns the root cause; if the issue cannot be fixed there, the KPI itself is adjusted in Phase I.

KEY TAKEAWAYS

• Phase V findings can trigger iteration back to any earlier phase.
• Iteration is methodology-correct, not failure.

☐ Lesson 36: Phase IV Go/No-Go (General Closeout)

Beyond IV.5 and IV.6 (the explicit ECO gates), Phase IV has a general closeout: confirm all Phase IV objectives are met, all artifacts are documented, all decisions are traceable. This isn't a separate ECO task but is part of how the PM tracks Phase IV completion.

KEY TAKEAWAYS

• Phase IV closeout = objectives met + artifacts documented + decisions traceable.
• Cumulative tracking, not a separate ECO gate.

Module 5 Summary

The one thing: IV.6 is the gate that authorizes entry into operationalization, and it judges operational fit — not just model accuracy.

PM's job in this module: Verify readiness against documented criteria with stakeholders, then hand off to Domain V. A GO authorizes deployment; it does not perform it.

Exam signals: Stems asking what a GO decision authorizes, or who can declare it.

Trap to watch: Reading IV.6 as a technical sign-off the data science lead can give. It is a stakeholder decision against documented criteria.

ECO tasks: IV.6

Quick Reference: The Two Gates (IV.5 + IV.6)

Both gates are PM-facilitated, stakeholder-engaged, documented decisions. Wrong-answer traps are always: PM decides alone, technical workaround applied, or "proceed and fix later."

Quick Reference: Model Evaluation Checklist (IV.2 + IV.6)

Cross-Domain Links

• IV.4 (Data Transformation) ↔ Domain III: Phase III work begins after III.8 GO. Transformation is informed by III.1 (defined data) and III.7 (evaluation).
• IV.5 (Data Quality Gate) ↔ III.8: Two distinct gates at adjacent phase boundaries. III.8 = "do we have what we need?" IV.5 = "is the prepared output sufficient to train on?"
• IV.1 (Technique) ↔ Phase I AI Pattern: Technique selection is constrained by the AI pattern from Phase I. Misalignment = loop back to Phase I.
• IV.2 (QA/QC) ↔ Domain I (Tasks I.2, I.3, I.5): QA/QC overlaps transparency (I.2), bias checks (I.3), accountability documentation (I.5).
• IV.6 (Operationalization Gate) ↔ Domain V (Task V.1): The V.1 deployment plan is drafted iteratively before the gate — IV.6 judges operational fit against the environment that draft targets. IV.6 GO then authorizes V.1 finalization and deployment execution. Misalignment with the operational environment = loop back to V.1 or IV.1.
• IV.3 (Training) ↔ Domain V (Task V.4): Training metrics define the baseline that V.4 production metrics compare against.

Knowledge Check

### Question 1

Data preparation pipelines are complete and the data engineer reports the data is ready for training. The PM is asked to authorize the start of training. What's the BEST move?

A. Authorize training to proceed

B. Run the IV.5 verification gate with stakeholders before authorizing training — confirm the preparation plan was executed, the workbook steps are complete, no roadblocks remain, and the data is in the shape and format the model needs

C. Have the data scientist start training in parallel with the gate review

D. Defer the gate until after a few training iterations show whether the data is good enough

### Question 2

Model training has been running for 5 days against a planned 2-day window. The data scientist says one more day should do it. What should the PM do?

A. Allow another day since they're close

B. Pause training, conduct structured review of root cause (data, technique, resources, results), reassess against project plan, and make a documented decision on whether to continue, change approach, or escalate

C. Have them switch to a smaller model immediately

D. Cancel and restart from scratch

### Question 3

The team has completed model training, QA/QC, and evaluation. The data scientist proposes deploying to production. What should the PM do?

A. Authorize deployment

B. Run the IV.6 operationalization-readiness gate with stakeholders — does the model align with the goals set, does it perform on the carefully prepared data, are testing, validation, and metrics analysis complete, and are the Phase V workbook responses adequate?

C. Have the ML engineer start deployment while the gate is being scheduled

D. Defer deployment until production observes actual performance

### Question 4

True or False: ECO Tasks III.8 and IV.5 are the same gate.

### Question 5

The data scientist proposes a deep learning model for a high-stakes medical-imaging classification task. The healthcare client requires that AI decisions be explainable. What's the PM's BEST response?

A. Approve the deep learning approach since it offers higher accuracy

B. Document the technique selection and ensure trade-off between performance and explainability is presented to stakeholders for decision; consider interpretable-by-design alternatives

C. Have the data scientist proceed and add post-hoc XAI explanations after training

D. Reject deep learning and require an interpretable model

### Question 6

During QA/QC of a recommendation model, the team finds that recommendations show measurable demographic bias. What should the PM do?

A. Have the data scientist add a fairness post-processing layer

B. Treat as an ECO IV.2 + Domain I (Task I.3 — bias checks) issue: document the finding, escalate per accountability procedures, engage stakeholders for remediation decision, do not authorize IV.6 GO until bias is within tolerance

C. Deploy with a "monitor closely" flag and address bias in production

D. Reject the entire model and start over

### Question 7

A team is iterating a model with 4 training runs over 2 weeks, each one improving slightly but not meeting the success criteria. The data scientist suggests a 5th iteration. What should the PM do?

A. Approve the 5th iteration

B. Pause and review the iteration trajectory: are improvements converging or plateauing? Is the technique a fit? Is the data sufficient? Document a decision: continue, change technique, rescope, or escalate.

C. Have the data scientist try a different algorithm

D. Cancel the project

### Question 8

During the IV.6 gate review, the team confirms model performance meets success criteria but notes the chosen technique requires GPU compute that isn't available in the planned cloud production environment. What should the PM do?

A. Authorize GO and procure GPU compute concurrently

B. Treat it as a NO-GO at IV.6 — a model that cannot run where it is going is not ready for operationalization. Document the decision and iterate: to V.1 (deployment plan) to address infrastructure, or to IV.1 (technique) to choose a model that fits the planned environment.

C. Have the ML engineer optimize the model for CPU

D. Deploy to GPU on a different cloud while the team works the issue

### Question 9

True or False: An AutoML pipeline that automatically selects an algorithm and tunes hyperparameters bypasses the need for ECO Task IV.1 documentation.

### Question 10

The team has completed Phase IV. The data scientist asks whether to begin operationalization (Domain V) work in parallel with the IV.6 gate. What should the PM do?

A. Approve parallel work to save time

B. Confirm IV.6 must complete before Domain V work begins; sequential, not parallel; gate authorizes the transition

C. Have the ML engineer prepare deployment artifacts but not actually deploy until IV.6 GO

D. Defer the IV.6 gate and let Domain V work proceed

Closing reminders for Domain IV

• Domain IV is gate-heavy. Two of six tasks are explicit gates — a third of the domain by task count, and the ECO's other gate (III.8) feeds directly into them. Master the gate decision pattern.
• The PM holds the gate. When the data scientist wants to keep iterating, the ML engineer wants to keep tuning, the business wants to ship — the PM is the one running the documented decision against documented criteria. Do not let the gate become "rubber-stamped."
• III.8 ≠ IV.5. They look similar; they're not. III.8 = "do we have what we need?" (Phase II close). IV.5 = "is the prepared data sufficient to train?" (Phase III close). Tested directly.
• Operational fit is part of IV.6. A model that performs well but can't run in the planned production environment fails IV.6. Most exam stems test this through "the model needs GPU but the cloud is CPU" or "the model needs real-time but the environment is batch."
• Cross-domain pulls are dense. Domain II success criteria (II.8) feed IV.6's performance check. Domain I (Trustworthy AI) flows through IV.2 QA/QC. The V.1 deployment plan is drafted before IV.6 and finalized and executed after GO. Recognize the pulls in stems.

Next: domain-I-trustworthy-ai.md (Domain I — Responsible & Trustworthy AI Efforts, 15% weight, full guide)