Domain IV: Manage AI Model Development and Evaluation — Comprehensive Study Guide

Note from docs/ECO_TASK_REFERENCE.md: the score report flagged Task IV.2 (Oversee AI/ML model QA/QC) as having no questions on his form. Cover it anyway — the retake form is randomized.

>

Two of six tasks are explicit go/no-go gates — that's 33% of the domain by task count. Combined with the gate in Domain III (III.8), three gates concentrate ~10-15 exam questions. Master all three.

Overview

Domain IV is the most procedurally complex of the three weak domains. It spans three CPMAI methodology phases (Data Preparation, Model Development, Model Evaluation) and contains two of the three explicit go/no-go gates in the entire ECO. Every task begins with an oversight verb: oversee, manage, verify. The PM is responsible for ensuring that data preparation produces sufficient quality, model technique selection is sound, training is managed, QA/QC standards are upheld, and the model is verified ready before it crosses into Operationalization (Domain V).

The unifying pattern: Domain IV tests whether the project manager can hold the gate. The data scientist wants to keep iterating. The ML engineer wants to keep tuning. The business stakeholder wants to ship. The PM is the one who facilitates the documented decision against documented criteria — and who is willing to call ITERATE or DESCOPE when the criteria aren't met.

Most wrong-answer traps in Domain IV are technically-correct moves that bypass either a gate, an iteration trigger, or a stakeholder decision. The same oversight-verb framing from Domain III applies — and it applies more sharply because Domain IV's two gates have very specific decision criteria.

Table of Contents

• Module 1: Data Preparation — Pipelines, Quality, and the Prep Gate (Lessons 1-7)
• Module 2: Model Technique and Selection (Lessons 8-13)
• Module 3: Model Development and Training (Lessons 14-20)
• Module 4: Model QA/QC, Evaluation, and Iteration (Lessons 21-31)
• Module 5: The Operationalization Gate and Phase V Closeout (Lessons 32-36)
• Quick Reference: The Two Gates (IV.5 + IV.6) Cheat
• Quick Reference: Model Evaluation Checklist
• Cross-Domain Links
• Knowledge Check
• Memory Aids & Mnemonics Summary

Module 1: Data Preparation — Pipelines, Quality, and the Prep Gate

Lesson 1: ECO Task IV.4 — Manage Data Transformation to Conduct Data Preparation

After the data is gathered (III.5) and the Phase II gate (III.8) has decided GO, the team enters Phase III — Data Preparation. The PM's job is to manage the transformation effort: ensure pipelines are built, transformations are documented, quality is preserved, and the prep work feeds the success criteria from Domain II.

The PM does not write transformation code. The PM coordinates the data engineering team, tracks pipeline development, and ensures the prepared dataset is usable before model training begins.

KEY TAKEAWAYS

• Data transformation = Phase III work. Begins after III.8 GO.
• PM manages transformation; data engineers execute it.

💡 Memory Aid — TRIM Data Prep

PM Oversight Angle

• PM owns: Coordinating data preparation execution; tracking pipeline development; ensuring transformations are documented; verifying prepared dataset is usable for training.
• Deliverable: Data Preparation Plan + status tracker; documented pipelines; prepared dataset ready for IV.5 verification.
• Iteration trigger: Transformation reveals data issues that III.7/III.8 missed → loop back to III.7 (re-evaluate) or III.1 (re-define requirements).
• Escalation trigger: Transformation cost or timeline exceeds project tolerance; data quality remediation requires new data sourcing.
• Wrong-answer trap: "Have the data scientist start training on the partially-prepared data while the engineer finishes." Training before prep is complete inserts data quality issues into the model.
• Question pattern signal: Stems mentioning "the team is preparing data," "data transformation is in progress," "the data engineer is building pipelines."
• ECO task tag: Domain IV, Task 4 — Manage data transformation to conduct data preparation

Lesson 2: Data Preparation Concepts

Data preparation is the work of making raw data usable for training. It includes:

• Cleaning — removing errors, duplicates, outliers, inconsistencies.
• Transforming — changing format, structure, or representation (e.g., normalization, encoding, aggregation).
• Imputing — filling missing values (or flagging and excluding).
• Augmenting — generating additional training examples (rotation/cropping for images, paraphrasing for text).
• Splitting — dividing into training, validation, and test sets.
• Labeling — for supervised learning (often expensive and time-consuming).

Most AI projects spend 70-80% of their time on data preparation. Underestimating this is a top reason projects miss deadlines.

KEY TAKEAWAYS

• Data prep includes clean, transform, impute, augment, split, label.
• Typically 70-80% of project time.

Lesson 3: Data Engineering and Pipelines

Data engineering builds the pipelines that move data from sources through preparation and into the model's training environment. Key pipeline concepts:

• ETL (Extract, Transform, Load) — extract from source, transform to target schema, load to destination.
• ELT (Extract, Load, Transform) — load raw, transform in destination (modern cloud pattern).
• Streaming pipelines — continuous data flow vs scheduled batch.
• Data lakes vs data warehouses — lakes hold raw data flexibly; warehouses hold structured, cleaned data.

The PM coordinates pipeline ownership and ensures the deployment plan (V.1) accounts for pipeline maintenance in production.

KEY TAKEAWAYS

• ETL = transform before load. ELT = transform after load.
• Lakes = raw flexibility. Warehouses = structured precision.
• Pipeline ownership = PM coordination concern.

Lesson 4: Data Collection and Ingestion

Collection brings data into the pipeline; ingestion is the technical implementation. Both happen in Phase III but are heavily informed by Phase II decisions (III.3 sources, III.5 gathered data). Common ingestion patterns: API pulls, file drops, database replication, streaming connectors, batch uploads.

PM concerns: ingestion reliability, error handling, data validation at entry, source-side rate limits or licensing.

KEY TAKEAWAYS

• Collection + ingestion = bringing data into the pipeline.
• Ingestion is informed by Phase II source decisions.

Lesson 5: Data Preparation Pipelines

Phase III's signature deliverable is a data preparation pipeline that:

1. Ingests from identified sources
2. Validates input format and content
3. Cleans (remove errors/duplicates)
4. Transforms (format/schema)
5. Imputes missing values
6. Augments if needed
7. Splits into training/validation/test
8. Outputs to model training environment

The pipeline is reusable and reproducible — same input + same pipeline = same output. Reproducibility is a governance requirement (V.3).

KEY TAKEAWAYS

• Pipeline = reproducible end-to-end transformation of raw data into training-ready form.
• Reproducibility is a governance requirement.

Lesson 6: Pipeline Complexity

Real-world pipelines are complex. PMI flags this as a project risk:

• Multiple data sources with different formats and refresh cadences.
• Dependencies between pipeline stages (one step's output is another's input).
• Failure handling — what happens when a source is unavailable or input is malformed?
• Versioning — pipelines themselves change as data and requirements evolve.
• Monitoring — pipelines need observability or failures go undetected.

The PM doesn't design the pipeline but tracks complexity as a project risk and ensures observability is built in.

KEY TAKEAWAYS

• Real pipelines are complex; complexity = risk.
• Failure handling, versioning, observability = required, not optional.

Lesson 7: ECO Task IV.5 — Verify Data Quality (GATE)

This is distinct from III.8, which asked "do we have the data and understanding?" — a Phase II close-out gate. IV.5 asks "now that we've prepared the data, is the prepared output of sufficient quality to train on?" — a Phase III close-out gate.

The decision criteria:

• Quality dimensions (ACCTUVI) all evaluated and within tolerance.
• Coverage of required attributes from III.1.
• Bias measurements within trustworthy-AI tolerance.
• Volume sufficient for the chosen technique.
• Pipeline reproducibility verified.

The decision has three outcomes (same as III.8): GO (proceed to training), ITERATE (loop back to fix), DESCOPE (reduce model scope to what data supports).

KEY TAKEAWAYS

• IV.5 ≠ III.8. III.8 = Phase II close (data understanding). IV.5 = Phase III close (data prep complete).
• Three outcomes: GO / ITERATE / DESCOPE.
• Decision criteria: quality dimensions, coverage, bias, volume, reproducibility.

💡 Memory Aid — QCBVR Gate Criteria

PM Oversight Angle

• PM owns: Facilitating the documented data-quality verification with stakeholders. Compiling pipeline output evaluation into a gate decision package.
• Deliverable: Phase III Go/No-Go Decision — quality findings, coverage, bias measurements, volume assessment, reproducibility verification, decision (GO/ITERATE/DESCOPE), stakeholders engaged.
• Iteration trigger: Quality, coverage, or bias findings below threshold → ITERATE back to IV.4 (more transformation work) or III.7 (re-evaluate raw data) or III.1 (re-define requirements).
• Escalation trigger: ITERATE that requires Phase II changes; DESCOPE that materially changes project value.
• Wrong-answer trap: "Begin model training and improve data quality in parallel." Bypasses the gate. Quality issues compound into model defects.
• Question pattern signal: Stems mentioning "data preparation is complete," "the team is ready to train," "data quality is being verified," "the data engineer says the pipelines are done."
• ECO task tag: Domain IV, Task 5 — Verify data quality for go/no-go decision to conduct data preparation

Module 2: Model Technique and Selection

Lesson 8: ECO Task IV.1 — Oversee AI/ML Model Technique(s)

The PM oversees the team's selection of model technique(s) — the algorithmic approach (supervised/unsupervised/reinforcement learning), the model family (linear, tree-based, neural network, transformer), and any pretrained-model decisions. The PM doesn't pick the technique; the data scientist does. The PM ensures the choice is documented, tied to the AI pattern from Phase I, and aligned with the project's success criteria.

A common exam scenario: the data scientist proposes a complex deep-learning model. The right PM response is rarely "approve" — it's "ensure the choice is documented and justified against the AI pattern, success criteria, and operational constraints (cost, latency, explainability)."

KEY TAKEAWAYS

• Technique selection = data scientist's call, PM-overseen.
• Documentation + justification against pattern + criteria + operational constraints.

PM Oversight Angle

• PM owns: Overseeing technique selection; ensuring the choice is documented, justified, and aligned with Phase I AI pattern + success criteria.
• Deliverable: Model Technique Justification — section of the CPMAI workbook documenting algorithm/family/pretrained choices, rationale, alignment with pattern, operational implications.
• Iteration trigger: Selected technique reveals operational constraint mismatch (e.g., real-time latency required but technique can't deliver) → loop back to V.1 deployment plan or IV.1 re-selection.
• Escalation trigger: Technique requires resources, cost, or vendor relationships beyond project authority.
• Wrong-answer trap: "Approve the data scientist's choice and proceed." Approval without documentation is a governance gap.
• Question pattern signal: Stems mentioning "the data scientist proposes [model]," "the team is choosing between [techniques]," "an algorithm has been selected."
• ECO task tag: Domain IV, Task 1 — Oversee AI/ML model technique(s)

Lesson 9: Machine Learning Fundamentals — Algorithm vs Model

Two terms commonly confused on the exam:

• Machine learning algorithm — the procedure for learning patterns from data (e.g., gradient descent, decision tree induction).
• Machine learning model — the result of running an algorithm on data (the trained artifact that makes predictions).

You train an algorithm on data to produce a model. The model is what gets deployed.

KEY TAKEAWAYS

• Algorithm = procedure. Model = trained artifact.
• You train an algorithm to produce a model.

Lesson 10: ML Algorithm Basics

ML lets computers learn patterns from data and make predictions. Three high-level categories:

• Supervised learning — learn from labeled examples (input → known output).
• Unsupervised learning — find structure in unlabeled data (clustering, dimensionality reduction).
• Reinforcement learning — learn by trial-and-error in an environment with rewards.

The choice of category depends on the AI pattern and the data available:

• Recognition / Classification → typically supervised.
• Pattern discovery → unsupervised.
• Sequential decision-making → reinforcement.

KEY TAKEAWAYS

• 3 categories: supervised, unsupervised, reinforcement.
• Category choice depends on AI pattern + data availability.

Lesson 11: Pretrained Models, Foundation Models, and GenAI

Modern AI rarely trains from scratch. Three patterns:

• Pretrained model — a model already trained on a generic task that you adapt for your specific use.
• Foundation model — a very large pretrained model (e.g., GPT-4, Claude, LLaMA) that can be specialized via prompting or fine-tuning.
• GenAI — generative AI that produces new content (text, images, audio, code) — typically built on foundation models.

Using pretrained / foundation / GenAI models reduces the data needed for training (Phase II decisions reflect this — see III.8 questions about "can you use pretrained models?").

KEY TAKEAWAYS

• 3 patterns: pretrained, foundation, GenAI.
• All reduce required training data — feedback into Phase II gate (III.8).

Lesson 12: Transfer Learning and Third-Party Models

KEY TAKEAWAYS

• Transfer learning = pretrained + fine-tune on your task.
• Third-party models bring governance and provenance questions to V.3.

Lesson 13: Automated Machine Learning (AutoML)

For the PM: AutoML doesn't remove the need for documented justification (IV.1). The output of AutoML is a chosen technique; it still needs to be documented, evaluated, and gated through IV.5/IV.6.

KEY TAKEAWAYS

• AutoML = automated technique selection, hyperparameter tuning, feature engineering.
• Doesn't bypass IV.1 documentation or gates.

Module 3: Model Development and Training

Lesson 14: ECO Task IV.3 — Manage AI/ML Model Training

Once technique is selected (IV.1) and prepared data is gated (IV.5), training begins. The PM manages training — coordinates the team's effort, tracks progress, monitors for issues (training time overruns, loss curves not converging, resource exhaustion), and surfaces blockers.

A specific exam scenario PMI tests: model training has been running 5 days against a planned 2-day window. The data scientist says "one more day should do it." What does the PM do? The right answer is to pause and conduct a structured root-cause review (data, technique, resources, hyperparameters), reassess against project plan, and make a documented decision. NOT "let them keep going" and NOT "switch to a smaller model."

KEY TAKEAWAYS

• Training = data scientist executes, PM manages.
• 2.5x time overrun = project event, not technical hiccup. Pause, review, decide.

💡 Memory Aid — DTHR Training Triage

When training overruns: review Data (quality, volume, distribution), Technique (algorithm fit), Hardware/resources, Results so far. Four categories to root-cause before proceeding.

PM Oversight Angle

• PM owns: Managing training execution; tracking progress; surfacing blockers; coordinating root-cause review when training overruns or fails.
• Deliverable: Training Status Tracker; root-cause documentation when training events occur; documented training-completion declaration.
• Iteration trigger: Training reveals technique mismatch → loop back to IV.1. Training reveals data issues → loop back to IV.4 or III.7.
• Escalation trigger: Training cost overrun beyond budget; resource constraints requiring infrastructure decisions.
• Wrong-answer trap: "Let the data scientist switch to a simpler model immediately" — bypasses root-cause review. The technique choice is documented in IV.1; changing it without review is governance bypass.
• Question pattern signal: Stems mentioning "training is taking longer than planned," "the data scientist requests more time," "training has failed," "model performance is below expected."
• ECO task tag: Domain IV, Task 3 — Manage AI/ML model training

Lesson 15: AI Model Development Phase Overview

Phase IV — Model Development — is where the team applies the chosen technique to the prepared data to produce a model. The phase is iterative: train, evaluate, adjust, retrain. Multiple iterations are normal; a "one-shot training run" is rare.

The PM ensures iterations are tracked, lessons are captured per iteration, and the cumulative time/resource cost stays within budget.

KEY TAKEAWAYS

• Phase IV = train + evaluate + adjust + retrain, iteratively.
• Iterations are normal; cumulative cost is the PM's tracking concern.

Lesson 16: Model Validation

Validation is the practice of testing the model on data it didn't see during training. Common approach: split the prepared dataset into training (~70%), validation (~15%, used to tune hyperparameters), and test (~15%, used for final unbiased evaluation).

The PM ensures validation is performed and results are documented before declaring training complete.

KEY TAKEAWAYS

• Validation = test on unseen data. Typical split: 70/15/15 train/validation/test.
• Required before training is declared complete.

Lesson 17: Generalizing to New Data

The goal of training is generalization — performing well on data the model hasn't seen. Two failure modes:

• Overfitting — model memorizes training data, fails on new data. Symptom: high training accuracy, low validation accuracy.
• Underfitting — model fails to learn patterns even on training data. Symptom: low accuracy on both training and validation.

Both are technical problems the data scientist addresses, but the PM tracks them as project risks and ensures evaluation reports include them.

KEY TAKEAWAYS

• Overfit = memorizes, fails on new data.
• Underfit = doesn't learn even on training.
• Both are PM-tracked project risks.

Lesson 18: Building GenAI Systems

GenAI systems differ from traditional ML in development:

• Foundation model is given — you don't train from scratch.
• Customization via prompting, RAG, or fine-tuning — not retraining the foundation.
• Output evaluation is harder — generative outputs are subjective; traditional accuracy metrics don't apply directly.

The PM coordinates GenAI development against the same technique-selection (IV.1), training-management (IV.3), and gate (IV.6) framework — but recognizes the work is more about prompt engineering, retrieval design, and evaluation criteria than traditional model building.

KEY TAKEAWAYS

• GenAI = customize a foundation model via prompting / RAG / fine-tuning.
• Same ECO framework, different specifics.

Lesson 19: Retrieval-Augmented Generation (RAG)

RAG enhances a foundation model by retrieving relevant context at inference time and feeding it into the prompt. The model's output is grounded in retrieved documents rather than pure parametric memory.

When to use: when the foundation model needs domain-specific or current information that wasn't in its training data. Example: answering customer questions from your product documentation.

KEY TAKEAWAYS

• RAG = retrieve relevant context + generate from foundation model.
• Use when grounding in current/domain-specific information matters.

Lesson 20: Fine-Tuning LLMs

Fine-tuning adjusts a foundation model's weights using task-specific data, producing a custom model that performs better on your task than the base model.

When to use: when prompting and RAG aren't sufficient; when task-specific patterns need to be learned; when the volume of task data is sufficient (typically thousands of examples minimum).

When NOT to use: small data, generic tasks, when prompting suffices, when RAG suffices, when latency is critical.

KEY TAKEAWAYS

• Fine-tuning = adjust foundation model weights with task-specific data.
• Use when prompt + RAG aren't enough AND task data is sufficient.

Module 4: Model QA/QC, Evaluation, and Iteration

Lesson 21: ECO Task IV.2 — Oversee AI/ML Model QA/QC

QA/QC = configuration management + model performance verification. The PM oversees quality assurance practices throughout development:

• Configuration management — versioning of code, data, model artifacts, hyperparameters, and environment.
• Performance verification — measuring against documented success criteria.
• Bias measurement — informational bias across user segments.
• Documentation — what was tested, with what data, what the result was.

Asterisked task: the first attempt form had no IV.2 questions. The retake form may differ. Cover it.

KEY TAKEAWAYS

• QA/QC = config management + performance verification + bias measurement + documentation.
• The PM oversees the QA/QC regime; the data scientist + ML engineer execute.

PM Oversight Angle

• PM owns: Overseeing the QA/QC program — ensuring config management, performance verification, bias measurement, and documentation are happening throughout development.
• Deliverable: QA/QC reports per training iteration; consolidated quality summary feeding into IV.6 gate.
• Iteration trigger: QA/QC reveals quality issues that exceed tolerance → ITERATE back to address (more training, different technique, more data).
• Escalation trigger: QA/QC reveals systemic issues that require resource or scope decisions.
• Wrong-answer trap: "Skip QA/QC since the data scientist is confident in the model." QA/QC is regime-driven, not confidence-driven.
• Question pattern signal: Stems mentioning "the team is testing the model," "model performance is being measured," "QA hasn't started yet."
• ECO task tag: Domain IV, Task 2 — Oversee AI/ML model QA/QC

Lesson 22: Why Model Evaluation Matters

Model evaluation answers "is the model good enough to ship?" Without evaluation, you have no objective basis for the IV.6 gate decision. PMI's framing: model evaluation is a discipline, not a step — done continuously during development, not just at the end.

KEY TAKEAWAYS

• Evaluation = objective basis for IV.6 gate.
• Evaluation is continuous, not a one-time end-of-development step.

Lesson 23: When Model Evaluation Falls Short

Consequences of inadequate evaluation:

• Production failures — model performs differently on real data than test data.
• Compliance violations — bias or fairness issues surface post-deployment.
• Business impact — predictions drive bad decisions; revenue or trust erodes.
• Reputational damage — public AI failure (incorrect denials, biased outputs).

PMI's exam frequently tests recognition of "evaluation gap" scenarios — the wrong answer is usually "deploy and observe; we'll catch issues in production."

KEY TAKEAWAYS

• Inadequate evaluation = production failures, compliance violations, business impact, reputation risk.
• Wrong-answer trap: "deploy and observe" — evaluation is pre-deployment.

Lesson 24: How to Evaluate a Model Effectively

Effective evaluation answers structured questions:

• Performance against success criteria (Domain II, II.8) — accuracy, F1, recall, latency, business KPIs.
• Performance across user segments — does the model perform consistently across demographics, geographies, time periods?
• Edge case coverage — how does the model handle inputs at the distribution boundary?
• Failure mode analysis — when the model fails, how does it fail? Catastrophically? Gracefully?
• Comparison vs baseline — does the AI outperform a non-AI baseline (rules, prior model, human)?
• Bias measurement — informational bias measured and within tolerance?

KEY TAKEAWAYS

• Effective evaluation = 6 dimensions (criteria, segments, edge cases, failure modes, baseline, bias).
• Comparison-to-baseline is critical — if AI doesn't beat the rule-based baseline, you don't have a project.

Lesson 25: Model Iteration — Why and When

Model iteration is the practice of repeatedly training, evaluating, adjusting, and retraining. Reasons to iterate:

• Performance below success criteria.
• Bias detected.
• New training data becomes available.
• Failure modes identified.
• Hyperparameters need tuning.

The PM tracks iteration count, cumulative time, and remaining budget. Iteration is normal; runaway iteration without convergence is a project risk.

KEY TAKEAWAYS

• Iteration is normal — train + evaluate + adjust + retrain.
• Runaway iteration without convergence = project risk; PM-tracked.

Lesson 26: When to Retrain the Model

Triggers for retraining:

• Scheduled refresh — periodic retraining on accumulated new data.
• Data drift detected — production data has shifted from training distribution.
• Model drift detected — model performance has degraded.
• New requirements — business success criteria changed; model must adapt.
• External shock — environment changed (e.g., COVID-19 e-commerce example).

The retraining decision is PM-coordinated with stakeholders, not data-scientist-unilateral.

KEY TAKEAWAYS

• 5 retraining triggers: scheduled, data drift, model drift, new requirements, external shock.
• Retraining is stakeholder-coordinated, not unilateral.

Lesson 27: Data Drift and Model Drift

(Same concepts that surface in Domain V monitoring — Domain IV is where the response capability is built.)

• Data drift — production data distribution shifts away from training distribution.
• Model drift — model predictions degrade over time even on similar data.

Both are detected through monitoring (V.4), but the response (retrain, recalibrate, replace) is built into the model life cycle plan from Phase V.

KEY TAKEAWAYS

• Data drift and model drift = inevitable post-deployment.
• Detection = V.4 (monitoring). Response = built in Phase V planning.

Lesson 28: KPIs — Business and Technical

Two KPI tiers must align:

• Business KPIs — what the project was supposed to deliver in business terms (revenue, cost saving, customer satisfaction, decision quality).
• Technical KPIs — model performance metrics (accuracy, F1, recall, latency, throughput).

Technical KPIs that don't translate to business KPIs are vanity metrics. Business KPIs without technical KPI underpinning are unmeasurable. Both must be defined, tied to Domain II success criteria, and tracked throughout development and operations.

KEY TAKEAWAYS

• Business KPIs = business outcomes. Technical KPIs = model performance.
• Both required. Either alone is insufficient.

Lesson 29: Audit Trails and Auditability

AI audit trails document the full path from data collection through model training to deployment to inference outputs. Why they matter:

• Compliance — regulators may inquire about specific decisions.
• Liability — when AI causes harm, audit trail enables accountability.
• Debugging — when production issues occur, audit trail enables root-cause analysis.
• Trust — stakeholders trust AI more when its behavior is auditable.

Audit trails should capture: input data, model version, prediction, timestamp, decision rationale, human-in-the-loop overrides.

KEY TAKEAWAYS

• Audit trails span data → training → deployment → inference.
• Required for compliance, liability, debugging, trust.

Lesson 30: AI Transparency

Two distinct transparency concepts:

• Systemic transparency — visibility into all components and ingredients of the model: data sources, preprocessing, architecture, training parameters.
• Decision transparency — visibility into why a specific prediction was made.

Systemic transparency is achievable for most models. Decision transparency is hard — many modern models (especially deep learning) are "black boxes."

KEY TAKEAWAYS

• Systemic transparency = how the model was built. Achievable.
• Decision transparency = why this specific prediction. Often hard.

Lesson 31: Explainability vs Interpretability

Often used interchangeably, technically distinct:

• Explainability (XAI) — methods to make decisions of any model understandable (post-hoc explanations).
• Interpretability — building inherently understandable models (interpretable by design).

For high-stakes decisions (healthcare, finance, legal), interpretability is preferred. For low-stakes decisions (recommendations), explainability post-hoc may suffice.

Not all algorithms can be fully explained — deep learning is famously a black box. The trade-off between performance and explainability is a project decision tied to V.3 (governance) and Domain I (Trustworthy AI).

KEY TAKEAWAYS

• XAI = post-hoc explanations of any model.
• Interpretability = inherently understandable models.
• High-stakes → prefer interpretability. Low-stakes → XAI post-hoc may work.

Module 5: The Operationalization Gate and Phase V Closeout

Lesson 32: ECO Task IV.6 — Verify Model Ready for Operationalization (GATE)

Decision criteria:

• Performance against success criteria (Domain II) verified.
• Bias measurements within trustworthy-AI tolerance.
• Robustness to edge cases evaluated.
• Comparison vs baseline favorable.
• Reproducibility of training pipeline confirmed.
• Audit trail and documentation complete.
• Trustworthy AI alignment (Domain I) — privacy, security, governance, transparency, ethics — all satisfied.
• Operational fit — can the chosen technique actually run in the planned production environment (V.1)?

Three outcomes (same pattern): GO (proceed to Domain V deployment), ITERATE (loop back to address), DESCOPE (reduce model scope or capabilities).

KEY TAKEAWAYS

• IV.6 = the gate authorizing entry to Domain V (operationalization).
• 8 decision criteria including operational-fit check (does it run in production environment?).
• Three outcomes: GO / ITERATE / DESCOPE.

💡 Memory Aid — PBRBARTAO Gate Criteria

PM Oversight Angle

• PM owns: Facilitating the documented operationalization-readiness decision with stakeholders. Compiling QA/QC + evaluation findings into a gate decision package.
• Deliverable: Phase V Go/No-Go Decision Document — performance, bias, robustness, baseline, audit, reproducibility, trustworthy-AI, operational-fit findings; decision (GO/ITERATE/DESCOPE); stakeholder sign-off.
• Iteration trigger: Any criterion below threshold → ITERATE. Most common: performance below criteria, bias breach, baseline not beaten.
• Escalation trigger: ITERATE that requires Phase I/II rework; DESCOPE that materially changes project value; trustworthy-AI breach requiring legal or regulatory engagement.
• Wrong-answer trap: "Deploy to production and validate against business KPIs there." Bypasses the gate. Production validation isn't the gate — pre-deployment evaluation is.
• Question pattern signal: Stems mentioning "the model is ready to deploy," "the data scientist says training is complete," "the team wants to move to operationalization," "the model is being evaluated for production."
• ECO task tag: Domain IV, Task 6 — Verify model ready for operationalization go/no-go decision

Lesson 33: Phase V — Preparing for Deployment / Model Readiness

Once IV.6 = GO, the project transitions to Domain V (operationalization). The "deployment readiness" deliverables include: trained model, audit trail, reproducible pipeline, monitoring plan, deployment plan (which V.1 builds), governance plan (which V.3 builds), contingency plan (which V.7 builds).

KEY TAKEAWAYS

• Post-IV.6 GO = project transitions to Domain V.
• Readiness deliverables = trained model + audit trail + pipelines + plans (deployment, governance, contingency, monitoring).

Lesson 34: Phase V — Planning for Improvement (Iteration Plan)

Even after deployment, the model will need to improve. The iteration plan (built before deployment, executed throughout production) covers:

• Retraining cadence — scheduled or trigger-based.
• New data integration — how new data feeds back into retraining.
• Performance benchmarks — when does the model need to be improved vs replaced?
• Sunset criteria — when does the model retire?

The iteration plan is part of the deployment plan (V.1) and is monitored through V.4.

KEY TAKEAWAYS

• Iteration plan = retraining cadence + new data integration + benchmarks + sunset.
• Built before deployment, executed in production.

Lesson 35: Iterating Back to Previous CPMAI Phases

Phase V findings can trigger iteration back to earlier phases (same pattern as Domain III's 12 iteration triggers). Common triggers from Phase V:

• Evaluation reveals data quality issues missed earlier → loop to Phase III.
• Evaluation reveals technique mismatch → loop to Phase IV (technique selection).
• Evaluation reveals scope mismatch with business needs → loop to Phase I.
• Evaluation reveals trustworthy-AI gap → may loop to Phase II (data sourcing) or Phase I (problem definition).

KEY TAKEAWAYS

• Phase V findings can trigger iteration back to any earlier phase.
• Iteration is methodology-correct, not failure.

Lesson 36: Phase IV Go/No-Go (General Closeout)

Beyond IV.5 and IV.6 (the explicit ECO gates), Phase IV has a general closeout: confirm all Phase IV objectives are met, all artifacts are documented, all decisions are traceable. This isn't a separate ECO task but is part of how the PM tracks Phase IV completion.

KEY TAKEAWAYS

• Phase IV closeout = objectives met + artifacts documented + decisions traceable.
• Cumulative tracking, not a separate ECO gate.

Quick Reference: The Two Gates (IV.5 + IV.6)

Quick Reference: Model Evaluation Checklist (IV.2 + IV.6)

Cross-Domain Links

• IV.4 (Data Transformation) ↔ Domain III: Phase III work begins after III.8 GO. Transformation is informed by III.1 (defined data) and III.7 (evaluation).
• IV.5 (Data Quality Gate) ↔ III.8: Two distinct gates at adjacent phase boundaries. III.8 = "do we have what we need?" IV.5 = "is the prepared output sufficient to train on?"
• IV.1 (Technique) ↔ Phase I AI Pattern: Technique selection is constrained by the AI pattern from Phase I. Misalignment = loop back to Phase I.
• IV.2 (QA/QC) ↔ Domain I (Tasks I.2, I.3, I.5): QA/QC overlaps transparency (I.2), bias checks (I.3), accountability documentation (I.5).
• IV.6 (Operationalization Gate) ↔ Domain V (Task V.1): IV.6 GO authorizes V.1 deployment plan execution. Misalignment with operational environment = loop back to V.1 or IV.1.
• IV.3 (Training) ↔ Domain V (Task V.4): Training metrics define the baseline that V.4 production metrics compare against.

Knowledge Check

Question 1

Data preparation pipelines are complete and the data engineer reports the data is ready for training. The PM is asked to authorize the start of training. What's the BEST move?

A. Authorize training to proceed

B. Run the IV.5 verification gate — quality dimensions, coverage, bias, volume, reproducibility — with stakeholders before authorizing training

C. Have the data scientist start training in parallel with the gate review

D. Defer the gate until after a few training iterations show whether the data is good enough

Question 2

Model training has been running for 5 days against a planned 2-day window. The data scientist says one more day should do it. What should the PM do?

A. Allow another day since they're close

B. Pause training, conduct structured review of root cause (data, technique, resources, results), reassess against project plan, and make a documented decision on whether to continue, change approach, or escalate

C. Have them switch to a smaller model immediately

D. Cancel and restart from scratch

Question 3

The team has completed model training, QA/QC, and evaluation. The data scientist proposes deploying to production. What should the PM do?

A. Authorize deployment

B. Run the IV.6 operationalization-readiness gate with stakeholders, evaluating performance, bias, robustness, baseline, audit, reproducibility, trustworthy-AI alignment, and operational fit

C. Have the ML engineer start deployment while the gate is being scheduled

D. Defer deployment until production observes actual performance

Question 4

True or False: ECO Tasks III.8 and IV.5 are the same gate.

Question 5

The data scientist proposes a deep learning model for a high-stakes medical-imaging classification task. The healthcare client requires that AI decisions be explainable. What's the PM's BEST response?

A. Approve the deep learning approach since it offers higher accuracy

B. Document the technique selection and ensure trade-off between performance and explainability is presented to stakeholders for decision; consider interpretable-by-design alternatives

C. Have the data scientist proceed and add post-hoc XAI explanations after training

D. Reject deep learning and require an interpretable model

Question 6

During QA/QC of a recommendation model, the team finds that recommendations show measurable demographic bias. What should the PM do?

A. Have the data scientist add a fairness post-processing layer

B. Treat as an ECO IV.2 + Domain I (Task I.3 — bias checks) issue: document the finding, escalate per accountability procedures, engage stakeholders for remediation decision, do not authorize IV.6 GO until bias is within tolerance

C. Deploy with a "monitor closely" flag and address bias in production

D. Reject the entire model and start over

Question 7

A team is iterating a model with 4 training runs over 2 weeks, each one improving slightly but not meeting the success criteria. The data scientist suggests a 5th iteration. What should the PM do?

A. Approve the 5th iteration

B. Pause and review the iteration trajectory: are improvements converging or plateauing? Is the technique a fit? Is the data sufficient? Document a decision: continue, change technique, descope, or escalate.

C. Have the data scientist try a different algorithm

D. Cancel the project

Question 8

During the IV.6 gate review, the team confirms model performance meets success criteria but notes the chosen technique requires GPU compute that isn't available in the planned cloud production environment. What should the PM do?

A. Authorize GO and procure GPU compute concurrently

B. Treat as an operational-fit failure of IV.6 — documented ITERATE outcome. Loop back to V.1 (deployment plan) to address infrastructure OR loop back to IV.1 (technique) to choose a model that fits the planned environment.

C. Have the ML engineer optimize the model for CPU

D. Deploy to GPU on a different cloud while the team works the issue

Question 9

True or False: An AutoML pipeline that automatically selects an algorithm and tunes hyperparameters bypasses the need for ECO Task IV.1 documentation.

Question 10

The team has completed Phase IV. The data scientist asks whether to begin operationalization (Domain V) work in parallel with the IV.6 gate. What should the PM do?

A. Approve parallel work to save time

B. Confirm IV.6 must complete before Domain V work begins; sequential, not parallel; gate authorizes the transition

C. Have the ML engineer prepare deployment artifacts but not actually deploy until IV.6 GO

D. Defer the IV.6 gate and let Domain V work proceed

Memory Aids & Mnemonics Summary

Closing reminders for Domain IV

• Domain IV is gate-heavy. Two of six tasks are explicit gates. Combined with III.8, three of the ECO's gates = ~10-15 exam questions. Master the gate decision pattern.
• The PM holds the gate. When the data scientist wants to keep iterating, the ML engineer wants to keep tuning, the business wants to ship — the PM is the one running the documented decision against documented criteria. Do not let the gate become "rubber-stamped."
• III.8 ≠ IV.5. They look similar; they're not. III.8 = "do we have what we need?" (Phase II close). IV.5 = "is the prepared data sufficient to train?" (Phase III close). Tested directly.
• Operational fit is part of IV.6. A model that performs well but can't run in the planned production environment fails IV.6. Most exam stems test this through "the model needs GPU but the cloud is CPU" or "the model needs real-time but the environment is batch."
• Cross-domain pulls are dense. Domain II success criteria (II.8) feed IV.6's performance check. Domain I (Trustworthy AI) flows through IV.2 QA/QC. Domain V (V.1 deployment plan) is downstream of IV.6 GO. Recognize the pulls in stems.