Domain V: Operationalize AI Solution — Comprehensive Study Guide

Exam weight: 17% of PMI-CPMAI exam (~17 scored questions, of ~20 total)

Score-report framing: ❌ Below Target — PRIORITY 2 for rebuild

Maps to CPMAI methodology phase: Phase VI — Operationalization

Number of ECO tasks: 7 (V.1 through V.7)

Estimated study time: 11 hours

Note from docs/ECO_TASK_REFERENCE.md: the score report flagged Task V.6 (Manage AI solution transition plan) as having no questions on his form. Cover it anyway — the retake form is randomized.

Overview

Domain V is the smallest of the three weak domains by weight (17%) but covers the most distinct business surface: what happens after the model passes the operationalization gate (IV.6) and goes into production. Every task in this domain begins with an oversight verb: manage, oversee, prepare. The PM is responsible for ensuring the deployment plan is built, the deployment is managed, governance is overseen, metrics are monitored, contingencies are planned, transitions are managed, and lessons are captured.

The unifying pattern: Domain V tests whether the project manager understands that an AI project doesn't end at deployment. Models drift. Data shifts. Real-world conditions diverge from training. A project that "deployed and is done" is exactly the failure mode PMI is testing for. The right answers always involve continuous monitoring, governance, contingency, and iteration.

Domain V has no formal go/no-go gate (the model→ops gate is IV.6, in Domain IV). But Domain V questions often test the readiness checklist that comes before deployment can be authorized — and the wrong-answer traps are typically "deploy and assume the team will handle it" or "the data scientist owns post-deployment performance."

Table of Contents

• Module 1: Operationalization Foundations (Lessons 1-4)
• Module 2: The Four AI Technology Environments (Lessons 5-9)
• Module 3: Deployment — How and Where (Lessons 10-20)
• Module 4: Continuous Operations and Life Cycle (Lessons 21-29)
• Module 5: Governance, Monitoring, and Trustworthy AI in Production (Lessons 30-37)
• Module 6: Closeout — Reporting, Limits, and Next Iteration (Lessons 38-40)
• Quick Reference: Model Monitoring Checklist
• Quick Reference: Trustworthy AI in Production
• Cross-Domain Links
• Knowledge Check

Module 1: Operationalization Foundations

Lessons 1-4 | What operationalization means and why it's different from app deployment.

☐ Lesson 1: ECO Task V.1 — Manage Creation of AI Solution Deployment Plan

The PM's first job in Domain V: ensure a deployment plan exists before any model is pushed into production. Deployment planning isn't a check-the-box step — it's a documented artifact answering: how will the model serve predictions, where will it run, who will operate it, what performance is expected, what fails-over, and what monitoring is in place from day one.

The PM does not write the deployment plan alone. The data scientist, ML engineer, platform team, and operations team all contribute. The PM coordinates the contributions, ensures the plan is documented, and ensures stakeholders sign off before deployment begins.

KEY TAKEAWAYS

• The output of V.1 is a documented deployment plan with stakeholder sign-off — not a slide deck.
• The PM coordinates plan creation; the team builds it; stakeholders authorize it.

💡 Memory Aid — Deployment Plan

A deployment plan answers: How served (batch / real-time / microservice / stream), Operation location (on-prem / edge / cloud / hybrid), Performance expected, Escalation path on failure, Monitoring from day one, Stakeholder sign-off.

PM Oversight Angle

• PM owns: Coordinating the cross-team effort to produce a documented deployment plan that covers serving method, location, performance criteria, monitoring, fail-over, and stakeholder authorization.
• Deliverable: AI Solution Deployment Plan — typically a workbook section in CPMAI. Includes serving architecture, environment selection, monitoring approach, ownership, performance baseline, contingency triggers.
• Iteration trigger: Deployment plan reveals an architectural assumption from Domain IV that doesn't fit production constraints → loop back to IV.1 (technique selection) or IV.6 (operationalization gate).
• Escalation trigger: Deployment plan requires infrastructure, security, or regulatory approval beyond the project's authority.
• Wrong-answer trap: "Have the ML engineer deploy the model and write the plan after." Plans precede deployment, not the reverse — production incidents often trace back to "we didn't plan for this."
• Question pattern signal: Stems mentioning "the model is ready to deploy," "the team is preparing for production," "before deployment begins" — testing whether you stop and confirm a plan exists.
• ECO task tag: Domain V, Task 1 — Manage creation of AI solution deployment plan

☐ Lesson 2: The "Inference" Phase of an AI Project

ML projects have a different cycle from traditional software:

Inference is the phase where a trained model is used in production to generate predictions on real-world inputs. Most of Domain V deals with inference — the operational lifetime of the model.

KEY TAKEAWAYS

• ML has two phases (training + inference), not five (design/build/test/deploy/manage).
• Inference is where the operational lifetime begins — and where most production failures surface.

☐ Lesson 3: What Is Operationalization?

Operationalization is "the term AI practitioners use to describe putting machine learning models into real-world environments." Note PMI's distinction:

• Operationalization ≠ deployment. Deployment is the act of placing the model. Operationalization is the broader process of ensuring it works, is monitored, governed, and maintained.

A model can be placed anywhere it needs to provide predictions: in a mobile app, on a server, in the cloud, on a desktop, in a web browser (via JavaScript), on an edge device, or even in a medical imaging system. The location is a deployment-plan decision, not a default.

KEY TAKEAWAYS

• Operationalization is the broader process; deployment is the placement act.
• The placement choice is explicit — anywhere from a mobile app to medical imaging.

☐ Lesson 4: Model Operationalization and Life Cycle Questions

PMI lists the questions a project must answer before it's "successfully operationalized." Two categories:

Operationalization Questions:

1. How will you put the model into operation? (Batch / real-time / microservices / hybrid)
2. Where will you operationalize? (On-prem / cloud / edge / hybrid)
3. How will you manage versioning? (Multiple model versions in production simultaneously)
4. What guidance for developers on model scaffolding?
5. How will you implement GenAI in production?

Model Life Cycle Questions:

1. Approach for life cycle management?
2. Integration with DevOps/DevSecOps?
3. Tools for MLOps?
4. Methods for monitoring (usage, abuse, performance, drift)?
5. How to manage the data life cycle in production?
6. Approaches for governance and security?

The PM doesn't answer these technically — the PM ensures they're answered, documented, and signed off as part of V.1.

KEY TAKEAWAYS

• Two categories of pre-deployment questions: operationalization (how/where/versioning/scaffolding/GenAI) and life cycle (lifecycle/DevOps/MLOps/monitoring/data/governance).
• All questions must be documented and answered before deployment, even if the answer is "iterative — start simple."

Module 1 Summary

The one thing: Operationalization is the broader process; deployment is just the act of placement inside it.

PM's job in this module: Manage creation of a documented deployment plan with stakeholder sign-off. Drafting it precedes IV.6 — the gate judges operational fit against it — but finalization and execution wait for the GO.

Exam signals: Stems asking what must exist before deployment, or where the deployment plan sits relative to the gate.

Trap to watch: Treating the plan as a deliverable produced after the GO. The gate needs something to judge against.

ECO tasks: V.1

Module 2: The Four AI Technology Environments

Lessons 5-9 | The four distinct technology environments AI requires, and why one platform can't do it all.

☐ Lesson 5: The Four AI Environments — Overview

There is no "one platform to rule them all" for AI. Different roles and stages have different requirements. PMI defines four core environments:

1. Model Development and Training Environment — where ML engineers and data scientists train and build models and work with algorithms and data
2. Big Data / Data Engineering Environment — moving, cleaning, preparing, integrating, augmenting, and labeling data
3. Model Scaffolding Environment — everything needed to make the model usable in real-world applications; where application developers, coders, web and mobile developers, and hardware engineers integrate the model into the application
4. Model Operationalization Environment — the IT environment that keeps the model working in its destination (server, cloud, or edge), covering availability and uptime, access control, scalability, and versioning

These four are often treated as analogous because they share many similarities, but they are actually quite different. The PM coordinates whichever combination the project needs.

KEY TAKEAWAYS

• Four environments: Model Development and Training, Big Data/Data Engineering, Model Scaffolding, Model Operationalization.
• No single solution fits all requirements — expect a multi-tool stack, and expect different roles to live in different environments.

💡 Memory Aid (Four Environments)

Development and training (build the model), big data and data engineering (move and prep the data), scaffolding (make the model usable inside an application), operationalization (keep it running where it lives).

☐ Lesson 6: Model Development and Training Environment

Where data scientists build, experiment, and train models. Typically: Jupyter notebooks, ML frameworks (TensorFlow, PyTorch, scikit-learn), GPU compute, experiment tracking tools (MLflow, Weights & Biases), and access to staged training data.

PM concern: ensuring the development environment has the compute, data access, and tooling the data science team needs — coordinated as part of V.1 / aligns with III.4 (workspace).

KEY TAKEAWAYS

• Development environment = build/train tooling (notebooks, frameworks, GPUs, tracking).
• PM coordinates access and capacity, doesn't pick the framework.

☐ Lesson 7: Big Data / Data Engineering Environment

Where data is gathered, cleaned, transformed, and pipelined. Typically: data warehouses, data lakes, ETL/ELT tools (Spark, Airflow, dbt), streaming platforms (Kafka, Kinesis), and data catalogs.

PM concern: ensuring data pipelines are built and operate reliably. Pipeline ownership is a deployment-plan question (V.1) and an ongoing operational concern (V.4).

KEY TAKEAWAYS

• Big data / data engineering environment = data pipelines, warehouses, lakes, streaming.
• Pipeline ownership is named in the deployment plan.

☐ Lesson 8: Model Scaffolding Environment

The "scaffolding" environment is where the model gets integrated into the application or system that consumes it. This is the layer where developers (often without ML backgrounds) interact with the model — APIs, SDKs, client libraries, integration patterns.

PM concern: providing developer guidance on how to consume the model, version it, handle errors, and fall back gracefully. Often overlooked until production incidents reveal that consumers don't know what to do when the model returns unexpected results.

KEY TAKEAWAYS

• Scaffolding environment = integration layer between the model and the app/system that uses it.
• Developer guidance is PM-coordinated, not assumed to exist.

☐ Lesson 9: Model Operationalization Environment

The production environment where the model runs and serves predictions. Typically: model-serving infrastructure (TensorFlow Serving, TorchServe, Seldon, SageMaker, Vertex AI, custom REST APIs), monitoring stack, logging, and version routing.

PM concern: ensuring the operationalization environment matches the deployment plan's choices for serving method (batch/real-time/microservice/stream) and location (on-prem/cloud/edge/hybrid).

KEY TAKEAWAYS

• Operationalization environment = production serving infrastructure + monitoring + logging + version routing.
• It must match the V.1 deployment plan choices.

Module 2 Summary

The one thing: Four environments carry an AI solution: development and training, big data and data engineering, scaffolding, and operationalization.

PM's job in this module: Know which environment owns which work so handoffs and resourcing land in the right place — scaffolding especially, since it is where developers with no ML background connect the model to the app.

Exam signals: Stems asking where a specific piece of work belongs, particularly integration work.

Trap to watch: Collapsing scaffolding into development. Wiring the model into the product is its own environment.

ECO tasks: V.1, V.2

Module 3: Deployment — How and Where

Lessons 10-20 | The how (serving methods) and where (locations) of model deployment.

☐ Lesson 10: ECO Task V.2 — Manage AI Solution Deployment

Deployment is the act of moving the model from staging into production according to the V.1 plan. The PM doesn't push the model — the ML engineer or DevOps team does. The PM manages the deployment — coordinates timing, ensures the plan is followed, surfaces blockers, and confirms success criteria are met before declaring deployment complete.

A subtle but exam-critical point: deployment success is not "the model is running." Deployment success is "the model is running, monitored, performant against the success criteria, and in compliance with governance." The PM declares completion against the full criteria, not the runtime check.

KEY TAKEAWAYS

• The PM manages deployment — coordinates and confirms — but does not push.
• Deployment success = runtime + monitoring + performance + compliance, not just runtime.

PM Oversight Angle

• PM owns: Coordinating execution of the V.1 deployment plan; surfacing blockers; confirming success against documented criteria; declaring deployment complete.
• Deliverable: Deployment Status Report — documented confirmation that the plan was executed, monitoring is live, performance baseline met, governance in place.
• Iteration trigger: Deployment reveals an unmet plan requirement (monitoring gap, performance miss, governance gap) → halt deployment, address, then resume — do not declare complete with gaps.
• Escalation trigger: Deployment failure that cannot be remedied within the plan; rollback decision; security or compliance incident.
• Wrong-answer trap: "Declare deployment successful as soon as the model is serving requests." Misses monitoring, performance, and governance criteria.
• Question pattern signal: Stems mentioning "the model has been deployed," "deployment is in progress," "the team is ready to declare deployment complete."
• ECO task tag: Domain V, Task 2 — Manage AI solution deployment

☐ Lesson 11: Batch Prediction

The model runs on a schedule (e.g., nightly) against a batch of inputs and produces a batch of predictions. Used when:

• Predictions don't need to be real-time.
• Inputs are accumulated over time.
• Cost optimization matters (batch is typically cheaper than real-time).

Examples: monthly customer churn predictions, weekly fraud risk scoring, daily demand forecasts.

KEY TAKEAWAYS

• Batch = scheduled, non-real-time, cost-efficient prediction mode.
• Best fit when inputs accumulate and predictions can wait.

☐ Lesson 12: Microservices for AI

The model is exposed as an API that other services call on demand. Each call returns a prediction synchronously. Used when:

• Predictions are needed on-demand but volume is moderate.
• The model needs to integrate with existing application architecture.
• Different consumers need different SLAs.

Examples: a recommendation API called by the product page, a sentiment-analysis service called by support tooling.

KEY TAKEAWAYS

• Microservices = on-demand API, synchronous, integrates with app architecture.
• Best when predictions are moderate-volume and consumer-driven.

☐ Lesson 13: Real-Time Prediction

The model produces predictions in real time on continuously incoming data, typically with strict latency requirements. Used when:

• Predictions must be returned in milliseconds.
• The application can't function without immediate prediction results.
• Volume is high and latency-sensitive.

Examples: fraud detection at point-of-sale, autonomous vehicle perception, real-time bidding.

KEY TAKEAWAYS

• Real-time = millisecond-latency, high-volume, latency is the success criteria.
• Best when the app fails without immediate predictions.

☐ Lesson 14: Stream Learning

Stream learning continuously updates the model as new data arrives — the model both serves predictions and learns from incoming data simultaneously. Distinct from real-time prediction, which serves predictions but doesn't update the model.

Used when:

• Data distributions change rapidly (concept drift).
• The model must adapt without explicit retraining cycles.
• Examples: recommendation engines with rapidly changing taste, financial trading models, adaptive ad targeting.

KEY TAKEAWAYS

• Stream learning = continuous model update + serving in one stream.
• Different from real-time: real-time serves; stream serves AND learns.

☐ Lesson 15: Cold Path vs Hot Path Analytics

Not unique to AI but shows up in deployment-plan questions:

A typical AI architecture uses both: hot path for immediate response, cold path for retraining and longer-horizon analytics.

💡 Memory Aid — Hot vs Cold

Hot = Hurry (milliseconds, alerts, real-time). Cold = Consider (hours, aggregation, retraining).

KEY TAKEAWAYS

• Hot path = low latency, immediate response.
• Cold path = high latency, aggregation/reporting/retraining.
• Most AI systems run both.

☐ Lesson 16: On-Premises Deployment

The model runs on infrastructure the organization owns and operates. Reasons:

• Data residency / regulatory requirements (some jurisdictions require on-prem).
• Sensitive data that can't leave organizational boundaries.
• Existing on-prem infrastructure investment.
• Latency-critical applications where round-trip to cloud is too slow.

Trade-offs: higher operational burden, harder to scale, more capital expense.

KEY TAKEAWAYS

• On-prem deployment = organizational control, regulatory fit, latency-sensitive.
• Trade-off: higher operational burden, harder scaling, capital expense.

☐ Lesson 17: Edge Device Deployment

The model runs on a device close to the data source — a phone, an IoT sensor, a self-driving vehicle, a medical device. Reasons:

• Latency: predictions can't wait for a cloud round-trip.
• Bandwidth: too much data to send to cloud.
• Privacy: data shouldn't leave the device.
• Connectivity: device may operate offline.

Trade-offs: limited compute and memory, harder to update, model must be smaller.

KEY TAKEAWAYS

• Edge deployment = on the device — phones, IoT, vehicles, medical equipment.
• Trade-offs: constrained compute, harder updates, smaller model footprint.

☐ Lesson 18: Cloud ML Deployment

The model runs in a cloud provider's managed environment — AWS SageMaker, Google Vertex AI, Azure ML, or similar. Reasons:

• Scale on demand.
• Managed infrastructure (less ops burden).
• Integrated tooling (training, serving, monitoring).
• Pay-as-you-go cost model.

Trade-offs: vendor lock-in, data egress costs, regulatory considerations.

KEY TAKEAWAYS

• Cloud deployment = scale, managed services, integrated tools, pay-as-you-go.
• Trade-offs: lock-in, egress, regulatory fit.

☐ Lesson 19: Self-Hosted vs API-Hosted GenAI Models

Specific to GenAI, two production patterns:

The PM coordinates this decision against trustworthy-AI constraints (Domain I) and the deployment plan (V.1).

KEY TAKEAWAYS

• Self-hosted = control, privacy, capital cost.
• API-hosted = speed, scale, vendor terms.
• The choice ties to trustworthy-AI constraints and operational scale.

☐ Lesson 20: Risks of GenAI in Production

PMI treats deploying GenAI into production as its own risk set — these are the Phase VI risks, distinct from the Phase I risks of choosing GenAI at all:

• Output quality needs constant monitoring. Hallucinations are real, and outputs need continuous moderation so they do not go rogue. This matters most when the system is exposed to customers: you cannot set it and forget it; every response requires monitoring and oversight.
• Accuracy and performance can change quickly. Response quality can degrade over time even when you have changed nothing, so monitoring covers both appropriateness and quality.
• API-accessed versions and cost can change quickly. The versions of API models you pay for per request can change, and so can the cost. Getting to a given level of response may mean more interactions, more prompts, or a more expensive model.
• Adversarial prompts and jailbreaks are real and dangerous. Users try to manipulate the system into revealing sensitive information or generating inappropriate content. Ironically, GenAI often requires more humans in the loop than simpler systems.
• You have less control than with cloud-based AI systems. Whether the model is self-hosted or third-party, you do not have as much control over output quality — a risk, cost, and resource issue to plan for.

Mitigations belong in the deployment plan (V.1) and ongoing monitoring (V.4): content filters, prompt validation, output review, escalation paths, audit logging.

Cross-reference — the Phase I risk set. When the decision was whether to use GenAI at all, PMI's five risks were hallucination, IP misappropriation, inappropriate responses, prompt injection, and private data sharing. Domain V questions may reference either set; the production five above are the ones tied to running the system.

KEY TAKEAWAYS

• Production GenAI risks: output quality monitoring, fast-changing accuracy/performance, API version and cost churn, adversarial prompts and jailbreaks, less control than cloud AI.
• The Phase I risk set (hallucination, IP, inappropriate responses, prompt injection, private data) still applies — it just answers a different question.
• Mitigations are deployment-plan items (V.1) and ongoing monitoring (V.4), not afterthoughts.

Module 3 Summary

The one thing: Serving method and deployment location are two separate decisions, and both belong in the deployment plan.

PM's job in this module: Manage the deployment — coordinate and confirm — without performing it. Serving method drives latency expectations and monitoring design; location drives control, compliance, and cost.

Exam signals: Latency requirements, regulatory constraints on where data may sit, and any stem weighing self-hosted against API-hosted GenAI.

Trap to watch: Mixing the two axes — offering edge as a serving method, or batch as a location.

ECO tasks: V.2

Module 4: Continuous Operations and Life Cycle

Lessons 21-29 | Why deployment isn't done — managing ongoing operation, drift, and contingency.

☐ Lesson 21: Failure Reason — AI Life Cycles Are Continuous

A common AI project failure: organizations treat the model as a "one-and-done" deliverable. They deploy, declare success, and move on. Six months later, the model is producing degraded predictions — and nobody is monitoring.

PMI's framing: AI project life cycles are continuous. Models drift. Data shifts. The world changes. The deployment plan must include continuous management, monitoring, and iteration — or the project is failing on a delayed timer.

KEY TAKEAWAYS

• AI projects don't end at deployment — they continue through monitoring, drift detection, and iteration.
• "Deployed and done" is a project failure mode on a delay.

☐ Lesson 22: AI Life Cycle Challenges — COVID-19 E-Commerce Example

PMI's classic real-life example: e-commerce demand forecasting models trained pre-2020 dramatically failed during COVID-19. Consumer behavior shifted overnight; training data no longer reflected reality.

The lesson: external shocks invalidate model assumptions. The deployment plan must include monitoring for distribution shift and a contingency plan for retraining or rollback when shift is detected.

KEY TAKEAWAYS

• External shocks can invalidate model assumptions overnight (COVID-19 / e-commerce).
• Monitoring for distribution shift + contingency plan = required, not optional.

☐ Lesson 23: ECO Task V.4 — Oversee AI solution metrics (e.g., KPI, model performance)

The PM oversees metrics that track AI solution health in production. Metrics include:

• Business KPIs — what the project was supposed to deliver (revenue, cost reduction, accuracy of operational decisions).
• Model performance — accuracy, precision, recall, F1, latency, throughput, drift indicators.
• Operational metrics — uptime, error rate, request volume, prediction-to-action conversion.

Metrics must tie back to the success criteria from Domain II (Task II.8). Without that linkage, you're measuring without baseline.

KEY TAKEAWAYS

• Metrics span business KPIs, model performance, operational health.
• Metrics must link to Domain II success criteria for context.

💡 Memory Aid — Metrics

Business KPIs, Model performance, Operational health. Three tiers, all monitored, all linked to Phase I/II success criteria.

PM Oversight Angle

• PM owns: Overseeing the metrics regime — ensuring metrics are defined, instrumented, monitored, and reviewed against success criteria.
• Deliverable: Metrics Plan + dashboards / reports per the deployment plan. Reviewed cadence (weekly/monthly), with stakeholder access.
• Iteration trigger: Metrics consistently below threshold → loop back to investigate root cause (data drift, model decay, scope miss). May trigger retraining or rescoping.
• Escalation trigger: Sustained metric breach affecting business KPIs; trustworthy-AI metric breach (bias, privacy incident); SLA violation.
• Wrong-answer trap: "Have the data scientist watch model performance and let the PM know if there's an issue." Metric ownership is PM-driven, not data-scientist-on-call.
• Question pattern signal: Stems mentioning "model performance has degraded," "the business KPIs are below target," "the metrics show drift."
• ECO task tag: Domain V, Task 4 — Oversee AI solution metrics

☐ Lesson 24: Model Life Cycle Management

Model life cycle management covers the model's entire production lifetime: deployment, monitoring, versioning, retraining, retirement. The PM coordinates the cadence, owners, and triggers for each:

• Deployment — V.2 task, executed via plan.
• Monitoring — V.4 task, ongoing.
• Versioning — version control of model artifacts, training data, and serving infrastructure.
• Retraining — triggered by drift detection or scheduled refreshes.
• Retirement — sunset old versions; document the replacement.

KEY TAKEAWAYS

• 5 life cycle phases: deployment, monitoring, versioning, retraining, retirement.
• Each has owners, triggers, and documentation requirements.

☐ Lesson 25: Managing the Data Life Cycle (in Production)

Data drift — the production data slowly diverging from training data — is a primary cause of model decay. The PM coordinates:

• Monitoring data inputs for distribution shift.
• Updating data pipelines as new data types or formats appear.
• Adapting data prep to changes.
• Triggering retraining when shift exceeds tolerance.

This is the production extension of the data life cycle you learned in Domain III (Lesson 29).

KEY TAKEAWAYS

• Data drift = production data diverging from training distribution.
• Monitoring + adaptation + retraining triggers = continuous data life cycle in production.

☐ Lesson 26: ECO Task V.6 — Manage AI Solution Transition Plan

Transition plans cover handoffs: model handed from project team to operations team, model handed from one operations team to another (reorg, vendor change), or AI solution transitioned to a successor project. The PM owns the transition plan — what's documented, what's transferred, what knowledge needs to move with the artifact.

Asterisked task: the first attempt form had no V.6 questions. The retake form may or may not. Cover it.

KEY TAKEAWAYS

• Transition plan = handoff documentation + knowledge transfer + ownership change.
• Often triggered by team change, vendor change, or successor project.

PM Oversight Angle

• PM owns: Producing a transition plan documenting handoff, ownership, knowledge transfer, and ongoing maintenance arrangements.
• Deliverable: AI Solution Transition Plan — recipient teams, documented artifacts, training/onboarding, escalation paths, sign-off from receiving team.
• Iteration trigger: Transition reveals undocumented dependencies → produce documentation, do not transfer until complete.
• Escalation trigger: No receiving team identified or willing; capability gap that prevents safe transition.
• Wrong-answer trap: "Email the operations team and let them figure it out." Transition is documented, signed off, and the receiving team confirms readiness.
• Question pattern signal: Stems mentioning "the project team is wrapping up," "the model is being handed to operations," "the vendor contract is ending and we need to bring it in-house."
• ECO task tag: Domain V, Task 6 — Manage AI solution transition plan

☐ Lesson 27: DevOps for AI

DevOps integrates development with operations — continuous integration, continuous deployment, monitoring, automation. For AI projects, DevOps practices apply but with added complexity from model artifacts and data dependencies.

DevSecOps adds security as a first-class concern. AI projects often need DevSecOps because of regulated data and trustworthy-AI requirements.

KEY TAKEAWAYS

• DevOps = CI/CD/monitoring/automation for software.
• DevSecOps adds security as first-class.
• AI projects benefit from DevOps practices, with extensions for model and data artifacts.

☐ Lesson 28: MLOps — Machine Learning Operations

MLOps is DevOps adapted for ML. It addresses:

• Model versioning and lineage — tracking which model version was trained on which data with which hyperparameters.
• Reproducibility — being able to recreate any production model.
• Continuous training — automated retraining pipelines.
• Continuous deployment — automated rollout of new model versions.
• Monitoring — model and data drift detection, performance metrics.

The PM doesn't build the MLOps stack but ensures the project plan accounts for it as a capability the operations team must have.

💡 Memory Aid — MLOps vs DevOps

DevOps = code-driven CI/CD. MLOps = code + model + data CI/CD. MLOps adds model versioning, data lineage, retraining automation.

KEY TAKEAWAYS

• MLOps extends DevOps with model + data + retraining + drift monitoring.
• The PM ensures the deployment plan accounts for MLOps capability.

☐ Lesson 29: ECO Task V.7 — Oversee AI Solution Contingency Plan

A contingency plan documents what happens when things go wrong: the model breaks, predictions are unreliable, an incident occurs, a data feed fails. The PM oversees creation of the plan and ensures it's tested and ready before production goes live.

Common contingency scenarios:

• Model failure → fallback to previous version, rule-based system, or human handoff.
• Data feed failure → alternative source, cached predictions, graceful degradation.
• Performance breach → automated rollback or pause.
• Trustworthy-AI incident → containment, audit, stakeholder notification, regulatory reporting.

KEY TAKEAWAYS

• Contingency plans = what to do when AI breaks.
• Tested and ready before production, not improvised after.

PM Oversight Angle

• PM owns: Overseeing creation of the contingency plan covering model failure, data feed failure, performance breach, trustworthy-AI incidents.
• Deliverable: AI Solution Contingency Plan — scenarios, triggers, response procedures, owners, escalation paths, tested.
• Iteration trigger: Contingency plan reveals an unmitigated risk → loop back to V.1 deployment plan to address before production.
• Escalation trigger: A scenario that exceeds the project's mitigation capability and requires external dependency or executive decision.
• Wrong-answer trap: "Document the contingencies and trust the operations team to respond." Plans are tested, not just documented, before production.
• Question pattern signal: Stems mentioning "the model has stopped working," "predictions are unreliable," "a data feed has failed," "an incident has occurred."
• ECO task tag: Domain V, Task 7 — Oversee AI solution contingency plan

Module 4 Summary

The one thing: AI life cycles are continuous: deployment is the start of operations, not the end of the project.

PM's job in this module: Oversee metrics across business KPIs, model performance, and operational health; manage the transition plan so ownership actually changes hands; keep a contingency plan for when the AI fails.

Exam signals: External shocks that invalidate model assumptions, handoffs to an operations team, and any stem asking what happens when the model degrades.

Trap to watch: Closing the project at go-live. Also treating the transition plan as documentation only — it is a change of ownership.

ECO tasks: V.4, V.6, V.7

Module 5: Governance, Monitoring, and Trustworthy AI in Production

Lessons 30-37 | The governance, monitoring, and trustworthy-AI controls that run continuously in production.

☐ Lesson 30: ECO Task V.3 — Oversee Model Governance

Model governance "provides controls, processes, procedures, and organizational guidance on how models are built, iterated, used, and shared." The PM oversees governance throughout production — ensuring it's defined, applied, and audited.

Model governance covers:

• Access control, authorization, security — who can use which model, with what authorization.
• Provenance and auditing — documentation of how the model was trained, tested, deployed.
• Audit logs — usage and iteration history.
• Version control — supporting multiple model versions in production simultaneously.
• Bias monitoring — measuring and addressing informational bias.
• Sharing and extension controls — governing how others use, fine-tune, or extend the model.

KEY TAKEAWAYS

• Model governance = controls + processes + procedures + organizational guidance for production models.
• 6 components: access, provenance/auditing, audit logs, versioning, bias monitoring, extension controls.

💡 Memory Aid — Governance

Access control, Provenance/auditing, Audit logs, Version control, Bias monitoring, Extension controls.

PM Oversight Angle

• PM owns: Overseeing the model governance program — ensuring governance is defined, applied, audited; coordinating governance with Trustworthy AI Framework (Domain I) and organizational policy.
• Deliverable: Model Governance Plan + ongoing governance reports (audits, access reviews, version logs).
• Iteration trigger: Governance gap detected (unauthorized access, version drift, missing audit) → halt, address, audit before resuming.
• Escalation trigger: Governance incident (bias breach, security incident, audit failure); regulatory inquiry.
• Wrong-answer trap: "Have the ML engineer maintain governance documentation." Governance is PM-overseen, organizationally-aligned, not engineer-maintained.
• Question pattern signal: Stems mentioning "an updated model was deployed," "access to the model is being granted," "the model needs an audit," "different versions of the model exist in production."
• ECO task tag: Domain V, Task 3 — Oversee model governance

☐ Lesson 31: Model Deployment with Governance Framework

Why governance is needed during deployment and beyond:

• Models change. Even if the model itself doesn't, the world, data, and environment do. Models are versioned and iterated continuously.
• Thresholds shift. Initially-acceptable accuracy (e.g., 92%) may rise (95%).
• Sensitivity changes. False-positive vs false-negative tolerance evolves.
• Use cases expand. A batch model may need real-time. An old API may need new capabilities.
• External users emerge. Other teams may extend or fine-tune your model in unintended ways.

The governance framework includes: model versioning, continuous evaluation, deployment management, iteration controls, A/B testing, access control, security, extension controls.

KEY TAKEAWAYS

• Models need governance because models change, thresholds shift, use cases expand, external users emerge.
• Governance framework spans versioning, evaluation, deployment, iteration, A/B, access, security, extension.

☐ Lesson 32: Model Monitoring

Model monitoring verifies that the production model performs as expected. Key aspects:

• Performance measures — latency, response time, request volume, prediction errors, accuracy, recall, F1, model-specific ML measures.
• Data visibility — monitoring inputs to detect drift; logging failures.
• Usage visibility — how is the model actually being used? Useful for training future versions.
• Drift measurement — both model drift (model performance degrading) and data drift (input data distribution shifting).
• Multi-dimensional dashboards — performance across user segments, time periods, geographies.

Model drift and data drift are inevitable, not exceptional. Monitoring quantifies how much.

💡 Memory Aid — Model Drift vs Data Drift

Model drift = the model's predictions degrade over time. Data drift = the inputs to the model shift over time. Drift is inevitable; monitoring is the response.

KEY TAKEAWAYS

• Monitoring covers performance, data, usage, drift, dimensional analysis.
• Both drifts are inevitable — monitoring quantifies them.

☐ Lesson 33: Trustworthy AI Considerations in Production

Production AI must be:

• Compliant — regulatory frameworks and legal requirements in operating jurisdictions.
• Safe, Reliable, Secure — protected from malicious AI; doesn't cause harm.
• Ethical — addresses ethical and trustworthy concerns.
• Privacy-respecting — data privacy and security maintained.

This pulls heavily on Domain I (Trustworthy AI). Production is where Domain I requirements are actually enforced.

KEY TAKEAWAYS

• Production trustworthy-AI = compliant, safe/reliable/secure, ethical, privacy-respecting.
• Production is where Domain I is enforced, not theorized.

☐ Lesson 34: AI System Safety and Reliability

Two distinct properties:

• Safety — AI must not endanger humans (through neglect or carelessness).
• Reliability — AI must operate as intended throughout its life cycle (software or hardware).

Both require: defined acceptable boundaries, fallback plans for when boundaries are crossed, resilience to operational failures.

PMI's example issue table (autonomous vehicles, facial recognition, content moderation) shows specific failure modes and mitigations — the PM understands these are project-level risks, not just technical concerns.

KEY TAKEAWAYS

• Safety = doesn't endanger. Reliability = operates as intended.
• Both require defined boundaries + fallback plans + resilience.

☐ Lesson 35: Malicious AI

Malicious AI is the intentional use of AI for criminal, unethical, dangerous, or harmful purposes. Two main categories:

• Cyberthreats — attacks via intelligent bots, model poisoning, automated zero-day discovery.
• Physical threats — autonomous-vehicle hijacking, voice-assistant social engineering, physical denial-of-service.

Specific patterns:

• Adversarial attacks — manipulating input data to deceive ML models (e.g., turtle classified as rifle).
• Model poisoning — corrupting training data so the model learns wrong patterns.
• Deepfakes — AI-generated fake media for fraud, public-figure impersonation, social engineering.
• AI-powered misinformation — generating and spreading fake content.

The PM coordinates defense via the contingency plan (V.7), monitoring (V.4), and trustworthy-AI framework (Domain I).

KEY TAKEAWAYS

• Malicious AI = cyberthreats + physical threats.
• Patterns: adversarial attacks, model poisoning, deepfakes, AI misinformation.
• Defense is layered across V.7 contingency, V.4 monitoring, Domain I framework.

☐ Lesson 36: Securing Machine Learning Models

PMI names three key factors to consider when securing ML models:

• Model Security — secure the models to prevent tampering. Especially important when you have multiple models and versions. Implementing it means installing access control mechanisms that continuously audit who, when, where, why, and how much a model is accessed; protecting the model itself along with its training data and settings; conducting vulnerability analysis and reporting significant changes to data or operations that impact performance; sanitizing data inputs regularly (remove what the model does not need); anonymizing data to enforce privacy; and securing APIs along with the resources and data they reach.
• Access Control — implement controls so only authorized individuals can access and modify the models. Access is both internal and external: secure it against outsiders intending harm and against employees or contractors attempting actions they are not authorized to perform.
• Backup Plans — have backup plans in place to restore models in case of accidental overrides or deletions, which is what keeps the system continuous and reliable.

Model metadata, configurations, and settings need the same protection as the models themselves — only the necessary personnel should be able to change them. These steps look basic but are challenging to implement in practice, and each one has to be checked against the regulations that apply. Security is part of governance (V.3) and trustworthy-AI (Domain I), enforced through monitoring (V.4) and contingency (V.7).

KEY TAKEAWAYS

• Three factors: model security, access control, backup plans. Backup plans are the one people forget.
• Model security includes access control mechanisms, protecting the model and its training data and settings, vulnerability analysis, data sanitization, anonymization, and API security.
• Metadata, configurations, and settings are as important as the models — secure them too.

☐ Lesson 37: Resolving Issues of Ethical and Trustworthy AI

Everything is probabilistic with AI and ML — including the probability that your project hits an ethical or trustworthy-AI issue, no matter how simple it is. Publishing a statement is not enough; what matters is taking affirmative action. PMI says issues of trustworthy AI are resolved three ways:

• Keep a human in the loop. As long as humans are part of the decision-making loop for all AI systems, you resolve many of the ethical, responsible, transparent, governed, and explainable AI issues that naturally occur. PMI's examples of what human oversight prevents or resolves: runaway systems (AI taking over beyond human control), inconsistencies (humans correct AI mistakes), ambiguities (humans guide AI when it lacks understanding), probabilistic issues (humans decode cases where AI cannot give confident results), cybersecurity threats (humans have intuition AI lacks and sense threats it misses), and physical threats (machines and robots are still not as nimble as humans and are easy to tip over).
• Represent AI systems with a human. A human should be the point of contact for customers. Do not delegate authority and control solely to an algorithm, ML or rules-based. People have been shut out of systems, denied physical access, hired unfairly, and had loan applications rejected because decisions were left to algorithms.
• Prioritize fairness and dignity. PMI's example: do not replace manual labor with an automated system before talking with your workforce. Explain why — cost, time, easing the burden — to gain buy-in. Skip the dignity and you get resistance, and you also miss what the workers closest to the process could have told you about your approach.

The Trustworthy AI Framework (Domain I) provides the structural reference; Domain V is where execution happens.

KEY TAKEAWAYS

• Three resolutions: human in the loop, a human representing the AI system, fairness and dignity prioritized.
• Human-in-the-loop prevents runaway systems, inconsistencies, ambiguities, probabilistic issues, cybersecurity threats, and physical threats.
• Trustworthy AI Framework = reference. Domain V = execution.

Module 5 Summary

The one thing: Model governance is the control structure around a production model, and human oversight is the foundation of resolving trustworthy-AI issues.

PM's job in this module: Oversee governance and monitoring — performance, data, usage, drift — and keep security, access control, and backup plans in scope.

Exam signals: Production incidents, expanding use cases, shifting thresholds, and any stem about resolving an ethics or fairness problem after launch.

Trap to watch: Reaching for an automated guardrail or an external audit when the answer is keeping a human in the loop.

ECO tasks: V.3, with pulls to I.1–I.5

Module 6: Closeout — Reporting, Limits, and Next Iteration

Lessons 38-40 | Closing the project loop with lessons learned, recognizing AI limits, and preparing for the next iteration.

☐ Lesson 38: ECO Task V.5 — Prepare final report/lessons learned

The final Domain V deliverable: a formal report capturing what was built, what was learned, what worked, what didn't, and what's recommended for future iterations.

Required content per CPMAI methodology:

• Project summary — what was built and delivered.
• Performance against success criteria — how the project measured against Domain II success criteria.
• Lessons learned — what worked, what didn't, what to do differently.
• Recommendations — for next iterations, related projects, organizational learning.
• Outstanding risks and dependencies — what remains for ongoing operations.

Lessons learned are not "the end" — they're input to the next iteration's Phase I. CPMAI is iterative; the final report is the bridge.

KEY TAKEAWAYS

• Final report = project summary, performance vs criteria, lessons learned, recommendations, outstanding risks.
• Lessons learned feed the next iteration's Phase I, not the trash.

PM Oversight Angle

• PM owns: Producing the final report and lessons learned, ensuring it's reviewed by stakeholders and contributes to organizational AI learning.
• Deliverable: Final Report / Lessons Learned document — typically a workbook section in CPMAI plus a stakeholder briefing.
• Iteration trigger: Lessons learned reveal a fundamental scope or strategy issue → input to next iteration's Phase I.
• Escalation trigger: Lessons learned reveal organizational learning gaps that need program-level attention.
• Wrong-answer trap: "Send the deployment status report and call it the final report." Final report is broader — includes lessons learned, outstanding risks, recommendations.
• Question pattern signal: Stems mentioning "the project is wrapping up," "the deployment is complete," "the team is preparing closeout."
• ECO task tag: Domain V, Task 5 — Prepare final report / lessons learned

☐ Lesson 39: The Limits of AI Technology

Even with successful operationalization, current AI and cognitive technologies still fall short of complex, real-world needs. PMI frames the limits three ways:

• We are at the knowledge level of the DIKUW pyramid (data, information, knowledge, understanding, wisdom). We are trying to gain deeper insights from data but have not fully understood that knowledge, and we are nowhere near the wisdom stage of making smart decisions. Machines generate text and process voice, but they do not truly understand what they are doing.
• The Intelligence Gap. AI-powered machines look intelligent but cannot do everything a human can — PMI's illustration is that they cannot even do what a dragonfly does with a brain the size of a pinhead: fly, maneuver, perceive the world to find food, react with precision, and evade predators. Humans have innate cognitive abilities, creativity, empathy, common sense, and reasoning grounded in perception. Replicating that is far off.
• Machine learning is narrow — task specific and data set specific. ML systems are specific to the problem type (image classification, natural language processing) and even to particular kinds of images or language problems. Solving a new task means designing new classifiers, collecting training data, training models, or tuning existing ones. A computer vision system cannot drive a car; a movie recommendation system cannot do anything else. We have three main approaches — supervised, unsupervised, reinforcement — which itself shows we do not fully understand the human brain.
• No artificial general intelligence. Transfer learning and foundation models give strong starting points, and general large language models may soon handle a wide range of natural language tasks, but they still fall short of AGI that can handle any intelligence task. For each new task you still prepare data, train, iterate, and evaluate — which is exactly why you follow CPMAI. There is no shortcut.

Recognizing limits is a Phase VI competency: it informs lessons learned, recommendations for the next iteration, and decisions about expanding or contracting the AI program.

KEY TAKEAWAYS

• We sit at the knowledge level of the DIKUW pyramid — not understanding, and not wisdom.
• The Intelligence Gap — machines still cannot match a dragonfly, let alone a human.
• Everything is narrow AI — task specific and data set specific; no AGI, and no shortcut around CPMAI.
• Recognition of limits is a Phase VI competency, not a defeatist stance.

☐ Lesson 40: Phase VI Go/No-Go — Ready for Next Iteration

CPMAI is iterative. Phase VI ends not with "the project is done" but with "what's the next iteration?" — and it uses the same hard-gate structure as the earlier phase gates. You are ready for the next iteration when:

• You have operationalized a model that meets all of the expected requirements.
• You have a model that functions as needed.
• You have a model that continues to perform as required, for as long as it is operational.
• You have adequate responses for the Phase VI workbook items — a well-defined answer for critical questions, and a good understanding of what is known and what is not known.

PMI is explicit about the stopping conditions: do not progress if you do not have any answers — positive or negative — to key Phase VI questions, and do not progress if you have roadblocks or warning signs that will prevent the project from being successful. You should be able to fully meet the requirements before moving forward to the next iteration.

Note what the third criterion demands: not "it worked at launch" but continues to perform. This is the gate that makes monitoring (V.4) and contingency (V.7) non-optional. Once the gate passes, the outcome feeds back into Phase I (Business Understanding) for the next iteration.

KEY TAKEAWAYS

• Phase VI is a hard gate, structured like the earlier gates — not a soft recommendation.
• Ready when: requirements met, functions as needed, continues to perform, workbook responses adequate.
• Do not progress with roadblocks or warning signs, or with no answers to key Phase VI questions.
• CPMAI's iterative design: Phase VI → next iteration's Phase I.

Module 6 Summary

The one thing: Phase VI closes with a hard gate: is the solution ready for the next iteration?

PM's job in this module: Prepare the final report and lessons learned — performance against the criteria set in II.8, outstanding risks, and recommendations.

Exam signals: Stems about what belongs in a final report, or whether a contingency that worked should still be documented.

Trap to watch: Treating Phase VI closeout as a soft recommendation. It is structured like the earlier gates.

ECO tasks: V.5

Quick Reference: Model Monitoring Checklist

Quick Reference: Trustworthy AI in Production

Cross-Domain Links

• V.1 (Deployment Plan) ↔ Domain IV (Tasks IV.5, IV.6 — Gates): Deployment plan responds to the operationalization gate. Plan is the bridge from "ready for ops" to "running in ops."
• V.3 (Model Governance) ↔ Domain I (all 5 tasks): Privacy plan (I.1), transparency (I.2), bias checks (I.3), regulatory compliance (I.4), accountability (I.5) all enforce in production via V.3.
• V.4 (Solution Metrics) ↔ Domain II (Task II.8 — Success Criteria): Production metrics tie back to the success criteria from Domain II. Without that linkage, metrics are abstract.
• V.5 (Lessons Learned) ↔ Phase I (next iteration): Final report's lessons learned become input to the next iteration's Domain II business needs.
• V.7 (Contingency Plan) ↔ Domain I + Domain V tasks: Contingency covers trustworthy-AI incidents (Domain I), data feed failure (V.4 monitoring), model failure (V.4 + V.3).
• V.6 (Transition Plan) ↔ V.1 (Deployment Plan): Transition responds to changes in the deployment plan (new ops team, new vendor, new scope).

Knowledge Check

### Question 1

The data scientist informs the PM that the model is ready to deploy and asks when production rollout can begin. The deployment plan has not yet been created. What should the PM do?

A. Approve deployment and have the team document the plan in parallel

B. Pause and coordinate creation of the deployment plan before deployment begins, including stakeholder sign-off

C. Have the ML engineer deploy to a staging environment while the plan is written

D. Schedule the deployment and have the data scientist write the plan that day

### Question 2

A model has been running in production for 3 months. The data scientist notices that prediction accuracy has degraded by 8 percentage points. What should the PM do?

A. Have the data scientist retrain the model immediately

B. Investigate root cause (data drift, model decay, scope shift), engage stakeholders, and decide between retraining, rolling back, or rescoping per the contingency plan

C. Continue monitoring for another month to see if it's a temporary issue

D. Roll back to the previous version

### Question 3

A PM is told that the operationalization environment is ready and the model can be pushed at any time. Which is the BEST sequence?

A. Deploy → monitor → declare success

B. Deploy → declare success

C. Verify deployment plan executed → confirm monitoring is live → confirm performance baseline met → confirm governance in place → declare deployment complete

D. Deploy → wait 30 days → declare success

### Question 4

True or False: Model drift and data drift are exceptional events that indicate the deployment was flawed.

### Question 5

During a routine governance review, the PM discovers that an updated model version was deployed last month without going through change control. What is the BEST response?

A. Document the deployment retroactively to close the gap

B. Treat as a governance and accountability incident: validate the deployed version against requirements, document the deviation, escalate per accountability procedures, and reinforce change control

C. Roll back to the previous version immediately

D. Note it for next quarter's governance review

### Question 6

The PM is finalizing the deployment plan and is asked whether to pre-define a contingency plan for model failure. What's the BEST PM response?

A. Defer contingency planning until after deployment and observe production for actual failure modes

B. Create the contingency plan now; test it pre-production; ensure response procedures, owners, and escalation paths are documented before going live

C. Have the operations team handle contingencies as they arise

D. Skip contingency planning since modern monitoring tools auto-detect failures

### Question 7

A model has been running successfully for 6 months. A new vendor will take over operations of the model from the current team. What should the PM do?

A. Email the operations team and let them figure out the handoff

B. Coordinate a transition plan including documented artifacts, knowledge transfer, training, escalation paths, and sign-off from the receiving team

C. Have the data scientist train the new vendor's team on the model

D. Have the new vendor inherit the model as-is and start fresh on operations

### Question 8

True or False: Lessons learned from a completed AI project are filed away for organizational record but don't influence ongoing or future projects.

### Question 9

A regulator inquires about how an AI-driven decision was made for a specific customer. The PM is asked to provide documentation. What's the BEST response?

A. Have the data scientist explain the model architecture

B. Provide the audit trail from model governance: input, model version, prediction, timestamp, decision rationale, and any human-in-the-loop overrides — sourced from the documented governance program

C. Decline since model decisions are confidential

D. Re-run the prediction and provide the new result

### Question 10

The team has deployed an AI model and is monitoring its performance. After 3 months, the model is still meeting performance targets. The PM is asked whether to declare the project complete. What's the BEST response?

A. Declare complete since performance targets are being met

B. Continue monitoring indefinitely — projects don't complete

C. Convene Phase VI close-out: document final report and lessons learned, decide on transition to operations or next iteration, capture outstanding risks and dependencies, formal stakeholder sign-off

D. Have the operations team take over and let them decide when to declare complete

Closing reminders for Domain V

• AI projects don't end at deployment. "Deployed and done" is a project failure mode on a delay. Continuous monitoring, governance, and contingency are the methodology.
• The PM doesn't push the model. The PM ensures the deployment plan exists, the deployment is managed against it, governance is in place, metrics are monitored, and contingencies are tested.
• Domain V cross-domain links to Domain I are heavy. Trustworthy AI principles enforce in production. When a stem mentions privacy breach, malicious AI, regulatory inquiry, or bias incident — Domain I + Domain V together.
• Drift is inevitable. Model drift and data drift are guaranteed; monitoring is the response. Stems that frame drift as a "surprise" are testing whether you treat it as exceptional (wrong) or expected (right).

Next: domain-IV-model-dev-eval.md (PRIORITY 3 — Domain IV has the two highest-density gates: IV.5 and IV.6)