This is a ready-to-use real-world performance monitoring plan for a deployed AI-enabled device or software as a medical device. Post-market monitoring is the half of the total product lifecycle most likely to be weak, because the work happens after authorization; a monitoring plan with no defined response is decoration. Replace every <<FILL: ...>> placeholder. A worked filled specimen follows. Confirm cited references against the current source. This is general guidance to adapt, not legal or regulatory advice.
Control header
| Field | Entry |
|---|---|
| Plan number | <<FILL: PLAN-ID>> |
| Device / model | <<FILL: name and version>> |
| Intended use | <<FILL: clinical task, population, workflow>> |
| Authorization / pathway | <<FILL: De Novo / 510(k) / PMA / IND-NDA-BLA constituent / CE>> |
| Linked PCCP | <<FILL: PCCP reference or "none">> |
| Owner / date | <<FILL>> |
| QA / Regulatory approval / date | <<FILL>> |
1. Scope and objective
To demonstrate, on a continuing basis, that the deployed model still performs as it did at authorization, and to detect performance degradation, data drift, and unexpected behavior across sites and populations before it harms patients. A model that performed well at authorization is not assumed to keep performing; this plan shows it. Monitoring is live from day one of deployment, not retrofitted.
2. Authorized performance baseline
The reference the monitoring compares against. Record the authorized values with uncertainty, by subgroup.
| Metric | Authorized value (with CI) | By subgroup? |
|---|---|---|
<<FILL: sensitivity>> | <<FILL>> | <<FILL: yes, table attached>> |
<<FILL: specificity>> | <<FILL>> | <<FILL>> |
<<FILL: other clinical metric>> | <<FILL>> | <<FILL>> |
3. What is monitored
| Signal | What it detects | Method | Frequency |
|---|---|---|---|
| Real-world clinical performance vs baseline | Degradation | Labeled sample from routine use, by site and subgroup | <<FILL: monthly / quarterly>> |
| Input data distribution vs training distribution | The model is seeing a population it was not built for (data drift) | Distribution comparison on input features | <<FILL: continuous / weekly>> |
| Override / disagreement rate | Early, cheap signal of degradation (often moves before formal metrics) | Rate at which the clinician’s final decision differs from the model output | <<FILL: continuous>> |
| Complaints, adverse events, use errors tied to the AI function | Harm and use-error signals | Feed from the standard post-market channels | <<FILL: continuous>> |
Report by site and subgroup, not only in aggregate, so a localized failure does not hide in the overall number.
4. Thresholds and the defined response
The response is the point of the plan. For each signal, state the threshold and exactly what happens when it trips: who is notified, whether the model is paused or escalated to fuller human review, and how the event is investigated.
| Signal | Alert threshold | Action threshold | Defined response when action threshold trips |
|---|---|---|---|
| Clinical performance vs baseline | <<FILL: e.g. sensitivity drops by X>> | <<FILL>> | <<FILL: notify [role]; pull labeled sample; assess; escalate to human review / pause per SOP>> |
| Input drift | <<FILL>> | <<FILL>> | <<FILL>> |
| Override rate | <<FILL>> | <<FILL>> | <<FILL>> |
| Subgroup performance | <<FILL: any subgroup drop beyond tolerance>> | <<FILL>> | <<FILL>> |
5. PCCP linkage
Where the device has an authorized PCCP, monitoring and the change plan work together: monitoring detects the degradation, the PCCP provides the pre-authorized, controlled path to correct it. State the link: <<FILL: which monitoring signal maps to which PCCP modification (e.g. a scanner-specific sensitivity drop maps to the PCCP's controlled-extension pathway); when a fix is NOT in the PCCP scope, a new submission is required>>.
6. Roles
| Role | Responsibility |
|---|---|
| Post-market / Quality | Owns the plan, the reviews, and the escalation |
| Data Science | Runs the performance and drift analyses, investigates signals |
| Clinical | Confirms clinical relevance of a signal and the human-oversight impact |
| Regulatory | Decides PCCP-in-scope vs new submission for any corrective change |
| System owner | Executes a controlled update and version stamping |
7. Records and retention
Every monitoring cycle produces a controlled, retained, reviewable record: the metrics, the comparison to baseline, any signal, the response taken, and the disposition. Retention: <<FILL: period>>. The monitoring evidence must be reconstructable end to end.
8. Acceptance criteria for the plan
- Monitoring is live from day one of deployment.
- It covers clinical performance, input drift, and the human-AI interaction, by site and subgroup.
- Thresholds and the defined response are pre-defined, not decided after a signal appears.
- The PCCP linkage (or the new-submission path) is explicit.
- The monitoring evidence is a controlled record, retained and reviewable.
9. References
Good Machine Learning Practice guiding principles (FDA, Health Canada, MHRA, October 2021): monitor deployed models and manage retraining risk. FDA final guidance, Marketing Submission Recommendations for a Predetermined Change Control Plan for AI-Enabled Device Software Functions (final December 2024, reissued August 2025). Applicable post-market surveillance and complaint-handling requirements for your pathway and market.
Confirm the current version of each reference before issue.
10. Revision history
| Version | Date | Author | Summary of change |
|---|---|---|---|
<<FILL: 1.0>> | <<FILL>> | <<FILL>> | Initial issue. |
Filled specimen (excerpt)
Illustrative for a locked chest X-ray pneumothorax triage SaMD deployed across twelve hospitals.
Thresholds and response (excerpt):
| Signal | Alert threshold | Action threshold | Defined response |
|---|---|---|---|
| Sensitivity vs baseline (site-level) | Drop of 3 points | Drop of 5 points or lower CI bound below 0.90 | Notify post-market quality and data science same day; pull a 200-case labeled sample from the site; if confirmed, escalate that site to full radiologist read and open an investigation |
| Input drift (scanner mix) | New scanner share over 15% | New scanner share over 25% or a KL-divergence flag | Confirm sensitivity on that scanner; if a real drop, use the PCCP M3 controlled-extension pathway to retrain and revalidate within pre-authorized criteria |
| Override rate | 10% relative rise over 30 days | 25% relative rise | Investigate; the override rate is treated as a leading indicator, so a rise triggers a performance pull even if the formal metric has not yet moved |
Worked event: three months in, the input-drift monitor flagged one site skewing toward a scanner under-represented in training. Before any complaint arrived, the team pulled a labeled sample, confirmed a sensitivity drop for that scanner, and used the PCCP M3 pathway to retrain and revalidate within the pre-authorized acceptance criteria, deploying with version stamping and updated transparency. The monitor caught it, the PCCP fixed it, no patient harm and no new submission. That is the framework working as intended.
Common findings this plan prevents
- Deploy-and-forget: no monitoring, so the validated state is asserted rather than demonstrated.
- Aggregate-only monitoring that masks a single failing site or subgroup.
- Thresholds defined but no response defined.
- Treating a clear performance signal as a reason to quietly patch the model outside the PCCP, converting a controlled update into an ungoverned change.
How to adapt this plan
- Record the authorized baseline with uncertainty, by subgroup.
- Set thresholds and, for each, the exact response, before launch.
- Wire the monitoring signals to the PCCP modifications they map to, and state the new-submission path for out-of-scope fixes.
- Make every cycle a controlled, retained record.
- Confirm references against the current source before issue.