This is a ready-to-use checklist for the oversight a vendor-supplied AI system needs after go-live, which a standard supplier does not. An AI product can change behavior with no version change and no action on your part, through vendor model updates or through drift in the data the model sees, so “deploy and forget” is the most common post-go-live failure. Run this on your defined cadence and on every notified change. Mark Pass / Fail / NA and act on every Fail. Replace every <<FILL: ...>> placeholder. This content is educational and general; adapt it and verify it before use.
Header
| Field | Entry |
|---|---|
| System / model | <<FILL>> |
| Supplier | <<FILL>> |
| Intended use / risk class | <<FILL>> |
| Review period | <<FILL: from>> to <<FILL: to>> |
| Reviewer | <<FILL>> |
| Review date | <<FILL>> |
Section A: Change-notification processing
| # | Item | Pass / Fail / NA | Notes |
|---|---|---|---|
| A1 | Every vendor model-change notification in the period was received and logged | ||
| A2 | Each notified change was opened as a change-control record with an intended-use impact assessment | ||
| A3 | The confirmatory testing the predetermined change plan requires was run on the locked test set | ||
| A4 | The change was approved for GxP use before it went live, not after | ||
| A5 | Any change that failed confirmatory testing was held, escalated, or the version pinned |
Section B: Performance and drift monitoring
| # | Item | Pass / Fail / NA | Notes |
|---|---|---|---|
| B1 | Performance was verified against the acceptance criteria on representative production samples on the defined cadence | ||
| B2 | The human override rate was tracked continuously as a leading indicator | ||
| B3 | A defined action was taken when a monitoring threshold tripped (pause, route to fuller review, investigate) | ||
| B4 | Input-distribution or scope changes (a new product line, a new site, a new data source the model was not trained for) were watched for | ||
| B5 | For an API model the vendor can change without notice, compensating monitoring was in place and running |
Section C: Periodic supplier re-evaluation
| # | Item | Pass / Fail / NA | Notes |
|---|---|---|---|
| C1 | The supplier was re-evaluated on the risk-based cadence and after any significant vendor or product change | ||
| C2 | Any quality issues, and any change to the supplier’s development or change-management process, were assessed | ||
| C3 | The supplier’s current security and data-handling posture (including data reuse) was re-confirmed | ||
| C4 | The supplier’s performance evidence is still current for the version in use |
Section D: Incident handling and records
| # | Item | Pass / Fail / NA | Notes |
|---|---|---|---|
| D1 | A working channel exists for the supplier to report a model defect or performance problem | ||
| D2 | Every supplier-reported incident in the period had a defined customer response | ||
| D3 | Monitoring, change, and re-evaluation records are retained and reviewable as the evidence the validated state still holds |
Overall disposition
| Field | Entry |
|---|---|
| Any item Fail? | Yes / No |
| Validated state | Maintained / Action required |
| Actions and owners | <<FILL>> |
| Reviewer signoff (name, date) | <<FILL>> |
| QA signoff (name, date) | <<FILL>> |
Do not assert a maintained validated state while any item is Fail; record the action and re-review.
References
EU GMP Annex 11 (computerised systems; keeping systems in a validated state). ICH Q9(R1), Quality Risk Management (risk-based re-evaluation cadence). GAMP 5 (Second Edition) and the FDA Computer Software Assurance approach, for sizing oversight to intended use and risk (reference by title; do not reproduce their text).
Confirm the current version of each reference before use.
Filled specimen
The following shows selected items completed for a complaint-classification SaaS model over one quarter. Illustrative content; replace with your own.
| # | Item | Result | Notes |
|---|---|---|---|
| A2 | Notified change opened as change control | Pass | Month 7 base-model update; CC-2026-204 opened with impact assessment |
| A3 | Confirmatory testing run | Pass | Recall held; precision dropped slightly, judged acceptable for the screening use, deploy with a watch note |
| B2 | Override rate tracked | Pass | Rose sharply in month 9 with no vendor notice |
| B3 | Action on threshold trip | Pass | Pulled a labeled sample; confirmed degradation on a new product line; routed those complaints to full human review; opened supplier discussion |
| C1 | Supplier re-evaluated | Pass | Annual re-evaluation completed; no change to the vendor’s change-management process |
In this example both the notified change and a later silent drift were caught, one by change control and one by override-rate monitoring, because oversight was running. Neither was caught by the qualification done at go-live; ongoing oversight is what kept the system defensible.
Common inspection findings this checklist prevents
- Deploy and forget: no monitoring after go-live, so the validated state is asserted, not demonstrated.
- Vendor model changes accepted with no impact assessment or confirmatory testing.
- No periodic supplier re-evaluation after the initial qualification.
- No defined response when monitoring trips, so the monitoring is decoration.
- For an API model, no compensating monitoring where the vendor can change the model without notice.
How to adapt this checklist
- Set the cadence and the sampling approach for your risk class.
- Define the specific thresholds and the action each threshold triggers in Section B.
- Point Section D at your real incident channel and records.
- For an on-platform model with guaranteed notification, Section B5 may be NA; for an API model it is central.