This is a ready-to-use qualification protocol for the MLOps pipeline that produces and serves a GxP machine-learning model. Replace every <<FILL: ...>> placeholder with your own specifics, set your document numbers and dates, and route it through your normal validation process. A worked filled specimen test case follows the blank set. Verify each cited reference against the current source before you rely on it.
The reason this protocol exists is a common scaling failure: teams validate the trained model and forget that the machinery producing and serving models is what keeps the system in a validated state over time. If a new model version can reach production without passing through the validated, gated pipeline, the pipeline is not the control, and every model has to be validated by hand forever. Qualifying the pipeline is what lets routine updates flow through a controlled path instead of triggering bespoke validation each time.
Approval page
| Field | Entry |
|---|---|
| Protocol title | MLOps Pipeline Qualification for <<FILL: SYSTEM / USE CASE>> |
| Protocol number | <<FILL: PQ-ID, e.g. VAL-AI-014-PIPE>> |
| Version | <<FILL: version>> |
| Effective date | <<FILL: effective date>> |
| System owner | <<FILL>> |
| Role | Name | Signature | Date |
|---|---|---|---|
| Author (Validation / ML Engineering) | <<FILL>> | ||
| Reviewer (Data Science) | <<FILL>> | ||
| Approver (QA) | <<FILL>> |
1. Objective
To demonstrate that the machine-learning pipeline for <<FILL: SYSTEM>> extracts data correctly, preprocesses it consistently with training, trains reproducibly, deploys only the approved model through a gated path, logs the model version with every prediction, and triggers monitoring as designed, so that routine model updates can flow through a controlled, validated pipeline.
2. Scope
In scope: the data extraction step, the preprocessing and feature step, the training step, the model registry and promotion gate, the deployment step, the prediction-logging step, and the monitoring triggers. Out of scope: the model performance validation itself (covered by <<FILL: model validation protocol number>>) and the underlying infrastructure qualification (covered by <<FILL: infrastructure qualification reference>>).
3. System description
<<FILL: describe the pipeline: source systems, the extraction schedule, the preprocessing implementation shared between training and serving, the training environment and how it is pinned, the model registry, the deployment mechanism and gate, the prediction store, and the monitoring stack.>>
4. Prerequisites
- Approved requirements and design for the pipeline are on file.
- The environment is pinned and reproducible (library versions and runtime recorded).
- Test data sets are prepared, including at least one malformed-record set and one drifted-input set.
- Roles are assigned and testers are trained on this protocol.
5. Roles
| Role | Responsibility |
|---|---|
| ML Engineering | Executes technical test steps, provides pipeline evidence. |
| Validation / CSV | Owns the protocol, reviews evidence, compiles the report. |
| Data Science | Confirms reproducibility and preprocessing equivalence. |
| QA | Approves the protocol and report, oversees deviations. |
6. Acceptance criteria
- Extraction pulls the correct, complete records and records what was pulled and when.
- Serving preprocessing produces byte-for-byte (or within a defined numeric tolerance) the same features as training preprocessing on the same input (no training-serving skew).
- Two training runs from the same pinned inputs produce the same model within the defined tolerance (reproducibility).
- Only an approved, registered model version can be promoted to production; an unapproved version is blocked.
- Every prediction is stored with its input reference, output, model version, and timestamp.
- A malformed record triggers the defined behavior (rejection or fallback), not a silent wrong answer.
- A drifted-input test raises the monitoring trigger and the defined notification.
- A dependency or library version change is detected and blocked from silently altering output without a regression check.
7. Test cases (blank)
| ID | Test step | Expected result | Actual | Pass / Fail | Tester / date |
|---|---|---|---|---|---|
| TC-01 | Run extraction for a known period; compare pulled records to source | All target records pulled; extract log records count and timestamp | <<FILL>> | <<FILL>> | <<FILL>> |
| TC-02 | Feed one input through training preprocessing and serving preprocessing | Features match within tolerance; no skew | <<FILL>> | <<FILL>> | <<FILL>> |
| TC-03 | Train twice from identical pinned inputs | Models match within the reproducibility tolerance | <<FILL>> | <<FILL>> | <<FILL>> |
| TC-04 | Attempt to promote an unapproved model version | Promotion blocked; event logged | <<FILL>> | <<FILL>> | <<FILL>> |
| TC-05 | Deploy the approved version; run predictions | Each prediction stored with input ref, output, model version, timestamp | <<FILL>> | <<FILL>> | <<FILL>> |
| TC-06 | Submit a malformed record | Defined rejection or fallback occurs; no silent output | <<FILL>> | <<FILL>> | <<FILL>> |
| TC-07 | Submit a drifted-input batch | Drift trigger fires; system owner notified per the response plan | <<FILL>> | <<FILL>> | <<FILL>> |
| TC-08 | Change a pinned library version in a test environment | Change detected; regression check required before it can affect output | <<FILL>> | <<FILL>> | <<FILL>> |
8. Deviation handling
Record any deviation with its test case ID, description, and impact. Assess whether it affects a data integrity or performance-critical function. Resolve, document the correction, and re-test the affected case to a pass. QA approves the disposition of every deviation.
9. Summary and conclusion
<<FILL: summarize the results, the deviations and their resolution, and state whether the pipeline is qualified for its intended use.>>
10. Attachments
- Extract logs, preprocessing comparison output, reproducibility evidence, promotion-gate logs, prediction-store samples, monitoring alert evidence.
- Executed test sheets with tester signatures.
References
21 CFR Part 11 and EU GMP Annex 11 for records, audit trail, and access control the pipeline must satisfy. GAMP 5 Second Edition, ISPE, for the risk-based, configured-system approach (referenced by title; described, not reproduced). FDA guidance on Computer Software Assurance for production and quality management system software, for the least-burdensome, risk-based testing emphasis.
Confirm the current version of each reference before issue.
Revision history
| Version | Date | Author | Summary of change |
|---|---|---|---|
<<FILL: 1.0>> | <<FILL: date>> | <<FILL: author>> | Initial issue. |
Filled specimen: TC-02 (training-serving skew)
The following shows one executed test case, so you can see the evidence expected. The values are illustrative; replace them with your own.
| Field | Entry |
|---|---|
| Test case | TC-02, preprocessing equivalence (no training-serving skew) |
| Test step | Passed the same raw deviation record through the training-time preprocessing module and the serving-time preprocessing path, then compared the resulting feature vectors field by field |
| Expected result | Feature vectors identical within the defined tolerance; the same shared implementation used in both paths |
| Actual result | 41 of 42 features matched exactly; the free-text length feature differed (training trimmed whitespace, serving did not), producing a 3 percent value difference on that feature |
| Pass / Fail | Fail on first execution; deviation DEV-AI-014-02 raised |
| Resolution | Serving path corrected to call the same trimming function as training; re-tested; all 42 features matched; Pass |
| Tester / date | R. Nwosu, 20 July 2026 |
In this example the test caught a real training-serving skew before go-live: the serving path was not trimming whitespace the way the training path did, so one feature was computed differently in production than in training. That is exactly the invisible failure that makes a fine model underperform in production while every model-level check passes. The fix was to share the one preprocessing implementation, then re-test to a clean pass.
Common inspection findings this protocol prevents
- The model was validated but the pipeline that feeds and serves it was never qualified, so the validated state is not maintained.
- Training-serving skew degrades production performance while model-level checks pass.
- A model version reaches production without passing through the gated, approved path.
- Predictions are not stored with the model version that produced them, so a problem cannot be investigated.
- A malformed record or a dependency change silently changes output with no control.
How to adapt this protocol
- Set your protocol number, system description, and the cross-references to your model validation and infrastructure qualification.
- Set the reproducibility and preprocessing tolerances from your model type and risk.
- Add test cases for any pipeline-specific step your architecture includes (for example a retrieval index refresh for a generative system).
- Keep the executed protocol and evidence in the validation file, and re-execute the affected cases after any pipeline change.