Work Instruction: Classifying and Executing an AI Model Change for a Pharmacovigilance Use

This is a ready-to-use work instruction for the person who classifies and executes a change to a production pharmacovigilance AI model, whether that change is a routine retrain, a MedDRA dictionary version upgrade, a decision-threshold adjustment, a new data source, an architecture change, a vendor-driven base-model change, or the promotion of a new frozen version for a model under continuous or incremental learning. It operationalizes the change-classification logic at task level so the same kind of change is handled the same way every time, and feeds the fuller documentation held in Record: AI prompt, model version, and retrieval corpus change control for changes reaching prompt, model, schema, or corpus content. Replace every <<FILL: ...>> placeholder and run it under its parent SOP and validation protocol. A filled specimen follows. This content is educational reference, not legal or regulatory advice; adapt it to your validated system and quality system.

Control header

Before you start

• Confirm the model is in its validated state: the pinned model identifier and, where relevant, the pinned MedDRA version match what is deployed. If either differs from the last approved record, stop and route to <<FILL: role>> before proceeding; an already-drifted model cannot be the baseline for a new change.
• Confirm you are the change owner or delegate named in the AI register entry for this model.
• Have the current locked held-out test set, the current acceptance criteria, and the model’s rejection-sampling program (if risk pattern B) ready to reference.

Procedure

Step 1, Identify the trigger and describe the change plainly

Name what is actually changing and why: a scheduled retrain, a MedDRA dictionary release, a monitoring signal that suggests a threshold move, a new intake channel or product going live, a proposed architecture change, a vendor deprecation notice, or a scheduled promotion gate for a continuously learning model.

• Acceptance: the trigger and the plain-language business reason are recorded before classification.

Step 2, Classify the change using the table below

• Acceptance: exactly one primary classification is selected and recorded, with the reasoning, before any testing begins. Where a change plausibly fits two rows, take the higher-evidence path.

Step 3A, Routine retrain: confirmatory test

1. Run the retrained model against the current locked test set without adjusting thresholds to fit the result.
2. Confirm every acceptance criterion (recall, precision, and any PV-specific criterion such as verbatim preservation or negation handling) meets or exceeds the approved specification.
3. Document the result and release under a documented record; no revalidation protocol re-execution is required if the confirmatory test passes.

• Acceptance: confirmatory test executed on the locked, unmodified test set; result meets or exceeds spec; documented and QA-acknowledged.

Step 3B, MedDRA version upgrade: impact assessment

1. Identify every model whose output depends on the MedDRA version (coding models, and any duplicate-detection or signal model that keys on coded terms).
2. Assess whether the new dictionary version changes, retires, or splits terms the model was trained or tested against.
3. Re-test affected performance under the new version, including a spot check of terms known to have changed.
4. Update the pinned MedDRA version in the header and in the AI register only after the re-test passes.

• Acceptance: impact assessment completed and documented before the new dictionary version is used in production coding; affected performance re-tested; pinned version updated only after passing.

Step 3C, Threshold adjustment: targeted re-test

1. State the specific threshold changing (for example a confidence cutoff or a recall/precision trade-off point) and the reason.
2. Re-test the affected metrics on the locked test set at the new threshold.
3. Update the acceptance-criteria specification and its rationale to match the new threshold; do not leave the URS pointing to the old value.

• Acceptance: re-test executed at the new threshold; specification and rationale updated; QA approval before the new threshold goes live.

Step 3D, New data source or feature: targeted revalidation

1. Confirm the new source (a new intake channel, a new product, a new report type) is represented in an updated training or fine-tuning set with documented labeling.
2. Re-test overall performance and performance specifically on the new source’s records.
3. Assess whether the risk pattern or the risk assessment needs updating given the new scope.

• Acceptance: targeted revalidation complete on both the overall test set and a subset specific to the new source; risk assessment reviewed and confirmed current.

Step 3E, Architecture change: full revalidation

1. Treat this as a new model for validation purposes: fresh performance specification review, full test-case execution per the model’s validation protocol, and an updated model description and data record.
2. Do not carry forward the prior version’s acceptance criteria unexamined; confirm they still reflect the intended use and the consequence of error.

• Acceptance: full revalidation protocol executed and passed; QA release before production use.

Step 3F, Vendor base-model change: uninitiated-change path

1. Treat any vendor-driven change to the underlying model (a version bump, a silent weight update behind a stable-looking name) as a change the company did not initiate but must still assess.
2. Re-run the confirmatory test set immediately on notice of a vendor change, and hold deployment on the new version until it passes, using a pinned prior version in the interim where the platform allows it.
3. Record the vendor notice, the exact version string returned by the platform, and the confirmatory result.

• Acceptance: confirmatory test executed and passed before the vendor-updated model is relied on in production; if the platform does not allow holding a prior pinned version, escalate to the PV / Safety System Owner and QA before the vendor’s change reaches production.

Step 3G, Continuous-learning promotion: gate check

1. Confirm the model has been frozen into a candidate versioned state per the predetermined change control plan; a continuously updating, unversioned model cannot be promoted.
2. Evaluate the candidate version against the locked test set and the full PV-specific acceptance criteria (recall on the safety-relevant class, verbatim preservation, negation handling, and, for a screening model, the current rejection-sampling false-negative rate).
3. Promote only if every criterion is met; if not met, the model continues running the last promoted version while the candidate is investigated.

• Acceptance: the promoted version is the one evaluated and passed at the gate, not a version that continued learning after the gate; the promotion decision and evidence are recorded before the new version goes live.

Step 4, Update records and notify

1. Update the AI register entry, the model card, and the pinned version(s) in this work instruction’s header.
2. Notify reviewers of any change that affects what they see or how they should interpret model output.
3. Set or reset the post-change monitoring window per the parent change-control record, and confirm the monitoring owner.

• Acceptance: register, model card, and header updated; reviewer notification issued where relevant; monitoring window defined with a named owner.

Roles

Records generated

• The classification record (Step 2), retained per <<FILL: retention>>.
• The test evidence for the applicable path (Step 3A through 3G).
• The updated register, model card, and header (Step 4).

References

Parent SOP <<FILL: SOP-ID>> and the model’s validation protocol <<FILL: PRT-ID>>. Record: AI prompt, model version, and retrieval corpus change control, for the full per-event documentation this work instruction feeds. 21 CFR Part 11 and EU GMP Annex 11, for record and version controls. FDA and EMA, “Guiding Principles of Good AI Practice in Drug Development” (published jointly 14 January 2026), for lifecycle change-management expectations. ICH Q9(R1), Quality Risk Management, for sizing the test path to risk.

Confirm the current version of each reference before use.

Filled specimen

A change owner handles a MedDRA dictionary upgrade for a coding-suggestion model. Illustrative only.

The impact assessment did not assume the dictionary update was harmless because it was routine; it checked the 14 affected terms specifically and only updated the pinned version after the re-test passed. That order, assess before you rely on the new state, not after, is the entire point of routing a “just an update” through this work instruction rather than treating it as ordinary IT maintenance.

Common mistakes this work instruction prevents

• A MedDRA dictionary upgrade applied to the coding model with no impact assessment, so retired or merged terms silently corrupt new coding.
• A threshold changed in production with the URS left pointing at the old value, so the specification and the deployed behavior no longer match.
• A vendor-updated base model relied on in production before a confirmatory test was run against it.
• A continuously learning model promoted to production based on informal confidence rather than a defined gate evaluated against the full PV-specific acceptance criteria.
• Two different change types (for example a retrain that also adds a new data source) both classified under the lighter of the two paths instead of the higher-evidence one.

How to adapt this work instruction

1. Set the parent SOP, model ID, pinned versions, and roles in the header.
2. Confirm the seven change types in Step 2 match the changes your model actually undergoes; add a row for a change type specific to your platform if needed.
3. Point the records section at your real AI register, model card, and monitoring records.
4. Keep the QPPV notification step for any change touching a reporting-clock or signal-relevant model; do not let a “routine” label skip it.