Protocol: AI/ML Validation for Pharmacovigilance Case-Processing Models

This is a ready-to-use validation protocol for an AI or machine learning model used in a pharmacovigilance process, for example MedDRA auto-coding, intake triage, literature screening, or signal-detection augmentation. It builds on the general method in Protocol: AI/ML system validation and adds the test cases that are specific to drug safety: recall on the safety-relevant class, verbatim preservation, negation and uncertainty handling, MedDRA version pinning, and rejection sampling for a model that screens what a human sees. Replace every <<FILL: ...>> placeholder with your own specifics, set your document numbers and dates, attach your requirements and traceability, and route it through your normal document control, review, and approval. A worked filled specimen follows the template. The AI-specific regulatory framework for pharmacovigilance is still forming, so verify each cited reference against the current source and state plainly in the protocol where you are reasoning by analogy because no guidance fits cleanly. This content is educational reference, not legal or regulatory advice.

Document control header

1. Objective

This protocol demonstrates that <<FILL: MODEL NAME>>, model version <<FILL: model version>>, meets its approved performance specification for its intended use in the pharmacovigilance process named in section 2, behaves acceptably on negated and uncertain safety language, preserves the reporter’s verbatim where the use case touches case data, remains tied to a specific MedDRA version where it codes, can be shown to remain in a validated state over time, and operates under a defined and recorded human-oversight control appropriate to its risk pattern. The objective is to prove a performance distribution against a threshold sized to the consequence of a missed case or a missed signal, not to prove a single fixed input-output table once and freeze it.

2. Scope

This protocol covers the trained model instance named in the header, the safety data flowing into and out of it, the human-oversight step over its outputs, and the audit trail that records both the model output and the human decision. It applies to <<FILL: the specific PV process, e.g. inbound case intake triage / MedDRA coding of the verbatim / literature screening / signal-cluster review>> at <<FILL: site / business unit>>.

In scope: model performance against the specification on the safety-relevant class, verbatim handling, negation and uncertainty behavior, MedDRA version dependency, boundary and adversarial behavior, drift detection and the response to it, the human-oversight workflow, the audit trail and electronic-record controls, and traceability from the user requirements to the test evidence. For a screening model (risk pattern B, defined in section 4), rejection sampling of what the model did not surface to a human is explicitly in scope.

Out of scope: qualification of the underlying safety database or platform infrastructure (covered by <<FILL: infrastructure qualification / platform validation ID>>); the case-processing procedure itself, governed by <<FILL: SOP-ID for ICSR intake>> or <<FILL: SOP-ID for signal management>>; and periodic aggregate reporting content, which is out of scope unless the model drafts aggregate report sections, in which case <<FILL: reference the aggregate-report use case addendum>>.

3. Responsibilities

4. Intended use and risk pattern

State the intended use in one sentence naming the model output, the action it triggers, and the accountable role. If you cannot write that sentence cleanly, the intended use is not yet defined and validation cannot be sized correctly.

Pattern A is the lowest-risk pattern: a qualified person reviews every output before it has any effect. Pattern B is a model that decides what a human sees, so the dangerous failure is the false negative that never reaches a reviewer; this protocol adds rejection sampling (section 9.4) specifically for this pattern. Pattern C, an output that drives a regulatory-relevant action with no per-item human confirmation, requires the most evidence and a documented rationale for why a human is not in the loop; most PV deployments should not be validated as Pattern C without that rationale on file and approved by QA and the QPPV or delegate.

5. Data and model description

This section is the reproducibility record for what was validated. Without it, an investigator cannot rebuild the model and an inspector cannot trace the chain from the reporter’s original words to the coded, structured record.

5.1 Model description

5.2 Training, tuning, and test data

6. Acceptance criteria for a pharmacovigilance model

Acceptance criteria are performance metrics against thresholds, set before training and recorded in the URS, not after the model is measured. In pharmacovigilance the dangerous failure is almost always the miss, so recall on the safety-relevant class is usually the controlling metric; state explicitly if a different metric governs and why.

Never accept raw accuracy on an imbalanced dataset. Where the safety-relevant class is a small minority, a model that predicts the majority class every time scores misleadingly high accuracy while missing everything that matters. Report recall, precision, and, where positives are rare, a confidence interval.

7. Pre-execution requirements

Execution does not start until all of the following are confirmed and recorded in section 13.

8. Test approach

Depth of testing follows the risk pattern and the consequence of error. Each test case traces to one or more acceptance criteria in section 6 and to the URS requirement it satisfies; record the mapping in the traceability matrix. Performance is reported only on the locked held-out test set; a metric computed on training data is not validation evidence.

9. Test cases

9.1 TC-PERF: Performance against the specification

1. Run the frozen model against the entire locked test set; do not retrain, tune, or adjust thresholds during the run.
2. Generate the confusion matrix and compute each metric in AC-1 through AC-4.
3. Where positives are rare, compute and record the confidence interval per AC-4.
4. Acceptance: every metric meets or exceeds its threshold, and the lower confidence bound still meets the primary threshold.

9.2 TC-CAL: Confidence calibration

1. Bin predictions by stated confidence.
2. For each bin, compute the observed accuracy and compare to the stated confidence.
3. Acceptance: stated confidence matches observed accuracy within the AC-3 tolerance, sound for any workflow that routes by confidence threshold.

9.3 TC-NEGATION: Negation and uncertainty handling

1. Assemble a fixed, versioned set of paired sentences differing only in polarity or hedging (for example “denies chest pain” against “reports chest pain,” “no evidence of rash” against “rash noted”), plus real historical verbatims already known to contain a negation or a hedge.
2. Run the model against the set.
3. Acceptance: every pair resolves to the correct polarity per AC-6. Record any failure individually; a model that is otherwise accurate but fails negation systematically is not acceptable for release, because the failure lands exactly on the safety-relevant direction.

9.4 TC-VERBATIM: Verbatim preservation

1. Process a sample of test cases end to end through the model and any downstream storage step it touches.
2. Confirm the reporter’s original text is stored unaltered and separately from the coded or classified output, and that recoding does not overwrite it.
3. Acceptance: verbatim is preserved unaltered for 100 percent of the sample per AC-5.

9.5 TC-MEDDRA-VERSION: MedDRA version dependency (coding models only)

1. Confirm every coded output in the test run carries the MedDRA version pinned in the header.
2. Simulate or apply a MedDRA version change on a small subset and confirm the model output and the change are both captured as distinct, dated entries rather than an overwrite.
3. Acceptance: coded output is version-traceable per AC-7.

9.6 TC-REJECT-SAMPLE: Rejection sampling (risk pattern B only)

1. Draw a representative, sized sample of items the model did not surface to a human (dropped as “not relevant,” discarded as a likely duplicate, or filtered from literature screening).
2. Have a qualified reviewer independently assess each sampled rejection against ground truth.
3. Compute the false-negative rate within the sample.
4. Acceptance: the measured false-negative rate meets AC-8. This test case has no equivalent in a generic AI validation protocol; a Pattern B model without it is not validated, it is trusted, and trust is not evidence.

9.7 TC-BOUND: Boundary and out-of-distribution behavior

1. Submit inputs at the edge of the expected range and inputs from outside the training distribution (a new product, an unfamiliar reporting channel, a foreign-language verbatim if translation is in scope).
2. Observe whether the model flags low confidence or routes to human review rather than returning a confident wrong answer silently.
3. Acceptance: out-of-distribution and boundary inputs produce a defined safe behavior, not a silent high-confidence error.

9.8 TC-HUMAN: Human oversight in operation

1. Run representative outputs through the live or staged workflow with the real reviewer role.
2. Confirm the reviewer sees the model output, its confidence or rationale, and the model version, and can confirm or override.
3. Confirm the reviewer’s decision, the output reviewed, and the model version are recorded in a GxP record.
4. Monitor the rate at which suggestions are accepted unmodified as a signal for automation bias; treat an unusually high rate as something to investigate.
5. Acceptance: every in-scope output is reviewed and recorded per AC-10, and the review is defined and meaningful.

9.9 TC-AUDIT: Audit trail and electronic records

1. Generate model outputs and corresponding human decisions, then export the audit trail.
2. Confirm the audit trail captures the model output, the model version, the MedDRA version where relevant, the reviewer identity, the decision, and a time stamp.
3. Acceptance: the audit trail and record controls meet AC-11.

9.10 Test case execution record (per case)

10. Acceptance criteria for the protocol as a whole

• The intended-use statement and risk pattern (A, B, or C) are written and approved, with the ICH Q9(R1)-based rationale on file.
• The performance specification was written in the URS before training, with the safety-relevant metric, its threshold, and a justification tied to the consequence of the miss.
• Performance (AC-1 to AC-4) was reported on the locked, version-controlled test set the model never saw, and every metric meets its threshold.
• Verbatim preservation, negation and uncertainty handling, and MedDRA version dependency (AC-5 to AC-7, where applicable) meet their criteria.
• For a risk pattern B model, rejection sampling (AC-8) is complete and meets its criterion.
• Drift detection, human oversight, and the audit trail (AC-9 to AC-11) meet their criteria.
• Every open deviation is closed or assessed as not blocking, with the rationale recorded.
• Traceability runs from intended use to requirements to test evidence, with rationale recorded wherever guidance was silent.

11. Deviation handling

1. Record any actual result that does not meet an acceptance criterion contemporaneously with the test case ID and the objective evidence.
2. Classify by impact: a deviation on AC-1, AC-5, AC-6, AC-7, or AC-8 (the safety-critical criteria) blocks release until resolved; a deviation on a lower-risk criterion may be dispositioned with QA-approved rationale.
3. Investigate root cause. For a performance shortfall, determine whether the cause is the data, the labels, the metric choice, the threshold, or the model itself; do not move the threshold to make the result pass.
4. Define the correction and re-test the affected cases.
5. QA reviews and approves the disposition of every deviation. Route to <<FILL: SOP-ID for deviations>> where the impact warrants a formal investigation.

12. Summary and conclusion

13. Pre-execution and execution confirmation

14. References

ICH E2B(R3), Data Elements for Transmission of Individual Case Safety Reports; ICH E2D(R1), Post-Approval Safety Data. EU GVP Module VI (case management) and Module IX (signal management). FDA draft guidance, “Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products” (issued 6 January 2025), risk-based credibility framework; draft, confirm current status. FDA and EMA, “Guiding Principles of Good AI Practice in Drug Development” (published jointly 14 January 2026), foundational, non-binding principles spanning the medicine lifecycle including post-marketing. EMA, “Reflection paper on the use of artificial intelligence in the lifecycle of medicines” (adopted September 2024), non-binding. GAMP 5 Second Edition (ISPE, 2022), risk-based CSV with material on machine learning. ICH Q9(R1), Quality Risk Management, basis for sizing validation effort to risk. 21 CFR Part 11 and EU GMP Annex 11, electronic records, signatures, and audit trails. 21 CFR 314.80 / 600.80 (postmarketing adverse experience reporting) and 21 CFR 312.32 (IND safety reporting), as applicable.

Several references above are draft or foundational rather than final and binding. Confirm the current version, status, and dates of each before issue, and state in the protocol where a reference is draft rather than settled.

15. Revision history

16. Approvals

Filled specimen

The following shows the protocol completed for an example MedDRA coding-suggestion model with mandatory human confirmation (risk pattern A). The company, model, and numbers are illustrative; replace them with your own.

Intended use and risk pattern (filled)

Acceptance criteria (filled, primary lines)

Performance result (filled, from TC-PERF)

On locked test set v2 (2,000 verbatims), the model’s top suggestion exactly matched the coder-confirmed term in 1,720 of 2,000 cases, an agreement rate of 0.86, meeting AC-1. On TC-NEGATION, all 60 paired sentences resolved to correct polarity. On TC-VERBATIM, all 200 sampled cases retained the original verbatim unaltered and separately from the coded field.

Deviation (filled)

In this example the subgroup weakness was found by looking past the aggregate number, not by trusting it. That is exactly what section 6’s insistence on representativeness across the long tail is for.

Common inspection findings this protocol prevents

• A coding model validated only on aggregate agreement, hiding a systematic weakness on an underrepresented product class or event type.
• No verbatim-preservation test, so recoding to a later MedDRA version silently overwrites the reporter’s original words.
• No negation testing, so a model that reliably misreads “denies X” as “reports X” reaches production undetected.
• A screening model with no rejection-sampling evidence, so the false-negative rate the model produces in production is unknown.
• Coded output with no MedDRA version stamp, so a later dictionary change cannot be reconciled against what was coded and when.
• Acceptance criteria written after the model’s performance was known.

How to adapt this protocol

1. Set your document number, model identifier, MedDRA version, and effective date in the header.
2. Write the intended-use sentence and assign the risk pattern (A, B, or C) in section 4 before anything downstream is sized.
3. Fill the data and model description in section 5 completely, including the MedDRA version of the training and test data.
4. Set the acceptance criteria in section 6 before training, and keep AC-5 through AC-8 even if your first instinct is to skip them; they are the PV-specific evidence a generic AI protocol does not ask for.
5. Build the negation/uncertainty test set (9.3) from your own historical verbatims, not a generic sentence list.
6. If the model is risk pattern B, do not release without executing TC-REJECT-SAMPLE and sizing the sample to detect a meaningful false-negative rate.
7. Confirm every reference in section 14 against the current published version and status before issue.