This is a ready-to-use risk assessment for an AI or machine learning model used anywhere in a pharmacovigilance process. It answers two questions a generic AI/ML risk assessment does not: which of the three PV risk patterns (human-confirmed assistance, model-gated screening, or autonomous action) does this model fall into, and how much evidence does that pattern demand given pharmacovigilance-specific consequences such as a missed reporting-clock case or a distorted signal. It complements, rather than replaces, Risk assessment: AI/ML system, which covers the general patient/product/data impact scoring; use that worksheet’s method for a PV system’s non-safety-specific factors and this one for the factors unique to drug safety. Replace every <<FILL: ...>> placeholder with your own specifics, set your document numbers and dates, and route it through your normal document control, review, and approval. Confirm each cited regulation against the current source before you rely on it; the AI-specific PV framework is still forming, so treat draft and foundational references as such.
Document control header
| Field | Entry |
|---|---|
| Document title | AI Model Risk Pattern and Credibility Classification for <<FILL: MODEL / USE CASE NAME>> |
| Document number | <<FILL: RA-ID, e.g. RA-PV-AI-004>> |
| Version | <<FILL: version, e.g. 1.0>> |
| Effective date | <<FILL: effective date>> |
| Supersedes | <<FILL: prior version or "New">> |
| Document owner | <<FILL: role, e.g. PV Systems Owner>> |
| Model version assessed | <<FILL: model version / hash>> |
| Linked validation protocol | <<FILL: PRT-ID>> |
| QPPV / delegate informed | <<FILL: Yes / No, date>> |
1. Purpose
This assessment determines the risk pattern of <<FILL: MODEL / USE CASE NAME>> and, from that pattern plus a set of pharmacovigilance-specific risk factors, the evidence tier that sets validation depth, monitoring cadence, and whether a rejection-sampling program is required. The output lets an inspector reconstruct why the model was controlled the way it was, using the risk-based credibility reasoning the FDA’s January 2025 draft AI guidance and the January 2026 FDA/EMA joint principles both point toward, without treating either as settled law.
2. Scope
This assessment covers the AI/ML model <<FILL: MODEL / USE CASE NAME>>, used to <<FILL: business purpose, e.g. propose a MedDRA code / triage inbound messages / screen literature / augment signal review>> within <<FILL: process name>>. It covers the trained model, its input safety data, its output and the action that output drives, and the human and system controls around it. It does not replace the underlying case-processing or signal-management procedure, governed by <<FILL: SOP-ID>>, nor the platform’s own computerized system validation, governed by <<FILL: VP-ID>>.
3. Responsibilities
| Role | Responsibility |
|---|---|
| PV / Safety System Owner | Owns the intended-use statement and confirms the risk-pattern classification reflects the real workflow, not the intended one. |
| Data science / ML engineering | Provides the model risk-factor inputs (data quality, class balance, drift behavior) honestly, including known weaknesses. |
| Safety physician / PV scientist | Confirms the clinical consequence of a wrong output for signal- and narrative-touching models. |
| Validation / CSV lead | Facilitates the assessment and ties the output to the validation protocol and monitoring program. |
| Quality Assurance | Approves the risk logic, confirms the impact and factor scores are reasoned, and confirms the human-review design is real. |
| QPPV or delegate | Informed of the risk pattern and residual risk for any model that can affect a reporting-clock decision or a signal disposition. |
4. Definitions
- Risk pattern A, human-confirmed assistance: a qualified person reviews every model output before it has any effect. The lowest-risk pattern.
- Risk pattern B, model-gated screening: the model decides what a human sees; the dangerous failure is the false negative that never reaches a reviewer.
- Risk pattern C, autonomous or near-autonomous action: the output drives a regulatory-relevant action with no per-item human confirmation. The highest-risk pattern and the hardest to defend.
- Reporting-clock impact: the degree to which a wrong output could start, fail to start, or mis-time an expedited reporting obligation.
- Signal escalation impact: the degree to which a wrong output could change what is escalated to, or withheld from, a signal review board or a health authority.
- Class imbalance severity: how rare the safety-relevant class is in the population the model sees, and therefore how easily a naive model can look accurate while missing what matters.
- Verbatim integrity risk: the degree to which the model’s process could alter, obscure, or fail to preserve the reporter’s original words.
- MedDRA dependency: whether the model’s output is tied to a specific MedDRA version and would need re-assessment on a dictionary upgrade.
5. Method
5.1 Step 1, state the intended use and classify the risk pattern
Write the one-sentence intended-use statement, then classify the model as Pattern A, B, or C using the actual workflow, not the intended one. A model marketed as “advisory” that a busy processor never actually reviews is Pattern B or C in practice, whatever the design document says. Record the basis for the classification.
5.2 Step 2, score the pharmacovigilance-specific risk factors
Score each factor on the scale in section 5.3. These are in addition to, not instead of, the patient/product/data impact factors in the general AI/ML risk assessment.
- Reporting-clock impact. Could a wrong output start, fail to start, or mis-time the 15-day (or applicable) expedited clock? High for intake triage and validity/seriousness classification; typically lower for a coding suggestion that a human confirms before submission.
- Signal escalation impact. Could a wrong output change what reaches a signal review board or a health authority? High for signal-detection augmentation and aggregate-report drafting; lower for duplicate detection on non-serious cases.
- Class imbalance severity. How rare is the safety-relevant class, and how easily could a model that ignores it still look accurate? Score high where positives are a small minority and the miss is costly.
- Verbatim integrity risk. Could the model’s process alter, truncate, translate, or fail to separately preserve the reporter’s original words? High for narrative generation and translation; typically lower for a pure classification task that never touches the verbatim field.
- Negation and language risk. How much does the task depend on correctly reading negation, hedging, or non-English source text? High for any model reading free text; near-irrelevant for a model that only reads structured fields.
- MedDRA dependency. Does the output depend on a specific dictionary version such that a routine upgrade could shift the model’s labels under it? High for coding models; not applicable to most other use cases.
- Autonomy. How far does the output reach a regulatory-relevant action with no human or deterministic check in between? This factor anchors the risk pattern.
- Human oversight quality. Is there a defined, documented, meaningful human review, resistant to automation bias? A strong review lowers residual risk; a review that exists only on paper does not.
5.3 Scoring scale
| Score | Label | Meaning |
|---|---|---|
| 3 | High | This factor could reach a patient, a reporting obligation, or a signal decision, with no reliable control to catch it. |
| 2 | Medium | This factor could matter, but a named control would usually catch it before it reaches the outcome. |
| 1 | Low | This factor has no realistic path to a reporting, signal, or record-integrity consequence. |
5.4 Step 3, combine into an evidence tier
Take the risk pattern as the anchor (C dominates, then B, then A) and let a high score on reporting-clock impact, signal escalation impact, or class imbalance severity move a Pattern A or B assessment up a tier. Record the reasoning, not just the tier.
| Evidence tier | Typical pattern and factor combination | What it requires |
|---|---|---|
| Tier 1, standard | Pattern A, no factor scored High | Performance spec on a locked test set, defined human review, standard monitoring cadence |
| Tier 2, elevated | Pattern A with a High factor (for example high class imbalance on a rare serious event), or Pattern B with all factors Medium or lower | Full performance spec including confidence interval treatment, verbatim and negation test cases where applicable, rejection sampling if Pattern B, tightened monitoring |
| Tier 3, maximum | Pattern B with any High factor, or any Pattern C classification | Full evidence set including rejection sampling sized to detect a meaningful miss rate, a documented rationale for any absence of per-item human confirmation, QPPV or delegate sign-off, and the shortest monitoring cadence the process supports |
5.5 Step 4, define mitigations and residual risk
For each High or Medium driver, name a mitigation, its owner, and how its effectiveness is evidenced. Then state residual risk and whether it is acceptable for the intended use.
6. Risk register
| Item | Score (1-3) | Reasoning |
|---|---|---|
| Reporting-clock impact | <<FILL>> | <<FILL>> |
| Signal escalation impact | <<FILL>> | <<FILL>> |
| Class imbalance severity | <<FILL>> | <<FILL>> |
| Verbatim integrity risk | <<FILL>> | <<FILL>> |
| Negation and language risk | <<FILL>> | <<FILL>> |
| MedDRA dependency | <<FILL: or "not applicable">> | <<FILL>> |
| Autonomy | <<FILL>> | <<FILL>> |
| Human oversight quality | <<FILL>> | <<FILL>> |
| Field | Entry |
|---|---|
| Risk pattern (A / B / C) | <<FILL>> with <<FILL: reasoning based on the actual workflow>> |
| Evidence tier assigned | <<FILL: 1 / 2 / 3>> |
| Key mitigations (owner, evidence) | <<FILL>> |
| Residual risk | <<FILL: statement>> |
| Residual risk acceptable for intended use | Yes / No, <<FILL: justification>> |
7. Acceptance criteria
This assessment is acceptable when all of the following are true:
- The intended-use statement is written and the risk pattern reflects the actual workflow, verified against how the process runs today, not the design intent.
- Every pharmacovigilance-specific factor is scored with written reasoning, including honest statement of known weaknesses.
- The evidence tier is derived from the pattern and the factors, with reasoning, and matches the depth requirements in section 5.4.
- Every High or Medium driver has a named mitigation with an owner and an evidence source.
- Residual risk is stated and judged acceptable or not for the intended use.
- For any Pattern C classification, or any Pattern B classification with a High factor, the QPPV or delegate has been informed and the absence of full per-item human confirmation is explicitly justified in writing.
8. References
FDA draft guidance, “Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products” (issued 6 January 2025), risk-based credibility framework keyed to context of use; draft, confirm current status. FDA and EMA, “Guiding Principles of Good AI Practice in Drug Development” (published jointly 14 January 2026), foundational, non-binding principles covering post-marketing safety monitoring. EMA, “Reflection paper on the use of artificial intelligence in the lifecycle of medicines” (adopted September 2024), human-centric and risk-based, non-binding. ICH Q9(R1), Quality Risk Management, basis for sizing assurance to risk. EU GVP Module VI (case management) and Module IX (signal management). 21 CFR Part 11 and EU GMP Annex 11, for records and signatures the model touches, at full rigor regardless of tier.
Confirm the current version and status of each reference before issue. The AI-specific PV framework is moving quickly; where a reference is draft or foundational, say so in the assessment rather than presenting it as settled.
9. Revision history
| Version | Date | Author | Summary of change |
|---|---|---|---|
<<FILL: 1.0>> | <<FILL: date>> | <<FILL: author>> | Initial issue. |
10. Approvals
| Role | Name | Signature | Date |
|---|---|---|---|
| Author (Validation / CSV lead) | <<FILL>> | ||
| PV / Safety System Owner | <<FILL>> | ||
| Safety physician representative | <<FILL>> | ||
| Approver (QA) | <<FILL>> |
Filled specimen
The following shows the assessment completed for an example intake-triage model that classifies inbound messages as a possible adverse event or not relevant. The company, model, and numbers are illustrative; replace them with your own.
Intended use: The model classifies each inbound message as possible adverse event or not relevant; a possible message is queued for a case processor within one business day; every not relevant message is retained and a defined sample is human-reviewed. A processor confirms validity and owns the decision on every queued message.
| Item | Score | Reasoning |
|---|---|---|
| Reporting-clock impact | 3 | A missed possible-AE message never starts day zero; this is the model’s central failure mode |
| Signal escalation impact | 1 | Triage does not itself feed signal review |
| Class imbalance severity | 3 | Roughly 5 percent of inbound messages are a genuine possible AE; a lazy model could score well while missing most of them |
| Verbatim integrity risk | 1 | The model classifies; it does not alter or store a coded field over the verbatim |
| Negation and language risk | 2 | Messages arrive in three languages; negation errors are plausible and untested to date |
| MedDRA dependency | Not applicable | Triage does not code |
| Autonomy | 2 | Model-gated screening: not-relevant messages are discarded from the queue, but a defined sample is re-reviewed |
| Human oversight quality | 2 | Queued messages are reviewed within one business day by a trained processor; rejection sampling is newly proposed, not yet running |
| Field | Entry |
|---|---|
| Risk pattern | B, model-gated screening. The workflow discards not-relevant messages from the active queue; only the sampled subset is re-reviewed |
| Evidence tier assigned | Tier 3, maximum. Pattern B combined with High reporting-clock impact and High class imbalance severity |
| Key mitigations | Rejection-sampling program sized to detect a 5 percent false-negative rate at 90 percent confidence (owner: PV Quality, evidence: monthly sampling record). Recall-first acceptance criterion set before training (owner: Validation lead, evidence: PRT-PV-AI-002). Negation test set built from the three source languages (owner: Data science, evidence: validation report) |
| Residual risk | A possible-AE message missed by the model and not caught in the rejection sample. Mitigated by a recall threshold justified against the miss, a sized and running rejection-sampling program, and monthly override-rate monitoring |
| Residual risk acceptable | Yes, once the rejection-sampling program is running and its first three cycles show the false-negative rate within the accepted bound; not yet acceptable on the basis of design intent alone |
This specimen shows the point of scoring the factors separately from the pattern: the pattern alone (B) does not tell you the assessment is done. It was the combination of Pattern B with two High-scored PV-specific factors that pushed this model to Tier 3 and made rejection sampling a release condition rather than a nice-to-have.
Common inspection findings this risk assessment prevents
- A screening model classified as low risk because it is “just triage,” with no rejection-sampling evidence behind that classification.
- A risk pattern assigned from the design document rather than from how the workflow actually runs, missing that reviewers do not in fact see every output.
- Class imbalance never scored explicitly, so a model with strong aggregate accuracy and poor recall on the rare safety-relevant class passes review.
- A coding model’s MedDRA dependency never assessed, so a dictionary upgrade is treated as routine IT maintenance rather than a change requiring re-assessment.
- No QPPV or delegate visibility into a model whose output can affect a reporting-clock or signal decision.
How to adapt this risk assessment
- Set your document number, owner, model version, and linked validation protocol in the header.
- Write the intended-use sentence first, and classify the risk pattern from the actual workflow, not the design intent; re-verify this periodically, because workflows drift.
- Score every PV-specific factor honestly, marking “not applicable” only where genuinely irrelevant (for example MedDRA dependency for a non-coding model).
- Anchor the evidence tier on the pattern, then let the factors move it up, and record the reasoning.
- For every High or Medium driver, name a real mitigation with an owner and an evidence source.
- Route any Pattern C or high-factor Pattern B classification to the QPPV or delegate before finalizing.
- Confirm every reference in section 8 against the current published version before issue.