This is a ready-to-use performance specification for a GxP AI/ML model, the section of the user requirements that states what performance the model must achieve before it is trained. Writing it first is what prevents fitting the acceptance criteria to whatever the model happened to score. Replace every <<FILL: ...>> placeholder with your own specifics and route it through your requirements and validation process. A worked filled specimen follows. Verify each cited regulation against the current source before you rely on it.
Document control header
| Field | Entry |
|---|---|
| Document title | AI/ML Model Performance and Acceptance Specification, <<FILL: MODEL / SYSTEM NAME>> |
| Document number | <<FILL: SPEC-ID, e.g. URS-AI-009-PERF>> |
| Version | <<FILL: version>> |
| Effective date | <<FILL: date>> |
| Author | <<FILL: System Owner / Data Science>> |
| Approved by | <<FILL: QA>> |
| Status | Approved before model training: <<FILL: yes/no and date>> |
1. Purpose and intended use
This specification defines the performance the <<FILL: MODEL>> must demonstrate to be acceptable for its intended use, and the evidence required to prove it. Intended use, in one sentence naming the output, the action it triggers, and the accountable role:
<<FILL: e.g. The model assigns a preliminary criticality tier to each new deviation; the tier sets the investigation timeline; the QA reviewer confirms or overrides the tier within one business day and owns the final classification.>>
Risk class (advisory / automated classification / process control): <<FILL>>, with the ICH Q9(R1)-based rationale recorded in <<FILL: risk assessment ref>>.
2. Requirements (numbered and testable)
Each requirement has an ID, a statement, and a test method. A requirement that cannot be objectively tested is not a requirement.
| ID | Requirement | Threshold | Justification (consequence of error) | Test method |
|---|---|---|---|---|
| URS-PERF-001 | Primary metric | <<FILL: e.g. recall >= 0.90>> | <<FILL: a missed positive is the dangerous failure>> | Locked held-out test set |
| URS-PERF-002 | Secondary metric | <<FILL: e.g. precision >= 0.70>> | <<FILL: false positives handled by human review>> | Locked held-out test set |
| URS-PERF-003 | Confidence calibration | <<FILL: stated confidence within +/- X of observed accuracy per band>> | <<FILL: workflow routes by confidence>> | Calibration plot on test set |
| URS-PERF-004 | Performance on the rare/critical subclass | <<FILL: recall on the critical subgroup >= threshold>> | <<FILL: base-rate imbalance can hide failure>> | Stratified test set |
| URS-PERF-005 | Robustness across strata | <<FILL: metric holds across products/sites/instruments>> | <<FILL: representativeness of production>> | Stratified analysis |
| URS-PERF-006 | Statistical confidence | <<FILL: report a confidence interval, not a point estimate, where positives are few>> | <<FILL: small-n fragility>> | Interval or resampling |
3. Metric selection guide
Choose the primary metric from the consequence of the error you can least afford, and state why.
- Recall (sensitivity): prioritize when a missed positive is the dangerous failure (a missed critical deviation).
- Precision: prioritize when a false positive is costly or erodes trust.
- Specificity: prioritize when false alarms carry cost or volume.
- F1 or a composite: when precision and recall both matter.
- Calibration: required whenever the workflow acts on the confidence value.
Report the confusion matrix behind the metrics, not just the headline number.
4. Test-set and evidence requirements
- Three-way split. Training (the model learns), validation/tuning (hyperparameters and thresholds chosen), and a locked test set touched once, at the end, to report the acceptance numbers.
- Held out and version-controlled. The test set the model never saw, under version control, ideally drawn from a time period after the training data to mimic deployment.
- Reported honestly. Performance reported on training or tuning data is not acceptance evidence. Where positives are rare, report a confidence interval or a sensitivity check.
- Traceable. Each requirement ID maps to the test result that satisfies it.
5. Acceptance criteria for this specification
- The specification was written and approved before the model was trained (dated evidence).
- Every requirement has a threshold, a consequence-of-error justification, and an objective test method.
- The primary metric matches the error the intended use can least afford.
- The evidence is reported on a locked, held-out, version-controlled test set.
- Calibration and rare-subclass performance are specified where the workflow depends on them.
6. References
FDA guidance, Computer Software Assurance for Production and Quality Management System Software (draft September 2022; final 24 September 2025; current version issued 3 February 2026) for the risk-based, intended-use-driven approach. GAMP 5 Second Edition (ISPE, 2022) for the risk-based lifecycle (reference by title; describe, do not paste). ICH Q9(R1), Quality Risk Management, for sizing evidence to risk. 21 CFR Part 11 and EU GMP Annex 11 for the electronic-record controls around the model’s outputs. Track the draft EU GMP Annex 22 (Artificial Intelligence, 2025) and the FDA January 2025 draft on AI for regulatory decision-making; confirm status before relying on either.
Confirm the current version and clause numbers of each reference before issue.
7. Revision history
| Version | Date | Author | Summary of change |
|---|---|---|---|
<<FILL: 1.0>> | <<FILL: date>> | <<FILL>> | Initial specification, approved before training. |
8. Approvals
| Role | Name | Signature | Date |
|---|---|---|---|
| Author (System Owner) | <<FILL>> | ||
| Data Science | <<FILL>> | ||
| QA | <<FILL>> |
Filled specimen
The following shows the specification completed for an illustrative deviation-screening model. Numbers are illustrative.
Intended use: the model screens incoming deviations and flags those most likely to be critical, for prioritized human review; a QA reviewer makes the final classification. Risk class: advisory/screening.
| ID | Requirement | Threshold | Test result |
|---|---|---|---|
| URS-PERF-001 | Recall (missed critical is the dangerous failure) | recall >= 0.90 | 0.90 (90 of 100 critical caught) |
| URS-PERF-002 | Precision (false positives go to human review) | precision >= 0.70 | 0.75 (90 true of 120 flagged) |
| URS-PERF-003 | Calibration | stated vs observed within 0.10 per band | within 0.06 |
| URS-PERF-004 | Recall on the “critical safety” subclass | >= 0.95 | 0.96 |
Confusion matrix on the locked test set (1,000 deviations, 100 genuinely critical): 90 true positives, 10 false negatives, 30 false positives, 870 true negatives, giving recall 0.90, precision 0.75, specificity 0.97, F1 0.82. The specification was approved and dated before training, the test set was held out and touched once, and the primary metric (recall) matches the error the use case can least afford, a missed critical deviation. The 30 false positives are absorbed by the human review step by design.
Common inspection findings this specification prevents
- Acceptance criteria written after the test, fitted to the result the model achieved.
- Performance reported on training or tuning data, so the headline metric is inflated.
- A single “accuracy” number that hides poor performance on a rare critical subclass.
- No calibration evidence for a workflow that routes by confidence.
- No confidence interval on a metric estimated from a handful of positives.
How to adapt this specification
- Write and approve it before the model is trained; the date matters.
- Pick the primary metric from the error you can least afford and justify the threshold against that consequence.
- Define the three-way split and lock the test set under version control.
- Add calibration and rare-subclass requirements wherever the workflow depends on them.
- Map each requirement ID to the test result that will satisfy it in the validation report.