SOP: AI and Machine Learning Training Data Readiness Assessment

This is a ready-to-use SOP. Replace every <<FILL: ...>> placeholder with your own specifics, set your document numbers and dates, and route it through your normal document control, review, and approval. A worked filled specimen follows the template so you can see how a completed assessment reads. Verify each cited regulation against the current source before you rely on it.

Document control header

1. Purpose

This procedure defines how <<FILL: COMPANY NAME>> assesses whether a candidate dataset is fit to train, retrain, or run an AI or machine learning model used in a GxP-relevant workflow. The objective is to confirm, before any modeling begins, that the data is profiled, contextualized, truthfully labeled where applicable, governed under ALCOA+ and lineage controls, split without leakage, and checked for bias, so the eventual model rests on evidence that can be reproduced, investigated, and defended.

2. Scope

This procedure applies to structured and tabular data sourced from process historians, manufacturing execution and batch systems, laboratory information management systems, and quality management systems for training, retraining, or running an AI/ML model that influences a GxP decision, directly or through advisory output. It applies whether the model is built in-house or configured on a vendor platform using <<FILL: COMPANY NAME>> data. It does not cover the model build, testing, or validation itself, which is governed by <<FILL: SOP-ID for AI/ML validation>>, and it does not cover readiness of a text or document corpus for a retrieval-based generative assistant, which requires a companion procedure adapted from the freshness, redaction, chunking, and citation controls specific to that pattern.

3. Responsibilities

4. Definitions

• Intended use: the specific decision or output the model supports, and how much human review sits between the model and the decision. Drives how much readiness effort is warranted.
• Ground truth: the set of labels or outcomes the assessment team trusts enough to train a model against, established under a documented labeling procedure or drawn from an objective downstream outcome.
• Data lineage: the documented, traceable chain from a source record through every transformation to the value used in the dataset.
• Leakage: any way information from the test set, or from the future, reaches the training or tuning process, inflating reported performance.
• Readiness package: the collected evidence (profiling report, labeling record, feature specification, lineage log, split definition, representativeness analysis) that supports the QA sign-off in section 5.9.

5. Procedure

5.1 Define the intended use and the data it implies

1. Record the model’s intended use, the decision it supports, and whether output is advisory (human confirms) or acts with less oversight.
2. From the intended use, identify which source systems and record types the model needs.
3. Record the intended-use statement on the readiness assessment record (section 8).

5.2 Inventory sources and assign data owners

1. List every source system feeding the dataset (historian, MES, LIMS, QMS, or others) and name a data steward for each.
2. Confirm each steward has visibility into that source’s known quality issues, outages, or migrations that could affect the extract period.

5.3 Profile the data quality dimensions

1. Profile the candidate extract for completeness, accuracy, consistency, timeliness, uniqueness, validity, representativeness, and traceability.
2. Set acceptance thresholds for each dimension before profiling, tied to how load-bearing each field is, not a single fixed number applied everywhere.
3. Range-check numerics against the process specification, not the data’s own minimum and maximum, since the data may already contain the errors being screened for.
4. Document every flagged gap and its planned remediation or documented acceptance.

5.4 Contextualize and define features

1. Anchor raw values to their batch, phase, and process context using the execution record (MES or equivalent).
2. Define each engineered feature in a written feature specification: its source, its transformation, and confirmation it is reproducible.
3. Confirm the exact feature logic used to build the dataset is the same logic that will run at inference time.

5.5 Assess and establish ground truth

1. Determine the label source: existing structured records, purpose-built expert labeling, or an objective downstream outcome.
2. If labels come from a labeling program, confirm it runs under <<FILL: SOP-ID for AI/ML ground-truth labeling>> and that inter-rater agreement has been measured against a pre-set threshold.
3. Record the label version used and its agreement statistics on the readiness assessment record.

5.6 Assess representativeness and bias

1. Tabulate coverage of the dataset against the population the model will face in production (products, sites, instruments, operating regimes).
2. Quantify class balance for the target variable and select evaluation metrics that are not fooled by imbalance.
3. For any under-represented subgroup that is important to the intended use, decide to remediate (collect more data) or explicitly scope it out of the intended use, and record which.

5.7 Apply ALCOA+ and lineage controls

1. Keep the source data in its governed system; the extract is a documented true copy, never the only copy.
2. Capture the extract query, parameters, and date so the extract can be reproduced.
3. Build the cleaning and transformation pipeline as version-controlled, re-runnable code, not manual spreadsheet edits.
4. Document every exclusion with a reason on the dataset lineage and exclusion log (see the companion log, <<FILL: LOG-ID>> or dataset lineage and exclusion log).
5. Freeze the final dataset, generate a version identifier or hash, and retain it per <<FILL: retention period / SOP-ID>>.

5.8 Define the split and control leakage

1. Identify the grouping unit (batch, lot, or equivalent) and the time structure of the data.
2. Split by group to prevent the same batch or lot appearing in both training and test, and by time if the model runs forward in time in production.
3. Lock the test set, place it under version control, and use it once.
4. Fit every preprocessing transformation (scaling, imputation, feature selection) on training data only.
5. Audit each feature for availability at the moment of prediction; remove any that encode the outcome.

5.9 Assemble the readiness package and route for QA sign-off

1. Compile the profiling report, feature specification, labeling record, representativeness analysis, lineage and exclusion log, and split definition into the readiness package.
2. Route the package to QA for review against the acceptance criteria in section 6.
3. QA signs the readiness assessment record (section 8) before any modeling activity begins. Modeling that begins before sign-off is out of procedure and must be stopped and re-scoped.

5.10 Handle a “not ready” outcome

1. Where the package fails one or more acceptance criteria, QA returns it with the specific gaps identified.
2. The assessment team remediates (re-profile, re-label, re-source) and resubmits; do not proceed to modeling on a conditional or partial sign-off.
3. Record the remediation cycle on the readiness assessment record so the history of what was found and fixed is preserved.

6. Acceptance criteria

A readiness assessment is acceptable when all of the following are true:

• The intended use is defined and the data it requires is identified and sourced with named owners.
• Profiling against the quality dimensions is complete, with documented remediation or accepted, justified gaps.
• Features are contextualized, defined in a written specification, and confirmed identical between training and inference.
• Ground truth has a documented source; where labeling was used, inter-rater agreement meets the pre-set threshold and disagreements were adjudicated and recorded.
• Representativeness and bias have been assessed, with any under-covered subgroup either remediated or explicitly excluded from the intended use.
• ALCOA+ and lineage controls are in place: source preserved, extract reproducible, transformations version-controlled, exclusions documented, dataset frozen and versioned.
• The split is grouped and time-appropriate where relevant, the test set is locked and used once, and no feature leaks the outcome.
• QA has reviewed and signed the readiness assessment record before modeling began.

7. Records generated

• Readiness assessment record (section 8 of this SOP).
• Dataset profiling report.
• Feature specification.
• Labeling and inter-rater agreement record (where applicable).
• Dataset lineage and exclusion log.
• Split and leakage-check definition.

8. Record generated: readiness assessment record

9. References

21 CFR Part 11 (electronic records and signatures); EU GMP Annex 11 (computerised systems). FDA, Data Integrity and Compliance With Drug CGMP, Questions and Answers (Guidance for Industry, December 2018). MHRA GxP Data Integrity Guidance and Definitions; PIC/S PI 041, Good Practices for Data Management and Integrity. ICH Q9, Quality Risk Management (for sizing the assessment effort to intended use). GAMP 5 Second Edition (ISPE, 2022), for the computerized system lifecycle the readiness pipeline sits within (reference by title, do not reproduce clauses).

Confirm the current version and clause numbers of each reference before issue.

10. Revision history

11. Approvals

Filled specimen

The following shows the readiness assessment record completed for an illustrative model that classifies incoming deviations into criticality tiers to support, not replace, QA triage. The company, systems, and numbers are illustrative; replace them with your own.

The scoped exclusion of Product Line C is stated on the record rather than hidden. That is the disposition an inspector expects to see: a documented limit, not a silent gap discovered later.

Common inspection findings this SOP prevents

• Modeling began before any documented readiness assessment, so there is no evidence the data was fit for the intended use.
• A readiness claim exists but with no profiling report, feature specification, or lineage log behind it.
• Labels were used with no documented source or, where a labeling program applies, no measured inter-rater agreement.
• A known representativeness gap surfaces in production because it was never assessed or disclosed at readiness time.
• The dataset used to build the model cannot be reproduced because no version or hash was ever recorded.
• QA reviewed the model after it was built rather than the data before modeling began.

How to adapt this SOP

1. Set your document number, owner, and effective date in the header.
2. Point the cross-references in sections 2, 5.5, and 5.7 to your actual AI/ML validation SOP, ground-truth labeling SOP, and retention policy.
3. Adjust the source systems named in section 3 and section 5.2 to your real system inventory.
4. If your organization also builds retrieval-based generative assistants over a document corpus, issue a companion procedure for that pattern rather than stretching this one, since the readiness concerns differ (freshness and versioning, redaction, chunking, retrieval evaluation, and citation, rather than profiling, splitting, and leakage control).
5. Confirm every regulation in section 9 against the current published version before issue.