This is a ready-to-use log. Replace every <<FILL: ...>> placeholder with your own specifics. A worked sample entry follows the field table so you can see the level of detail an inspector expects. Verify each cited regulation against the current source before you rely on it.
Document control header
| Field | Entry |
|---|---|
| Log title | AI Training Dataset Lineage and Exclusion Log |
| Document number | <<FILL: LOG-ID, e.g. LOG-AI-009>> |
| Version | <<FILL: version>> |
| Effective date | <<FILL: date>> |
| Owner | <<FILL: role, e.g. Data Engineering Lead>> |
| Dataset this log covers | <<FILL: model / dataset name>> |
Instructions
- Open one row-set of this log per dataset build or rebuild (initial build, each retrain on newly extracted data, or any material re-extraction).
- Complete every field at the time the action occurs, not retrospectively from memory; this log is itself a GxP record and is subject to the same contemporaneous expectation as any other.
- Every exclusion requires a documented, specific reason. “Removed for data quality” is not sufficient; state what was wrong and how it was identified.
- Freeze and hash the dataset only after all transformations and exclusions for that build are complete and logged.
- Retain this log for the life of the model plus
<<FILL: retention period>>, so the dataset can be reproduced or investigated for as long as the model, or any decision it influenced, remains relevant.
Field table
| Field | Format | Required | Who completes | When |
|---|---|---|---|---|
| Dataset build ID | Text, unique per build | Yes | Data Engineering | At the start of the build |
| Source system(s) | System name(s) and version | Yes | Data Engineering | At extraction |
| Extract query / parameters | Query text or script reference, with parameters (date range, filters) | Yes | Data Engineering | At extraction |
| Extract date | Date | Yes | Data Engineering | At extraction |
| Record count at extraction | Integer | Yes | Data Engineering | At extraction |
| Transformation step (one row per step) | Ordered step number, description, script/pipeline reference | Yes | Data Engineering | As each transformation runs |
| Exclusion (one row per exclusion batch) | What was excluded, count, documented reason, who approved | Yes, if any exclusion occurs | Data Engineering, with Process SME or QA concurrence for judgment-based exclusions | At the time of exclusion |
| Record count after each transformation / exclusion | Integer | Yes | Data Engineering | After each step |
| Final record count | Integer | Yes | Data Engineering | At freeze |
| Dataset version / hash | Version identifier and hash (for example SHA-256) | Yes | Data Engineering | At freeze |
| Freeze date | Date | Yes | Data Engineering | At freeze |
| Reviewed by | Name, role, date | Yes | Data Steward or QA | After freeze |
Retention
Retain this log for the life of the model plus <<FILL: retention period, e.g. per the records retention schedule SOP-ID>>. If the model is retired, retain the log for the same period applied to its last training dataset, so a retrospective investigation remains possible.
Acceptance criteria
- Every dataset build has a complete row-set: source, extract query and parameters, every transformation in order, every exclusion with a documented reason, and the final frozen version and hash.
- No exclusion is recorded without a specific, reviewable reason.
- Record counts reconcile at each step (extraction count minus each exclusion count equals the count carried into the next step).
- The frozen dataset version or hash matches what was actually used to train or run the model, verifiable on request.
- The log was completed at the time of each action, not reconstructed afterward.
References
FDA, Data Integrity and Compliance With Drug CGMP, Questions and Answers (Guidance for Industry, December 2018), for the Complete and Original expectations this log evidences. 21 CFR Part 11 and EU GMP Annex 11, for electronic-record controls over the log itself. MHRA GxP Data Integrity Guidance and Definitions; PIC/S PI 041 (reference by title; describe, do not paste).
Confirm the current version and clause numbers of each reference before issue.
Filled specimen
The following shows one build row-set completed for an illustrative bioreactor yield-prediction model. Numbers are illustrative; replace with your own.
| Field | Entry |
|---|---|
| Dataset build ID | YIELD-BUILD-2026-08-15 |
| Source system(s) | Process Historian PHX-3 v11.2; MES-EBR v4.0; LIMS v9.1 |
| Extract query / parameters | extract_yield_dataset.sql v1.3, params: batch_date 2023-01-01 to 2026-07-31, product = <<FILL: product family>>, site = Site A |
| Extract date | 08 August 2026 |
| Record count at extraction | 1,842 batches |
| Transformation step 1 | Resolve historian tag names to canonical dictionary (tag_map_v6.csv); no records dropped |
| Transformation step 2 | Align historian time series to MES phase boundaries; 3 batches with unresolvable phase alignment flagged for exclusion |
| Transformation step 3 | Join LIMS potency results, honoring censored values (no coercion of ”< LOQ” entries); no records dropped |
| Exclusion 1 | 3 batches with unresolvable phase alignment (transformation step 2), reviewed and concurred by Process SME (D. Alvarez), reason: historian gap during a known outage window 14-16 Mar 2024 |
| Exclusion 2 | 9 batches with dissolved-oxygen tag mapped to a decommissioned sensor never remapped, reason: representativeness hole identified during profiling, tag corrected in tag_map_v6 for future builds, these 9 excluded from this build only |
| Record count after exclusions | 1,830 batches |
| Final record count | 1,830 batches |
| Dataset version / hash | YIELD-2026-08-15-v1, SHA-256: <<FILL: hash value>> |
| Freeze date | 15 August 2026 |
| Reviewed by | M. Okoye, Data Steward, 16 August 2026 |
The two exclusions total 12 batches out of 1,842 (0.65%), each with a specific, investigable reason and a named reviewer, which is exactly what lets a future investigator or inspector understand why the dataset the model trained on is smaller than the raw extract, rather than wondering what else might have been quietly dropped.
Common inspection findings this log prevents
- A dataset was rebuilt for a retrain and no one can say what changed from the prior version.
- Records were excluded but the reason is missing, vague (“cleaned up”), or was never reviewed by anyone beyond the person who removed them.
- The record count in the final dataset does not reconcile with the extraction count and no log explains the gap.
- The dataset used to train the deployed model cannot be matched to a specific version or hash, so it cannot be reproduced for investigation.
- The log was filled in after the fact from memory once an inspector asked for it, rather than maintained contemporaneously.
How to adapt this log
- Set your document number, owner, and retention period in the header and retention section.
- Adjust the source system row to match your actual system inventory (historian, MES, LIMS, QMS, or others).
- If your pipeline is fully automated, have the pipeline itself write transformation and exclusion rows programmatically at each step rather than relying on a person to transcribe them, and state that automation in the “who completes” column.
- Decide who has authority to approve a judgment-based exclusion (one that depends on SME or QA opinion, not a hard rule) versus a rule-based one your pipeline applies automatically, and reflect that split in the exclusion field’s required approvals.
- Confirm every regulation in the references section against the current published version before issue.