Data drift: a risk-based and gamp-aligned approach
Why it matters: LLMs can fall out of spec without any code change—because the inputs, policies, or real-world tasks evolve. That’s data drift. In GxP, we handle it with a continuous, risk-based approach: define intended use → set acceptance criteria → monitor → re-validate on triggers.
1) Define the context of use (CoU)
State exactly what the model may influence and the allowable autonomy (draft-only, HITL required, blocked actions). Tie it to process/scientific risk.
Example (Deviation/CAPA assistant): Suggests categories using the approved ontology; HITL required; never commits system-of-record changes.
2) Set acceptance criteria up front
Pre-register the bar so you know when drift matters.
Coverage/accuracy (gold set): ≥ 90–95% top-k on SME-labeled cases
Safety: 0% prohibited actions
Traceability: ≥ 95% of suggestions include source/rule citation
Contradictions/hallucinations: ≤ 1% on spot checks
Ops KPI: −30–50% time-to-first-draft; rework ≤ 10% needing >1 revision
3) Know the drift you’re watching for
Input/format drift: new document types, vendors, equipment, templates
Concept drift: updated taxonomy, new CAPA rules, new SOPs
Prior/frequency shift: distribution of cases changes (e.g., more of type X)
4) Monitor and act on triggers
Treat re-validation as triggered and proportional to risk.
Periodic review: keep a light cadence (e.g., quarterly) even without triggers.
5) Minimal evidence pack (inspection-ready)
CoU & allowable autonomy
Risk register (what can go wrong; key slices)
Acceptance criteria & test plan (pre-registered)
Gold set + results (with ALCOA+ lineage)
Monitoring plan + trigger log
Change-control entries (what changed, why, evidence)
6) Worked micro-example (new equipment type)
A new controlled rate freezer goes live → input drift.
Add a representative “equipment-X” slice to the gold set.
Re-run evals; require ≥ 92% top-k, 0% prohibited actions, ≥ 95% citation coverage.
Don’t enable suggestions on equipment-X until the slice meets the bar.
Update CoU, risk register, and change-control record.
Compatibility note: I run a continuous, risk-based lifecycle and map evidence to the CSA/GAMP guidance.
