An ML-aware companion to the ISO 14971 module: training-data documentation, subgroup validation, drift monitoring, explainability, predetermined change control and human oversight — illustrated through Kova, a conversational AI on the NHS IAPT waiting list.
Common issues: Extending an ISO 14971 risk file cleanly to AI/ML — bias, subgroup performance, drift, opacity, change control and human oversight — without hand-waving.
This module assumes familiarity with the ISO 14971 risk management process. If you have not completed Module 7 (ISO 14971), start there first.
Worked examples in this module use Kova: a conversational AI companion for young people aged 16 to 25 on an NHS IAPT waiting list. Parents and carers can access a separate dashboard showing weekly mood check-in scores.
The AI adapts its responses based on conversation history and check-in data. No diagnosis.
No treatment decisions.
ISO 14971 was designed for medical devices with deterministic, predictable behaviour. An ML model does not behave that way: outputs are shaped by training-data patterns, behaviour may change over time, performance may differ across subpopulations, and full explanation may be impossible.
These characteristics create risk pathways ISO 14971 was not designed to capture. AAMI TIR34971 extends it to address them, and is increasingly expected by Approved Bodies at Class IIa+ for AI/ML products.
An ML model's outputs are shaped by its training data. If the training data does not represent the intended population, or contains systematic biases, the model will perform differently across subgroups in ways aggregate metrics may not reveal.
For DMHT this is clinically significant: a model trained predominantly on data from one demographic may provide systematically worse — or potentially harmful — outputs to users from underrepresented groups. AAMI TIR34971 requires documentation of data sources, inclusion/exclusion criteria, demographic and clinical characteristics, gaps relative to the intended-use population, bias assessment methodology, and mitigations.
If the product builds on a third-party foundation model, training data is often unknown or inaccessible. This does not remove the documentation requirement.
The manufacturer must document what is known, infer likely characteristics, identify subpopulations that may be under-represented, and design post-market surveillance to detect differential performance. Silence on this question will not pass review.
Kova's AI is a foundation model fine-tuned on 50,000 conversations between young people and mental-health support workers, collected from three London NHS trusts (2022–2024). 78% female-identified; 22% male or non-binary identified.
85% White British. Mean age 19.2.
62% mild–moderate depression, 31% anxiety, 7% comorbid. All conversations in English.
Known gaps: under-representation of male and non-binary young people, ethnic minority populations, severe depression / active suicidal ideation, non-English speakers. Bias assessment: model tested on held-out data stratified by gender, ethnicity, severity.
Significant performance differences identified — male-identified users rated AI responses as less relevant (42% vs 71% for female-identified); ethnic-minority users reported more frequent culturally incongruent responses. Mitigations: intended use restricted to English-speaking users; male-identified and ethnic-minority subgroups flagged as priority for expanded training data in v2.0; subgroup performance monitored in PMS from launch; clinical advisory review of responses in underperforming subgroups before release.
Aggregate model performance can mask significant subgroup differences. A model that performs well on average may perform poorly — or harmfully — for specific subpopulations.
For DMHT, clinically relevant subpopulations are defined by the intended-use population: condition severity, comorbidity, age, gender, ethnicity, and any other characteristic that could plausibly affect outputs. Performance differences must be clinically interpreted, not just statistically reported.
Underperformance in male-identified users is clinically significant: young men with low mood are a high-risk group with documented help-seeking barriers; a product that performs poorly here may reinforce those barriers. Severe-depression underperformance is critical — highest clinical risk, lowest performance.
Response: severe depression added to contraindications; male-identified and ethnic-minority subgroups flagged for priority training-data expansion in v2.0; PMS includes mandatory subgroup performance monitoring from launch with a threshold that triggers immediate review if any subgroup drops below 50% helpful.
Training data reflects the world at a particular point in time. As language, user behaviour, clinical practice and population characteristics change, the training data becomes less representative of current reality — this is data drift.
It can degrade performance gradually in ways aggregate metrics do not detect. For DMHT this is a specific clinical safety risk: a conversational AI trained on pre-pandemic mental-health language may respond less safely to post-pandemic presentations.
AAMI TIR34971 requires a documented plan for monitoring drift and degradation with defined thresholds for re-evaluation or retraining.
A model can produce consistently formatted outputs while clinical appropriateness, safety and helpfulness degrade. Adequate drift monitoring requires clinically meaningful metrics — user-rated helpfulness disaggregated by subgroup, engagement metrics, and comparison of input distributions over time.
Format consistency tells you nothing about whether the model is performing safely.
Many ML models — particularly large language models — are not fully explainable: a specific output cannot be traced to a specific input. This opacity creates a clinical risk pathway: if the model produces a harmful output, neither the clinician, the CSO nor the developer can fully explain why, making root-cause analysis and prevention of recurrence difficult.
AAMI TIR34971 requires the manufacturer to document the degree of opacity, its clinical implications for the use case, and the controls — human oversight, override mechanisms, transparency to users, and limits on the clinical scope of outputs.
Architecture: large language model fine-tuned on mental-health support conversations. Outputs are not rule-based and cannot be fully traced to specific training inputs — the model is substantially opaque.
Clinical implications: if Kova produces a response that escalates distress or provides clinically inappropriate advice, the specific cause cannot be identified by reviewing internal processing, limiting targeted correction. Controls: outputs labelled "Kova AI response"; one-tap feedback to flag responses as unhelpful or distressing; flagged responses reviewed by the CSO within five working days; a pattern triggers immediate model review.
Scope: Kova does not assess clinical risk, diagnose, or recommend treatment — any output that appears to do so is a model failure to be flagged.
Users act on suggestions. The explainability requirement is a risk management document, not a user communication.
User awareness of AI does not remove the obligation to manage opacity risk.
A PCCP specifies in advance which model changes are pre-approved, which require internal review, which require formal change-control assessment, and which would require a fresh conformity assessment. Originated by the FDA and referenced by IMDRF; as of May 2026, the MHRA does not yet have a formal equivalent requirement, but Approved Bodies are increasingly expecting some form of PCCP for adaptive AI/ML products at Class IIa+.
Without a PCCP every update could theoretically require a new conformity assessment, which is impractical.
AAMI TIR34971 requires manufacturers to specify human oversight and to implement override mechanisms allowing users or clinicians to reject or modify model outputs. The appropriate level depends on clinical significance and population vulnerability.
For DMHT products deployed direct-to-consumer without clinical supervision, oversight cannot be a clinician in the loop — the product must be designed to support informed user oversight: clear AI labelling, simple flagging, escalation to human support, and contraindications that remove the AI from the pathway in acute crisis.
The absence of a clinician transfers oversight responsibility to the manufacturer's design. AI outputs must be clearly labelled; there must be simple flagging, a documented escalation pathway, and a way to disengage safely from the AI interaction.
For DMHT products placing on the EU market or in Northern Ireland, the EU AI Act classifies health AI as high-risk under Annex III, triggering obligations including data governance, transparency, human oversight, accuracy and robustness, and post-market monitoring. These obligations overlap significantly with AAMI TIR34971 requirements: a manufacturer who has implemented TIR34971 rigorously will have addressed the substantive technical requirements behind most EU AI Act obligations.
The gaps are primarily procedural rather than technical: EU AI Act conformity assessment for high-risk AI systems requires registration in the EU AI database, a declaration of conformity, and interaction with a notified body that goes beyond what TIR34971 compliance alone produces. For dual-market products, the practical approach is to implement TIR34971 as the technical spine and then map the resulting documentation against EU AI Act Article 9 (risk management), Article 10 (data governance), Article 13 (transparency), Article 14 (human oversight), and Article 17 (quality management system) to identify procedural gaps.
EU AI Act implementation timelines and notified body designation for AI are evolving as of May 2026, and readers should verify current requirements before relying on this section for compliance purposes.
Form your own view first. Reveal the reference answer to compare reasoning.
A developer says their AI only needs standard ISO 14971 documentation because it makes suggestions, not clinical decisions. Is AAMI TIR34971 relevant?
A developer says they cannot document training-data bias because they used a third-party foundation model. How should this be handled?
A DMHT product performs significantly worse for severe-depression users than for mild–moderate. The developer keeps severe depression in the intended use and adds an onboarding warning. Acceptable?
A developer retrains Kova on six months of new user data with a higher proportion of severe-depression users than the original training population. They classify this as pre-approved (it's just retraining). Correct under the Kova PCCP?
A DTC conversational AI has no clinician in the loop. The developer concludes human oversight is not applicable. Correct?
Educational resource. Not formal regulatory or legal advice.
