Module 10

Likelihood, Severity and the Risk Matrix Problem

Advanced60 to 90 min

Risk matrices under-specify likelihood by treating it as a single score, which fails for DMHT's sociotechnical harm pathways. The problem is structural across standards. Decomposing likelihood into exposure, conversion and control effectiveness keeps your ratings defensible and your hazard library clean.

Common issues: CSOs score likelihood confidently in the matrix but cannot articulate what they are actually estimating, or explain to a reviewer why the residual score is lower than the initial one.

Progress: 0 of 12 sections complete0%

Framing note

Educational content only — not regulatory advice

This module is educational and does not constitute formal regulatory or clinical safety advice. It uses the ISO 14971 (1–25) and DCB0129 (1–5 NHS Risk Level) scales explicitly separated, consistent with the rest of the academy.

Use it alongside your own organisation's risk management procedure, not in place of it.

This is not a DCB0129 problem. It is a risk-matrix problem.

Mark complete

Risk matrices have two axes and both are more complex than they appear.

Likelihood collapses three independent probabilities into one number: how often a user encounters the hazard, how likely that encounter converts to harm, and whether controls interrupt the pathway. Severity collapses three distinct dimensions into one label: the nature of the clinical harm, its reversibility, and the scale at which it occurs.

One of those severity dimensions, scale, overlaps structurally with the exposure component of likelihood. When scale is carried inside the severity rating, it is counted once in severity and again in likelihood if the CSO is rigorous about Component A.

That is not a drafting error in DCB0129. It is a structural feature of any matrix that tries to capture population-level consequences inside a per-hazard severity label.

Once you see this structure clearly, the DCB0129 severity table becomes legible rather than confusing.

The same analytical clarity applies to likelihood. DCB0129 tells you to score it but does not tell you what it is a probability of.

This module makes both axes explicit.

Risk matrices were designed for industrial process safety, where harm pathways are short and largely mechanical.
DMHT harm runs through user psychology, clinical vulnerability and design architecture simultaneously — multiple probabilistically independent steps, not one.
Collapsing that chain into a single likelihood score does not eliminate the underlying probabilities. It hides them, and produces risk files that look complete but are analytically hollow.
Rewriting DCB0129 would not fix this. The matrix would still be a matrix and the simplification would still apply. The remedy is to decompose the analysis underneath the matrix, not to redraft the standard.

What likelihood is actually measuring

Mark complete

Likelihood is not a property of the hazard in the abstract. It is the probability that the hazard leads to harm in a specific deployment context, before additional controls are applied.

Three probabilistically independent components are collapsed into one number — and each answers a different kind of question, owned by a different discipline.

Decomposing likelihood into three independent probabilities

Component A — ExposureHow often the hazard is encountered.

Component B — ConversionP(harm | hazardous situation).

Component C — Control efficacyDo controls interrupt the pathway?

Residual likelihoodOnly Component C moves between initial and residual.

Component A — P(hazardous situation arises). AKA exposure. A product usage and design question.

Answerable from product analytics and architecture review: how often does a user actually encounter the feature or failure mode?
A variable-ratio reinforcement schedule embedded in a daily-use product creates a hazardous situation every interaction.
A training-data leakage problem creates a hazardous situation at the point of model deployment.
Same matrix, structurally different exposure frequencies. Population-level exposure belongs here — not in the severity column.

Component B — P(harm | hazardous situation). AKA conversion. A clinical and epidemiological question.

Answerable only with knowledge of the target population's vulnerability profile.
Not every user exposed to compulsive-use architecture develops compulsive use.
Not every user who receives a hallucinated crisis contact fails to verify it independently.
Conversion depends on user vulnerability, deployment context and protective factors outside the product.

Component C — P(controls fail to interrupt the pathway). AKA control efficacy. An engineering and clinical governance question.

Requires evidence that specific controls actually interrupt the harm pathway, rather than merely existing on paper.
Controls do not eliminate the hazard. They reduce the probability that the hazard propagates to harm.
This is the only component that should change between initial and residual likelihood scoring.
If you cannot name which component a control attacks, you cannot defensibly lower residual likelihood.

Self-assessment

Which of the three likelihood components is the only one that legitimately changes between initial and residual scoring?

Severity is a property of the hazard. Likelihood is a property of the product.

Mark complete

The component breakdown above tells you what likelihood is made of. The practical consequence is an asymmetry in how the two axes are scored.

Severity is assessed once for the hazard and carried unchanged across products: a missed suicidal crisis is catastrophic whichever product failed to detect it. Likelihood is not portable in the same way.

It must be re-derived for every product and deployment, because its three drivers are all product facts: exposure frequency (Component A), population vulnerability (Component B) and protective design (Component C). The three examples below show the same hazard moving in likelihood across deployments while its severity stays fixed.

Exposure frequency (Component A): a product with 10,000 daily active users where every session involves an LLM response has a higher exposure frequency for hallucination hazards than one used weekly for structured questionnaire completion.
Population vulnerability (Component B): the same AI-persona design hazard is more likely to produce dependency harm deployed to people with emotionally unstable personality disorder than to people with mild occupational stress.
Protective design (Component C): a product with no crisis detection and no escalation route has a higher likelihood that a missed crisis signal produces harm than one with a human-staffed escalation pathway, even where both share the same underlying hazard.

Self-assessment

Is severity primarily a property of the hazard or of the product?

The DCB0129 severity scale conflates two distinct things in one column

Mark complete

As mentioned before, severity is meant to be a property of the hazard independent of exposure. However, in the DCB0129 severity table (reproduced in the reference appendix below), the 'patients affected' column shifts the same clinical outcome — death, permanent life-changing incapacity — from Major to Catastrophic on the basis of how many patients are affected.

That column is doing two things at once, and the standard does not distinguish between them.

One thing the column captures is covert likelihood. The number of patients affected is partly a function of the product's exposure frequency and deployment scale: a product with 100,000 daily users has a higher probability of producing multiple simultaneous serious harms than a product used by 50 clinician-referred patients per month, even where the underlying hazard and mechanism are identical. That is exposure information and properly belongs in likelihood Component A.
The other thing it captures is a genuine severity dimension that only exists at population scale. A mass-harm event is not just 'more individual harms'. It triggers qualitatively different consequences: systemic failure of clinical infrastructure, public-trust collapse, regulatory crisis, and cascading effects on other patients who lose access to services. That is a real severity escalation, defensible as a severity input.
The problem is that DCB0129 fuses these two things into one column, so the contamination and the legitimate population-level escalation are both hidden inside the same Major-to-Catastrophic step. The matrix therefore is not a clean Likelihood × Severity product: it is Likelihood × (Severity + covert Likelihood + genuine population-severity), and the three components cannot be inspected separately.
Practical consequence: a manufacturer deploying a high-risk product at small scale can legitimately score hazards as Major rather than Catastrophic. The same product at scale would score Catastrophic, even though the individual-patient harm is unchanged. Part of that shift is exposure smuggled into severity; part of it is a real systemic-harm escalation. This is why DMHT products systematically understate catastrophic risk at small scale and then face reclassification pressure when they grow.

Self-assessment

Why does a product moving from 500 users to 500,000 users often face reclassification pressure even where the individual-patient harm is unchanged?

The compliance-optics tension

Mark complete

The two-part approach described above is intellectually cleaner, but it is not how NHS reviewers are trained to apply Table 1. Reviewers apply the table as a single integrated lookup, not as a two-step process that separates individual-patient severity from population-scale severity and routes exposure into likelihood.

A risk file built on the refined approach will produce severity ratings that differ from what a reviewer expects to see, which creates a compliance-optics problem even where the underlying reasoning is superior. Be honest about this: the intellectually correct approach and the approach most likely to pass an NHS reviewer without triggering queries are not always the same thing.

Pragmatic option: apply Table 1 as written, but annotate the hazard log to record which part of the severity rating reflects individual-patient harm, which part reflects exposure-driven scale, and which part reflects genuine population-level systemic harm. The reviewer sees the standard rating; you retain the analytical separation for internal use and for any later defence of the score.
Reformist option: score severity on an individual-patient basis, carry exposure in likelihood Component A, and add an explicit population-severity factor for systemic and infrastructure consequences. Expect to defend the deviation in a covering note that maps your scoring back onto Table 1 cells so the reviewer can reconcile it.
Either path is defensible. The mistake to avoid is using the refined approach silently — producing scores that diverge from Table 1's expected outputs with no explanation, which reads to a reviewer as an inconsistent application of the standard rather than a more rigorous one.

Is it ok to use a hazard library?

Mark complete

A hazard library is a structured collection of pre-specified hazards that a CSO uses as a starting point for a product-specific risk assessment, rather than identifying hazards from scratch each time. Hazard libraries exist because many DMHT hazard categories are knowable in advance: the harm from a missed suicidal crisis, the mechanism of parasocial dependency, the severity of a hallucinated clinical recommendation are not product-specific facts but properties of the hazard itself, and a well-constructed library specifies them once so they do not need to be rederived for every assessment.

The analytical problem arises at the boundary between what the library can legitimately specify and what only the CSO can determine for a specific product. The library can specify everything that is true of the hazard regardless of which product is being assessed, but it cannot specify likelihood, because likelihood depends on how often this specific product exposes this specific population to this specific hazard, how vulnerable that population is to the harm pathway, and whether this product's controls actually interrupt it.

Those are product facts, not hazard facts. A risk file that copies likelihood values from a library without product-specific adjustment is not a risk assessment; it is a formatting exercise.

If severity (to an individual patient) is a property of the hazard and likelihood is a property of the product, a hazard library can only do half the job. The other half belongs to the CSO completing the assessment for a specific product.

Field	Hazard-intrinsic	CSO-assigned per product
Hazard description	Yes	Inherit unchanged
Harm description	Yes	Inherit unchanged
Mechanism / causal chain	Yes	May refine for product specifics
Suggested controls	Yes	Select, extend, justify omissions
Severity (S1–S5, individual-patient basis)	Yes — hazard-intrinsic	Inherit unless the product changes the harm category
Initial likelihood (L1–L5)	Optional floor or default only	CSO assigns based on exposure, vulnerability and protective design
Residual likelihood (L1–L5)	No	CSO assigns and must name which component the controls attack
Residual risk acceptability argument	No	CSO writes against the benefit-risk balance for this product and population

Self-assessment

Can a hazard library validly publish a single likelihood value that every adopting product carries through to its risk file?

Anchor descriptions for L1–L5 calibrated to DMHT

Mark complete

Generic ISO 14971 likelihood anchors are written for industrial and hardware contexts and do not map cleanly to digital mental-health hazards. The anchors below are DMHT-calibrated, drawing on whichever evidence is appropriate for the hazard — usage analytics, offline and live model evaluation, monitoring telemetry, or population data.

L1 — Improbable

The hazard is present in the product design or model but rarely encountered in normal use; fewer than 5% of active sessions involve exposure; the deployment population has no documented vulnerability for this harm pathway; and where the hazard involves model outputs, the failure mode has been observed only in offline evaluation at a rate below the deployment monitoring threshold.

L2 — Remote

The hazard is reachable in normal use but only through non-default pathways or at a low observed rate in live monitoring; population vulnerability is plausible but not concentrated in the deployment cohort; existing controls catch most occurrences before harm propagation.

L3 — Occasional

The hazard arises in standard use for a meaningful subset of users; the deployment population includes at least one subgroup with documented vulnerability for this harm pathway; where the hazard involves model outputs, the failure mode occurs at a rate comparable to the monitoring threshold and controls reduce but do not reliably interrupt the pathway.

L4 — Probable

The hazard is encountered in most sessions or is reproducible in live use; the deployment population includes users with known vulnerability for this harm pathway; existing controls address consequences but do not reliably prevent the hazard from arising.

L5 — Frequent

The hazard is central to how the product operates and is encountered in every session or observable at a rate well above monitoring thresholds; the deployment population includes users with documented vulnerability for this harm pathway; no effective control is present in the current design.

Anchor descriptions for S1–S5 (individual-patient basis)

Mark complete

Severity anchors aligned to DMHT clinical consequence, scored against the harm to an individual patient. These do not change with deployment context — the same anchors apply across products, because severity is hazard-intrinsic.

Population-level exposure is captured separately in likelihood Component A.

S1 — Minor

Transient inconvenience, no clinical impact, no care escalation required.

S2 — Minor clinical

Short-lived distress or symptom worsening; resolves without clinician input.

S3 — Moderate

Clinically significant deterioration requiring routine clinical input; no lasting harm.

S4 — Major

Serious clinical deterioration; admission, crisis-team contact or sustained functional impairment; recovery expected but not immediate or complete.

S5 — Catastrophic

Death, permanent severe harm or substantial irreversible deterioration to an individual patient.

Initial vs residual scoring — what actually moves

Mark complete

Apply the rule from Section 2 in practice: residual likelihood moves through Component C. Exposure (A) and conversion (B) are product-design facts and stay put unless the control redesigns the interaction or restricts the intended population — and a population restriction is an indication change, not a control.

If residual likelihood drops without an identified change to A, B or C, the score is unsupported.
A warning banner or consent screen does not change exposure or vulnerability and only marginally affects Component C.
If the control restricts the intended user population, write that as an indication change in the Intended Purpose, not as a likelihood reduction.
Document, against each residual score, which component the control attacks and the evidence that the attack is effective.

Worked illustration

Mark complete

Worked illustration — Kova internal-policy example

Kova is an internal company-policy worked example used across the academy. It does not represent a real product.

The same hallucinated-crisis-contact hazard is scored across two hypothetical deployment configurations to show how the library's severity is fixed (on an individual-patient basis) while the CSO's likelihood moves.

Hazard: hallucinated crisis contact returned to a user in distress. Library severity: S5 on an individual-patient basis (potential catastrophic outcome if the user acts on incorrect information during a crisis). Deployment scale does not change this — it is captured in Component A below.
Configuration 1 — DTC standalone, no human escalation: Component A high (every distress-flagged session can trigger the pathway, large user base), Component B concentrated (population vulnerability), Component C controls limited to output filtering. Initial likelihood L4; residual L3 after a verified-source lookup is added.
Configuration 2 — clinician-mediated, human escalation pathway, smaller referred cohort: Component A lower (escalation intercepts most cases before output is acted on; smaller exposure base), Component C controls effective. Initial likelihood L3; residual L2 after the same verified-source lookup, because the escalation pathway materially attacks Component C. The user is not left alone with the potentially hallucinated contact. The clinician can verify information, provide the correct resource, or handle the crisis directly. Therefore, the number of times a user is actually exposed to the bad AI output during a real crisis is significantly lower. Even if the AI still sometimes hallucinates, the effective exposure to the hazard (i.e. the user seeing and potentially acting on the fake contact) drops because of the human safety net.
Same hazard, same individual-patient severity, two different residual Risk Levels — driven by the product, not by the hazard.

Common failure modes when scoring

Mark complete

Likelihood values copied from a hazard library without product-specific adjustment. The library may publish a floor or default; the operative score must be assigned by the CSO based on this product's exposure frequency, population vulnerability, and protective design.
Letting deployment scale drift into the severity rating via the 'patients affected' column instead of carrying it in likelihood Component A.
Claiming a control reduces likelihood when in fact it has not changed how often users encounter the feature.
Treating disclaimers, consent screens and onboarding warnings as likelihood reducers. They shift liability; they do not change exposure or vulnerability.
Scoring residual likelihood lower than initial without naming which of A, B or C the control attacks.
Using generic ISO 14971 anchors written for industrial or hardware contexts when scoring behavioural-design and AI-output hazards.
Hiding a population restriction inside a 'control' instead of declaring it as an indication change in the Intended Purpose.

How this fits with the rest of the academy

Mark complete

Module 7 (ISO 14971).
Module 8 (AAMI TIR34971).
Module 9 (NHS DTAC).

Framing note

Appendix B — DCB0129 severity categories (reference)

Reference table based on DCB0129 Table 1 — included here so you can see where the 'patients affected' column smuggles exposure into severity (Section 4). Verify against the current published standard before relying on the exact wording.

Severity	Interpretation	Patients affected
Catastrophic	Death	Multiple
Catastrophic	Permanent life-changing incapacity and any condition for which the prognosis is death or permanent life-changing incapacity; severe injury or severe incapacity from which recovery is not expected in the short-term	Multiple
Major	Death	Single
Major	Permanent life-changing incapacity and any condition for which the prognosis is death or permanent life-changing incapacity; severe injury or severe incapacity from which recovery is not expected in the short-term	Single
Major	Severe injury or severe incapacity from which recovery is expected in the short-term	Multiple
Major	Severe psychological trauma	Multiple
Considerable	Severe injury or severe incapacity from which recovery is expected in the short-term	Single
Considerable	Severe psychological trauma	Single
Considerable	Minor injury or injuries from which recovery is not expected in the short-term	Multiple
Considerable	Significant psychological trauma	Multiple
Significant	Minor injury or injuries from which recovery is not expected in the short-term	Single
Significant	Significant psychological trauma	Single
Significant	Minor injury from which recovery is expected in the short-term	Multiple
Significant	Minor psychological upset; inconvenience	Multiple
Minor	Minor injury from which recovery is expected in the short-term; minor psychological upset; inconvenience; any negligible consequence	Single

Framing note

Appendix C — DCB0129 likelihood categories (reference)

Likelihood categories aligned to DCB0129 Table 2. DCB0129 defines likelihood qualitatively and does not attach numeric probability bands; the categories below collapse Components A, B and C into a single number, which is precisely the simplification Section 2 decomposes.

Likelihood	Score	Definition
Very high	5	Certain or almost certain to occur.
High	4	Not certain, but very possible; reasonably expected to occur in the majority of cases.
Medium	3	Possible; may occur in some circumstances.
Low	2	Could occur, but in the great majority of cases will not.
Very low	1	Negligible or near-negligible possibility of occurring.

Framing note

Appendix D — DCB0129 risk acceptability criteria (reference)

Reference thresholds based on DCB0129 — included so the look-up the matrix feeds is visible alongside the analysis; these criteria tell you what to do once the cell is selected, but they do not protect against the upstream issues addressed in Sections 1–4.

Risk Level	Acceptability criteria
5	Unacceptable — mandatory risk elimination required. Must implement multiple control measures or eliminate the hazard entirely.
4	Unacceptable — mandatory elimination of hazard or addition of control measures to reduce risk to an acceptable level.
3	Undesirable level of risk. Mandatory risk reduction required. Must eliminate hazard or implement control measures to reduce to level 2 or below.
2	Acceptable level of risk. Risk accepted where further risk reduction is impractical, with documented justification for why reduction is not reasonably achievable.
1	Acceptable, no further action required.

Exercises

Form your own view first. Reveal the reference answer to compare reasoning.

Exercise 1 — Three products, same severity, different likelihood

Three DMHT products share the same hazard: missed suicidal ideation in user input. Severity is S5 (individual-patient basis) for all three. Which configuration carries the highest initial likelihood, and which of the three likelihood components drives the ranking?

Highest initial likelihood

Pick an option for each question to compare against the reference answer.

Exercise 2 — The warning-banner fallacy

A CSO drops residual likelihood from L4 to L2 after adding an onboarding warning banner that says 'This product is not a substitute for professional care. In a crisis, call 999 or the Samaritans.' Which of the following best describes the score?

Assessment of the residual score

Pick an option for each question to compare against the reference answer.

Exercise 3 — Splitting the library from the assessment

A hazard-library entry contains: hazard, harm, mechanism, suggested controls, severity (S4 on an individual-patient basis), suggested initial likelihood (L3). An advisor is using the entry to assess a specific product deployed to an adolescent population with documented vulnerability for the harm pathway. Which fields can the advisor inherit unchanged, and which must the advisor override or assign?

Which statement is correct?

Pick an option for each question to compare against the reference answer.

Educational resource. Not formal regulatory or legal advice.