After the first two articles in this series explained the basics, framework conditions and general planning for risk management in medical device development, this article addresses a specific aspect of the risk management plan: the definition of risk acceptance criteria by the manufacturer.
Why an entire article just on this topic? Because risk acceptance criteria are an integral part of risk management and ultimately provide the answer to the provocative question ‘How many deaths per year associated with the use of your medical device are you as a manufacturer willing to accept?’ The spontaneous answer to this question will probably always be: ‘None!’. Although this is a noble request, it cannot be realised in practice for many medical devices and is therefore not conducive to the formulation of risk acceptance criteria. It is therefore better to rephrase the initial question as follows: ‘How many deaths per year in connection with the use of your medical device do you as the manufacturer have to accept due to the inherent risks of use without a negative risk-benefit ratio arising?
Normative requirement
The development of risk acceptance criteria must fulfil the requirements of ISO 14971:2019, Chapter 4.4 d). According to ISO/TR 24971:2020, chapter 4.4.5, the standard emphasises the following:
– The derivation of risk acceptance criteria shall be based on the manufacturer’s policy for the definition of acceptable residual risk. This includes situations in which the probability of occurrence of the damage cannot be estimated and the risk assessment is based only on the severity.
– The risk acceptance criteria can be standardised for several similar medical devices or a family of medical devices.
– The risk acceptance criteria must be defined BEFORE the risk assessment begins in order to protect them from being influenced by the results of the risk assessment
Important for implementation
The following points represent tried-and-tested practical tips for the concrete implementation of the determination of risk acceptance criteria:
According to ISO 14971:2019, chapter 4.4, the risk acceptance criteria are part of the risk management plan (RMP). In many cases, however, they will be developed in the risk analysis template, as these specifications are utilised there directly and possibly automatically. Conversely, as the RMP (including the risk acceptance criteria) must be approved before the risk analysis, it makes sense to transfer the risk acceptance criteria, including the most important considerations and specifications, to the RMP and to define this part of the risk analysis by approving it. This preserves the temporal logic of the events without jeopardising the practical advantages of automated processing of the risk acceptance criteria.
Once defined, the risk acceptance criteria must not be changed in the further course of development! If this is unavoidable due to new literature or new findings, all risks assessed up to this point must be reassessed!
A 5×5 matrix has proven to be a good compromise between granularity and controllability for defining the individual levels of severity and probability of occurrence. In the following, it is assumed that P1 denotes the level of the lowest probability of occurrence (‘hardly conceivable’) and P5 the highest (‘with every application’) Analogously for the degree of severity: S1 for hardly noticeable impairments and S5 for death of the patient, user or third party.
First things first
Before actually determining the risk acceptance criteria, you should think about the characterisation of the individual classes for severity and probability of occurrence. While the characterisation of severity is purely qualitative, i.e. by describing the expected injuries or damage as precisely as possible, the aim should always be to provide quantitative information on the probability of occurrence in addition to the qualitative description. The following section describes how this can be achieved either by studying the sources or, if the sources are insufficient, by making sensible estimates. The qualitative description of the probability of occurrence makes it easier to assign one’s own experience to the classes of probability of occurrence later in the discussions. In the same way, the characterisation of the degrees of severity later enables their allocation to specific damage that can occur in connection with the product to be developed.
An exemplary qualitative categorisation of the severity classes could be as follows:
Sources for defining the risk acceptance criteria
For the concrete formulation of risk acceptance criteria, it is necessary to specify the boundary line between unacceptable and acceptable risks on the basis of valid data. Possible sources for this data are
– published standards
– Scientific or technical studies
– Field data from similar medical devices already in use, including publicly available incident reports (e.g. BfArM, MAUDE database, in future also EUDAMED)
– Usability tests with typical users
– Clinical evidence
– Results of relevant studies or simulations
– Expert opinion
– External quality assessment procedures for in vitro diagnostic medical devices
Transfer to probability of occurrence
If at all possible, you should always aim to quantise the limits of the P classes. P1 always starts at 0 and P5 ends at 1 (100%). All limits in between must be determined individually. Ideally, the above methods and sources can be used to determine a state of the art that describes the maximum acceptable frequency of life-threatening damage. In most cases, only one event from class P1 will be declared acceptable for this degree of severity in order to have the possibility of mitigating the risk into the acceptable range. The maximum acceptable frequency for this severity level thus describes the limit between P1 and P2, or more precisely the lower limit of the P2min range of P2). All events with a probability of P<P2min can then be assigned to class P1. For a new device, no deterioration should occur, but an improvement should be aimed for. P2min should therefore tend to be set lower than in the state of the art. This can be transferred analogously to the PXmin of the other P classes: This process is transferred to each severity level and the dividing line between acceptable and unacceptable risks can be determined by the accumulation of exemplary risks. By assigning the probabilities of occurrence associated with these risks, the individual PXmin of the other P classes can then be determined.
Attentive readers will have noticed that, when categorising the degrees of severity as in the example above, the criterion mentioned leads to the limit for the acceptable probability of occurrence being sought for the second-highest S class, not the highest. The background to this is that direct death as an injury fortunately occurs rather rarely, but accordingly the data situation for such an event will not be as good. This is different in life-threatening situations, as in many cases death can still be averted. An example of such a situation is the induction of ventricular fibrillation by an implanted cardioverter defibrillator (ICD) as a result of a false positive shock delivery. This is certainly a life-threatening situation, but in most cases it can be resolved by delivering another shock. Such events therefore occur much more frequently, are better described in the literature and are therefore more suitable for quantifying the acceptable probability of occurrence.
What to do if there are insufficient sources?
The question now arises as to what to do if research into acceptance criteria has not yielded the desired results, e.g. because the product to be developed is so new that there are no precedents or comparable products? In this case, it is necessary to return to the question at the beginning of this article and ask yourself how many times a year you are prepared to deal with a death or a life-threatening situation in connection with the use of your own device. This question will (have to) be answered with higher values for manufacturers of products with an inherently high risk (e.g. defibrillators or heart-lung machines) than for manufacturers of products with a low risk (e.g. blood pressure devices) and can of course also be ‘once in ten years’.
The following approach can be used to determine usable limit values for the P classes:
1. determine the service life of the product (time of use in the field) in years
2. estimate the number of applications per device and year
3. estimate the duration of an application in hours
4. estimate the number of devices sold per year
5. now there are various ways to arrive at a reference value for the P1/P2 limit:
a. Number of events during the lifetime of a device: This corresponds to the number of devices sold per year
b. Number of events over the entire appliance collective in the field: This corresponds to the product of the number of appliances sold per year and their service life
c. Number of events across all applications per year: Multiply the number of devices in the field by the number of applications per device per year
d. Number of operating hours for all devices per year: Multiply the number of all applications per year by the application duration
The P1/P2 limit for severity S2 is then calculated as the product of the answer to the initial question and the reciprocal of the reference value determined above. This also makes it clear that the probability of occurrence becomes smaller the more detailed the above estimate becomes and reaches its minimum value in the example with the calculation of the total operating hours.
Analogue to this procedure, limit values for the probability of occurrence can also be determined or estimated for the other severity levels.
Incidentally, this procedure can also be used if there are sufficient sources within the framework of market observation in order to be able to check that the feedback from the field behaves as expected or whether there are deviations that should be investigated in more detail.
Risk acceptance matrix
Overall, the above steps result in the risk acceptance matrix, as shown below as an example. The risk acceptance criteria are formed by the boundary lines (shown in thick colour below) between the areas for which the risk is classified as acceptable and the areas for which it is classified as unacceptable.
Legend:
N=Not acceptable
A=Acceptable (‘as-low-as-possible’ range)
If possible, a single risk acceptance matrix should be defined in the project for the purpose of risk comparability.
Dealing with special risks
There are rare cases of medical devices in which this concept reaches its limits, namely when product-specific risks have strikingly different probabilities of occurrence than other, less specific risks. One example of such a constellation is the detection algorithm of an automated external defibrillator (AED). Despite the very high quality of the algorithms, in practice there are residual risks of one per mille to one per cent that the AED will make an incorrect decision about the presence of a rhythm requiring defibrillation during the analysis. The particular standard IEC 60601-2-4:2010+A1:2018 even only requires 90% sensitivity, i.e. up to 10% false negative shock recommendations would still be okay. There are no other risk-minimising measures beyond designing the algorithm to be as reliable as possible. Compared to the probability of occurrence that is considered acceptable for other life-threatening risks (e.g. in connection with electrical safety), the probability of a false positive or false negative decision by the detection algorithm is typically 100 to 1000 times higher. Now you have the choice between plague and cholera: Either you push the acceptance limit for the probability of occurrence so high that the residual risks of the detection algorithm are acceptable and thus distort the entire matrix in such a way that many risks that are actually unacceptable are easily within the acceptable range. This can lead to these risks not being mitigated as actually required.
At first glance, the alternative seems sobering: if the acceptance threshold is selected on the basis of the more frequently occurring risks, the risks for a false negative or a false positive shock decision remain unacceptably high with no chance of sufficient mitigation. Does that mean the end for such a product?
Fortunately, no, because now it’s time for chapter 7.4 of ISO 14971:2019: The risk-benefit analysis. According to this, a manufacturer can weigh up risks that are classified as unacceptable and for which further risk control is not possible against the benefits of the device according to its intended purpose on the basis of relevant data and literature. If the benefits are recognisably greater than the risks, a positive assessment of the risk-benefit ratio is still possible. Of course, this decision must be justified and substantiated in detail. Despite these more difficult conditions of proof, the honest categorisation of the risks as ‘unacceptable’ in accordance with the general risk acceptance criteria therefore appears to be the better approach. In the specific case study of the AED, this reasoning applies: as defibrillation is the only life-saving therapy for ventricular fibrillation, the risks of an incorrect shock decision are acceptable despite their magnitude.
Conclusion and perspective
Defining risk acceptance criteria is a key step in risk management that requires intensive research into data and literature. If necessary, the criteria can also be derived from reliable assumptions and estimates. The number and type of classes for severity and probability of occurrence are defined in advance. The risk acceptance criteria may not be changed in the further course of development or all risks must be completely revised.
Once this important step has been completed, you can turn your attention to identifying and evaluating the risks. The next article in this series will deal with the associated activities and methods.
Please note that all details and listings do not claim to be complete, are without guarantee and are for information purposes only.