A desirability of outcome ranking (DOOR) for periprosthetic joint infection – a Delphi analysis

Background: Treatment outcomes in studies on prosthetic joint infection are generally assessed using a dichotomous outcome relating to treatment success or failure. These outcome measures neither include patient-centred outcome measures including joint function and quality of life, nor do they account for adverse effects of treatment. A desirability of outcome ranking (DOOR) measure can include these factors and has previously been proposed and validated for other serious infections. We aimed to develop a novel DOOR for prosthetic joint infections (PJIs). Methods: The Delphi method was used to develop a DOOR for PJI research. An international working group of 18 clinicians (orthopaedic surgeons and infectious disease specialists) completed the Delphi process. The final DOOR comprised the dimensions established to be most important by consensus with >75 % of participant agreement. Results: The consensus DOOR comprised four main dimensions. The primary dimension was patient-reported joint function. The secondary dimensions were infection cure and mortality. The final dimension of quality of life was selected as a tie-breaker. Discussion: A desirability of outcome ranking for periprosthetic joint infection has been proposed. It focuses on patient-centric outcome measures of joint function, cure and quality of life. This DOOR provides a multidimensional assessment to comprehensively rank outcomes when comparing treatments for prosthetic joint infection.

1 Introduction

There has been minimal improvement in periprosthetic joint infections (PJIs) outcomes over the past 20 years by conventional measures (Xu et al., 2020b). Additionally, the absolute number of PJIs increases as the prevalence of people living with a joint replacement increase and arthroplasty numbers increase without a decrease in the incidence of infection (Manning et al., 2020). PJIs have variable treatment options, including debridement, one- or two-stage exchange arthroplasty and suppressive antibiotics, each with a spectrum of reported treatment success rates (Gehrke et al., 2015). Orthopaedic surgeons and infectious disease specialists often determine a preferred treatment course based on their best perceived outcome, but the most desirable outcome for the patient is often multifaceted and may have different priorities (Evans and Follmann, 2016). Treatment success has certain requirements, otherwise the outcome is reported as a failure (Diaz-Ledezma et al., 2013). Failure definitions for PJI treatment include recurrence of infection with the same organism, return to theatre for further debridement or exchange arthroplasty, ongoing antibiotics or death (Tan et al., 2018). Despite these not being equal outcomes, they may be categorised as such if only “success or failure” as a dichotomous variable or “time to failure” are considered the primary outcome measure. In another example, two patients may both be infection-free after exchange arthroplasty, but when one patient suffers disabling joint function post-operatively, then they would be considered to have an equivalent successful outcome when using a binary infection-free outcome measure (Rietbergen et al., 2016).

In practice, a spectrum of outcomes normally exists (Follmann et al., 2020). When a binary outcome is used, treatment success, including other factors important to the patient, can be overlooked (Evans et al., 2020) and does not capture the multifaceted decision-making clinicians undertake (Evans and Follmann, 2015). Each treatment approach can result in different cure rates (Kapadia et al., 2016), physical function (Klemt et al., 2021), surgical risks and benefits (Leta et al., 2019), antibiotic risks and benefits (Malahias et al., 2020), quality of life (Kuiper et al., 2018) and general well-being (Moore et al., 2015). Studies often evaluate treatments in terms of efficacy or safety separately, but this comes with the risk of finding a treatment superior for efficacy, yet it may have a worse safety profile, or vice versa (Molina and Cisneros, 2015). An ideal outcome would account for all of these. Desirability of outcome ranking (DOOR) is a composite endpoint and a more global approach to the patient's outcome (Evans and Follmann, 2015). To date, DOORs have been developed primarily to compare antibiotic treatments for diseases. A DOOR ensures such benefits (or efficacy) and risks (or safety) to patients are captured simultaneously rather than relying on a binary outcome (Evans et al., 2015). This is also different to the useful Musculoskeletal Infection Society (MSIS) reporting tool, which has ordinal surgical outcomes (Fillingham et al., 2019), and a DOOR could complement other such treatment effectiveness measures (Evans et al., 2015). Standardising the DOOR definition prior to implementation is important to avoid arbitrarily defined ranks (Celestin et al., 2017). The aim of this study was to develop a DOOR for periprosthetic joint infection (DOOR-PJI).

2 Materials and methods

A Delphi process was used to develop a DOOR-PJI (Sinha et al., 2011). We developed a questionnaire, including a series of ranking questions and discrete choice experiments, based on three patient scenarios (acute PJI, chronic PJI and a frail patient with PJI). The questionnaire incorporated a survey design strategy (Sprague et al., 2009) using published suggestions (Evans and Follmann, 2016). A literature review was performed using PubMed, MEDLINE and Google Scholar of English language studies to collate patient outcome measurements used for PJI (Cobo et al., 2011; Grammatopoulos et al., 2017; Helwig et al., 2014; Kuiper et al., 2018; Luu et al., 2013; Malahias et al., 2020; Moore et al., 2015; Mur et al., 2020; Poulsen et al., 2018; Rietbergen et al., 2016; Shah et al., 2020; Walter et al., 2021; Winkler et al., 2019), in addition to the already defined outcome of success (Diaz-Ledezma et al., 2013). For PJI, key treatment efficacy dimensions include joint function, overall physical and mental health (quality of life) and infection cure. Principal safety of treatment dimensions include antibiotic side effects, further surgery (repeat debridement or exchange arthroplasty), amputation and mortality. These efficacy and safety dimensions were all examined for each of the three scenarios, namely acute PJI, chronic PJI and frail patient with PJI (Table 1). An optional free text section was also available for outcomes, which were then evaluated and incorporated if applicable.

Table 1 Initial round questionnaire items for acute PJI, chronic PJI and frail patient with PJI.

^* Radical surgery examples included amputation, arthrodesis or excision arthroplasty. DAIR is the debridement antibiotics and implant retention.

All participants invited were clinicians who have extensive experience in treating PJI, have either published journal articles, chapters or textbooks on PJI, have special research interest in this area and were primarily from the previously established Global Arthroplasty Infection Association (GAIA) group (Atrey et al., 2021). Overall, 16 orthopaedic surgeons, 11 infectious diseases specialists and 1 clinical researcher across eight countries were invited to participate by email from July to August of 2021. The participants were informed of the study objectives, methods, principal investigators and estimated time required to complete the questionnaire.

The questionnaire was sent to participants and responses were entered via internet-based survey software (SurveyMonkey, San Jose, CA, USA). Participants were asked to answer questions by considering what is “most important for your patient”. For ranking questions, participants were asked to rank outcomes from 1 (most desirable) to 6 (least desirable) at a 12-month follow-up point (Xu et al., 2020a). For discrete choice experiments, participants were asked to choose which domain of the two choices presented was most important or desirable. Individual responses were anonymous and not visible to other respondents.

A Delphi method was used to establish the final DOOR. This occurred over two rounds. The first round was questionnaire based, as previously described. The first round began once the first questionnaire was sent out. Respondents had 2 weeks to complete the questionnaire. In the final round, respondents were asked to select between proposed DOORs which were determined based on the answers of the first round.

For the general ranking experiment, the three outcomes with the highest agreement were included in the second round DOOR option formation. For discrete choice experiments, the domains which were chosen by the majority of respondents in three of four (75 %) or four of four (100 %) choices were included in the second round. The second round began 2 months after the first round was completed. The same group invited to complete the first round were invited to complete the second round. Respondents again had 2 weeks to respond. Two different DOOR configurations were proposed based on the domains deemed most important in the first round. Participants were asked to select a DOOR considered to reflect the most important outcome rankings. Strong consensus of >75 % for the DOOR was required for the Delphi process to be completed. A comparison of responses between orthopaedic surgeons and infectious diseases was also collated. The institutional ethics department gave an ethics waiver for the study (Ethics Authorisation Number: AU20210621).

3 Results

A total of 28 participants were invited to complete the Delphi process (16 orthopaedic surgeons, 11 infectious diseases specialists and 1 clinical researcher). Of these, 19 out of 28 participants (68 %) completed the first round. One was excluded as the survey was incomplete, thus leaving 18 participants. Overall, in the second and final round, 18 of 28 (64 %) participants completed the questionnaire, consisting of 11 (61 %) orthopaedic surgeons and 7 (39 %) infectious diseases specialists from six countries. In total, 10 did not complete the questionnaire (5 orthopaedic surgeons, 4 infectious diseases specialists and 1 clinical researcher).

3.1 Desirability of outcome ranking

The two most important dimensions from the first round were joint function and overall health. The two proposed DOORs that respondents were asked to choose between either used overall health or joint function as the primary dimension. The desirability of outcome ranking was determined by strong group consensus (>75 % agreement), with 78 % of respondents selecting this DOOR. The final DOOR-PJI by consensus included the following four key dimensions: joint function, infection cure, mortality and overall health (Fig. 1).

Figure 1 Proposed consensus DOOR for PJI. Joint function was defined as good for knees, with the Oxford Knee Score ≥32.5, and for hips, with the Oxford Hip Score ≥37.5 (Hamilton et al., 2018). Infection cure was defined by the international consensus criteria (Diaz-Ledezma et al., 2013). The overall health score, using the 12-item short-form health survey (SF-12) with a cut-off for a good outcome of ≥36 on the physical scale and ≥53 on the mental scale (Helwig et al., 2014).

3.2 Discrete choice experiments

Both joint function and overall health were considered to be the most important dimension compared with antibiotic side effects, the need for further surgery or infection cure. This was true for all three scenarios. When compared head-to-head in the first round, overall health was considered more important than joint function in all three scenarios (Table 2). When comparing responses in the general discrete choice experiments for both specialties, the preferred outcome selected was the same in 24 out of 30 experiments (80 % agreement). The remaining discrete choice experiments in which the specialities' opinion disagreed are summarised in Table 3.

Table 2 Preferred results from discrete choice experiments in the first round of the Delphi process.

Abx is the antibiotics, Inf is the infection, Jt Fn is the joint function, OT is the operation, S/E is the side effects, and Overall is the overall health. ^* Results different to those selected in the acute scenario.

Table 3 General discrete choice experiments with speciality disagreement.

Abx is the antibiotics, Inf is the infection, Fn is the function, OT is the operation, and S/E is the side effects.

In the specific discrete choice experiments, the most important outcome selected was the same for both acute and chronic scenarios. For example, further operations with the possibility of curing the infection was preferred over infection suppression with antibiotics. A chronically infected joint with antibiotic suppression was preferred over a curative amputation in all scenarios.

In the free text section of the first round, there were no further dimensions or outcomes suggested. However, a recurring suggestion across the three scenarios was that the selected treatment depends on multiple factors, including patient's age, co-morbidities, wishes and expectations.

3.3 Ranking experiments

From the first-round general ranking experiment, joint function was consistently ranked as the most important domain across all three scenarios (ranked first by 83 %, 71 % and 78 % of responders for acute PJI, chronic PJI and frail patient PJI scenarios respectively). Infection cure and overall health also ranked second or third, depending on the scenario. Interestingly, severe antibiotic side effects (e.g. Clostridium difficile diarrhoea, acute kidney injury and central line-related deep venous thrombosis) and the need to return to surgery for further debridements or even revision surgery were considered to be the three least important domains in all scenarios (Fig. 2).

Figure 2 Domain ranking for general ranking experiments in the first round of the Delphi process. Percentages shown as a cumulative number of respondents who selected the domain as the most important or second most important from the ranking question.

The preferred ranking order from the specific ranking experiment which was chosen for acute, chronic and frail patient scenarios is shown in Table 4. The order selected by the majority of participants was the same in all three scenarios. The only exception was that, in the frail patient scenario, the fourth and fifth outcomes were ranked equally.

Table 4 Preferred ranking order from specific ranking experiments.

Note that this ranking was the preferred order for acute, chronic and frail patient scenarios. ^* Radical surgery is the amputation, arthrodesis or excision arthroplasty. PJI is the periprosthetic joint infection.

4 Discussion

In this Delphi analysis, joint function was considered to be the most important element of an outcome in PJI, followed by general health. Infection cure, traditionally used as the cardinal reporting outcome for PJI, was considered less important than joint function and general health in this study. It has been suggested that treatment efficacy measured with a single metric is insufficient, and quality of life outcomes following PJI treatment also need to be included (Molina and Cisneros, 2015). The Delphi technique's ability to elucidate new perspectives (Rodríguez-Mañas et al., 2013) has demonstrated this finding in our study, where patient-centric outcomes of joint function and overall health were considered more important than the need for repeat surgery or antibiotic-related outcomes. For example, a patient with poor joint function but who was determined to be free from infection and did not have further surgery can be identified by the DOOR-PJI but will be considered a treatment success when using a binary outcome analysis measuring infection cure or revision surgery alone.

Avoiding infection has been associated with the improved quality of life in PJI (Poulsen et al., 2018) and was incorporated into the DOOR-PJI. All-cause outcomes were chosen to measure patient-centric metrics rather than treatment-centric outcomes (Celestin et al., 2017). Additionally, in the current DOOR for PJI, for two patients with equal ranking, overall health can be used as a tie-breaker. Antibiotic duration is not appropriate as a tie-breaker, as this is usually protocol determined. The need for debridement or revision surgery could have been used as a tie-breaker; however, overall health was considered more important and ranked higher than treatment metrics such as repeated surgeries.

The MSIS reporting guide for PJI includes a list of surgical outcomes, including ongoing antibiotics without surgery, repeat debridement, antibiotics and implant retention (DAIR), revision arthroplasty, retained spacer, amputation, resection arthroplasty or arthrodesis (Fillingham et al., 2019). Our DOOR for PJI may be used alone or considered complementary to this by reporting patient-centric outcomes of joint function, infection cure and overall health. The DOOR can be used to analyse or compare treatment(s) by ranking methods, such as the DOOR probability itself and the win ratio, or by scoring methods, such as the partial credit scoring system (Evans et al., 2020).

The DOOR can be adapted clinically as some patients may simply prefer a functional joint with suppressive antibiotics and forgo further curative surgery, and they would be ranked higher than those with poor function. In contrast, other patients may prefer infection cure, even if it requires more radical surgery with potentially a worse joint function. In this case, if a patient prefers infection cure over joint function, then the partial credit strategy may be employed where priority in ranking is given to infection cure. A partial credit strategy assigns scores to outcomes to assist in comparing treatments. Care must be taken, however, as the partial credit strategy can produce different results.

The other important application is where studies find no significant difference in the originally measured outcome, and the DOOR may reveal a difference in the treatments (Montepiedra et al., 2016). For example, if two PJI treatments have similar infection cure rates but one produces a greater proportion with good joint function, then the DOOR-PJI will capture this difference in a meaningful way for patients.

A joint function score was determined to be the primary dimension in the DOOR. It is acknowledged that multiple joint function scores exist for each joint, for example, for knees and hips the Oxford Knee Score (OKS) and Oxford Hip Score (OHS) respectively are widely used (Murray et al., 2007). A cut-off score is required to separate “good” from “poor” joint function. The cut-offs for these are yet to be formally elucidated for PJI. However, for patients undergoing total knee arthroplasty (TKA) or total hip arthroplasty (THA) following osteoarthritis, the satisfactory Oxford score is ≥32.5 for the knee and ≥37.5 for the hip (Hamilton et al., 2018). These could be used as a surrogate until values are formally established in cohorts of patients with PJI. Furthermore, the DOOR-PJI deliberately allows for the application of a relevant joint function score, depending on the joint of interest.

Overall health can be used as a tie-breaker for patients. Patients with PJI often have worse general health than the general population (Helwig et al., 2014). When measuring overall health with the 12-item short-form health survey (SF-12), cut-off scores for patients successfully treated with PJI following TKA or THA have been reported at 36 on the physical scale and 53 on the mental scale (Helwig et al., 2014). It is acknowledged these patients may have confounders that affect their overall health scores; nevertheless, capturing a patient's overall health by raw score (or alternatively by amount changed) allow otherwise equal patients to be ranked if required in the DOOR-PJI. Overall health was deemed important but secondary to joint function in this DOOR. This likely reflects physicians' concerns with using a measure which may not be as specific when analysing PJI and could be heavily confounded by other non-PJI-related variables. Overall health measured with a quality-of-life score such as the SF-12 or EuroQol five-dimension (EQ-5D) score has also been correlated previously with Oxford hip and knee scores (Conner-Spady et al., 2018).

Generally, specialities agreed in most dimension comparisons; however, there were interesting differences noted in Table 4. When comparing dimensions, infectious disease physicians considered joint function, overall health and avoiding antibiotic side effects most important, while orthopaedic surgeons considered infection cure and avoiding a return to theatre more important. This may reflect differences in the ability of each specialty to influence the particular dimension or reflect how the specialities have seen these particular dimensions affect their patients and, thus, consider these dimensions to be most important. Future research might explore this in more detail and reveal if important dimensions also differ for patients with PJI.

The limitations of the study are acknowledged. First, the proposed DOOR only reflects the values of clinicians in the treatment of PJI when asked to consider what is most important to patients and not of patients themselves. An essential next step in our research programme will be to repeat this process with patients, as they may have differing views regarding what outcomes are most important (Moore et al., 2015). Second, in discrete choice experiments, avoiding further operations was considered more important than joint function in the frail patient scenario, so this DOOR may be less applicable in patients where frailty may impact the safety of undertaking an operation. Alternative DOORs may be developed if future studies found that different component outcomes were significant after evaluation (Evans et al., 2020). Third, our DOOR-PJI has only five categories, and a DOOR should not be overly simplified to avoid missing meaningful differences. Conversely, the power in DOOR studies is increased with fewer DOOR categories (Phillips et al., 2016), and care should be taken to avoid making a DOOR overly complex. Last, other variables that can have an important effect on the DOOR should be included in multivariate analyses to adjust for potential confounders.

This DOOR highlights the importance for clinicians to focus on the effects of the treatment and the end result rather than the treatment itself. A return to theatre, a more extensive surgery or ongoing antibiotics should not necessarily be considered a dichotomous measurement of failure if the patients' joint function is superior, their quality of life improves or they subsequently achieve infection cure (Barry et al., 2021).

We have proposed a novel DOOR for periprosthetic joint infection. This DOOR could be applied for acute PJI and chronic PJI. The key components are patient-centric metrics, including joint function, infection cure and overall quality of life. Future studies may use this DOOR to compare treatments by assessing patients more globally. The DOOR-PJI can also be incorporated into PJI studies and periprosthetic infection registries.