Use of the quality management system “JACIE” and outcome after hematopoietic stem cell transplantation

Abstract

Competent authorities, healthcare payers and hospitals devote increasing resources to quality management systems but scientific analyses searching for an impact of these systems on clinical outcome remain scarce. Earlier data indicated a stepwise improvement in outcome after allogeneic hematopoietic stem cell transplantation with each phase of the accreditation process for the quality management system “JACIE”. We therefore tested the hypothesis that working towards and achieving “JACIE” accreditation would accelerate improvement in outcome over calendar time. Overall mortality of the entire cohort of 107,904 patients who had a transplant (41,623 allogeneic, 39%; 66,281 autologous, 61%) between 1999 and 2006 decreased over the 14-year observation period by a factor of 0.63 per 10 years (hazard ratio: 0.63; 0.58–0.69). Considering “JACIE“-accredited centers as those with programs having achieved accreditation by November 2012, at the latest, this improvement was significantly faster in “JACIE”-accredited centers than in non-accredited centers (approximately 5.3% per year for 49,459 patients versus approximately 3.5% per year for 58,445 patients, respectively; hazard ratio: 0.83; 0.71–0.97). As a result, relapse-free survival (hazard ratio 0.85; 0.75–0.95) and overall survival (hazard ratio 0.86; 0.76–0.98) were significantly higher at 72 months for those patients transplanted in the 162 “JACIE“-accredited centers. No significant effects were observed after autologous transplants (hazard ratio 1.06; 0.99–1.13). Hence, working towards implementation of a quality management system triggers a dynamic process associated with a steeper reduction in mortality over the years and a significantly improved survival after allogeneic stem cell transplantation. Our data support the use of a quality management system for complex medical procedures.

Introduction

Implementation of a quality management system has become standard practice for industries when their products or services are associated with significant risks to human safety. Use of a quality management system contributes to better products and services. It raises consumers’ trust and confidence; it is associated with stronger customer loyalty, more repeat sales, less vulnerability to price pressures and lower marketing expenditures.21 As a consequence, “quality management” has become an essential component of today’s management strategies.73

Use of quality management systems and accreditation has been advocated as a putative driver for quality, safety and reduced costs in healthcare as well. First introduced about two decades ago in hospital pharmacies and laboratories,8 quality management has altered previously established mechanisms, induced structural changes and promoted high quality organizational processes. It has improved structures of health services’ organizations and altered professionals’ attitudes towards external and internal assessment.139 Competent authorities, healthcare payers and hospitals devote increasing resources to quality management systems and to accreditation or certification of parts or all of their activities. Still, evidence of improved patients’ outcome is scarce.1714 With increasing financial constraints, questions about the value of the large sums invested in health service accreditation arose in recent years.2017

Hematopoietic stem cell transplantation (HSCT) is an established treatment for many patients with severe congenital or acquired disorders of the hematopoietic system. Despite major improvements, it remains associated with substantial morbidity and mortality.2421 HSCT requires the cooperation of many categories of healthcare professionals. Hence, it presents a role model to assess the value of a quality management system which defines infrastructures, equipment, release of products or services, responsibilities, training of personnel, acceptable criteria for admission and discharge, and requires implementation of standard operating procedures and continuous improvement strategies as key elements.73

In this context, “JACIE” (www.jacie.org) and its US equivalent counterpart “FACT” (www.factwebsite.org) have developed an evolving set of identical standards (kindly provided initially by FACT and currently in its fifth version) that require an ongoing quality management system and apply to clinical, collection and processing activities. Centers seeking accreditation are subject to a detailed document review, on-site inspection and auditing procedure as is used in any industrial total quality management program.2625 A preliminary analysis, based on a small number of patients transplanted in an accredited center showed a significant stepwise reduction of early mortality by each phase of the accreditation process for patients transplanted in accredited centers.27 We, therefore, tested the hypothesis that improvement in outcome would begin long before final “JACIE” accreditation, extend during post-transplant follow-up and hence lead to a more rapid reduction of mortality over calendar years.

Methods

Study design

This retrospective observational analysis was based on a previously published cohort of the EBMT database; it begins at January 1 1999, 3 years before the first center in Europe was “JACIE”-accredited and ends at December 31 2006.27 Accreditation status was defined in two different ways: (i) depending on the particular phase of the accreditation process at the moment of the transplant (baseline: center had never applied for accreditation or 3 years before application; preparatory: the 3-year period before application; application: from application to accreditation; accreditation: after accreditation); and (ii) depending on the accreditation status of the respective transplant center in November 2012 (“JACIE”: 49,459 patients; 162 centers; “JACIE”: 58,445 patients; 423 centers). We postulated that the organizational and cultural changes associated with the introduction of a quality management system1493 should already be detectable during the years 1999 to 2006 in centers that achieved “JACIE” accreditation at any time thereafter. All EBMT teams are required to obtain patients’ consent and to have internal review board approval for evaluation of their transplant programs and for data transfer to the EBMT.

Patients

The cohort included 107,904 patients, 59% males, with a first allogeneic HSCT (n= 41,623; 39%) or autologous HSCT (n= 66,281; 61%) for an acquired hematologic disease (Table 1). There were significant changes in the population of patients from 1999 to 2006 with an increasing proportion of acute leukemia and a relative decrease in chronic leukemia as the main diagnosis and a steady increase in EBMT risk score.2927

Table 1.Characteristics of 107,904 patients, children and adults, who underwent an allogeneic (n= 41,623; 39%) or autologous (n= 66,281; 61%) HSCT between 1999 and 2007 in Europe transplanted in a center accredited by “JACIE” by November 2012 or not.

Statistical analysis

The focus of the statistical approach was on the interaction between “JACIE“ accreditation and calendar time on mortality reduction with adjustments for the key known risk factors. The main confounders, cluster or stratification variables were: main disease, EBMT risk score (score points 0–7: age of patient: <20 years = 0, 20–40 years = 1, > 40 = 2; disease stage: early = 0, intermediate = 1, advanced = 2; time interval from diagnosis to transplant: <1 year = 0, >1 year = 1; allogeneic HSCT only: donor type: HLA identical sibling = 0; other donor = 1; donor-recipient gender combination: all other = 0, female donor for male recipient = 12927), patient’s age, donor type, conditioning, calendar year, center, center size analyzed as successive quartiles, and Gross National Income per capita (GNI/cap) of the center’s country24 (data obtained from www.worldbank.org).

An extended Cox proportional hazards model was chosen. Cause-specific hazards were calculated, taking relapse and death as competing risks. Disease and conditioning were considered as stratification factors, since survival of patients with different diseases was not proportional (Figure 1) and conditioning was not a target of the analysis. The effect of accreditation was estimated by using a different baseline within each stratum and averaged over the categories. All covariates were truly patient-level covariates, except for “JACIE” accreditation, “GNI/cap” and “center size” which were shared among all patients transplanted at the same time and place. This “JACIE” variable as defined above served as a covariate to predict outcome at the patient level, not at the center level.

Figure 1.The figure depicts the diversity of the population of patients and the heterogeneity in outcome as illustrated by the Kaplan-Meier estimates of overall survival for 107,904 patients who underwent allogeneic (n=41,623) or autologous (n=66,281) HSCT in Europe between 1999 and 2006. (A) Allogeneic HSCT and (B) autologous HSCT showing overall survival by main disease category (acute leukemia, blue; chronic leukemia, orange; lymphoma, red; plasma cell disorders, purple; myelodysplastic disorders/myeloproliferative neoplasms, yellow; and bone marrow failure syndromes, allogeneic only, pink). No P values are given. The figure simply illustrates the heterogeneity; the study was not meant to assess differences between main disease categories. (C) Allogeneic HSCT and (D) autologous HSCT showing overall survival depending on EBMT risk score. Allogeneic HSCT: score 0+I (n=7,217; blue), score II+III (n=18,524; yellow), score IV+V (n=13,781; red), and score VI+VII (n=2,101; lilac). Autologous HSCT: score 0+I (n=3,889; blue), score II+III (n=36,290; yellow), and score IV+V (n=39,883; red). The hazard ratios for increasing risk with increasing score are depicted in Table 2. (E) Overall survival (OS) of 15,618 patients with an allogeneic HSCT in 1999 (4,742 patients; OS at 3 years 47.3%; blue), in 2002 (5,043 patients; OS at 3 years 50.5%; green) and in 2005 (5,833 patients; OS at 3 years 53.8%; yellow), illustrating the improvement over calendar time.

Endpoints in all analyses were overall survival, relapse-free survival, relapse incidence and non-relapse mortality.28 The interaction terms tested were “JACIE”*size, “JACIE*calendar year, “JACIE”*age, and “JACIE*GNI/cap. For reasons of comparability we evaluated the same models with and without the remaining significant interaction terms to get a separate as well as a pooled (adjusted) “JACIE” effect.

Results

There were differences between the groups. Centers with “JACIE” accreditation by 2012 were more likely to have performed a higher total number of transplants, to be in countries with a higher GNI/cap, to perform more allogeneic HSCT, to do so with a higher proportion of alternative donors, to have more patients with a higher EBMT risk score, and to have fewer missing values.

Main risk factors and outcome after hematopoietic stem cell transplantation

The outcome of the 107,904 patients treated between January 1 1999 and December 31 2006 by allogeneic (n=41,623) or autologous (nN=66,281) HSCT (Table 1) was influenced by type of transplant, main disease, and EBMT risk score (Figure 1).28 The probability of overall survival at 6 years was 47% for recipients of an allogeneic HSCT (non-relapse mortality 29%, relapse incidence 30%, relapse-free survival 40%) and 57% for patients who underwent autologous HSCT (non-relapse mortality 11%, relapse incidence 49%, relapse-free survival 40%) with wide variations depending on the main disease (Figure 1A,B).

Data showed a systematic decrease in overall and relapse-free survival related to a systematic increase in non-relapse mortality and relapse incidence with increasing EBMT risk score for both allogeneic (Figure 1C) and autologous (Figure 1D) HSCT [hazard ratio (HR) 1.21; 1.20 to 1.23 per two score points for overall survival in allogeneic; HR 1.17; 1.16 to 1.19 in autologous HSCT)]. Overall survival after allogeneic HSCT was significantly better for patients with HLA- identical siblings as donors compared to those with matched other donors (HR 1.15; 1.09 to 1.20) or mismatched donors (HR 1.47; 1.37 to 1.56). Outcome improved significantly over time, with an adjusted HR of 0.63 (0.59 to 0.67) for overall survival expressed as improvement per 10 calendar years (Figure 1E; Table 2).

Table 2A.Allogeneic HSCT. Probability of overall survival (OS), relapse-free survival (RFS), relapse incidence (RI), and non-relapse mortality (NRM) after HSCT depending on “JACIE” accreditation status in November 2012 of the respective transplant team, year of transplant and key pre-transplant risk factors. Numbers represent hazard ratios (HR), adjusted for all other risk factors by stratification (see Methods section for details).

Table 2B.Autologous HSCT.

Internal quality control: outcome of patients and “JACIE” accreditation phase of the transplant team at the time of the transplant

The quality control analysis validated the previous findings for patients with an allogeneic HSCT. Overall and relapse-free survival improved stepwise from baseline (n = 33,753; HR = 1) over the preparatory period (n = 4,890; HR 0.90; 085 to 0.96) and application period (n = 1,922; HR 0.87; 0.80 to 0.95) to the accreditation period (n = 1,058; HR 0.87; 0.77 to 0.98) for patients who were in that particular phase of the accreditation process at the moment of the transplant. Non-relapse mortality and relapse incidence also decreased as systematically. Clustered survival analysis applied a robust log rank score test, accounted for random center effects (likelihood ratio test = 13.09; P=0.0045) and confirmed the previously reported findings with an additional 5 more years follow-up information.27 In contrast, effects after autologous HSCT (baseline n=55,762; preparatory 6,799; application 2,487 and accreditation 1,233 phase) were no longer observed.

Reduction in mortality over time for patients transplanted between 1999 and 2006 depending on working towards and achieving “JACIEpos” accreditation status of the transplant team the latest by November 2012

The annual improvement over the 14-year observation period was significantly faster in accredited centers. The overall mortality rate was decreasing by a factor of 0.58 per 10 years (i.e. 5.3% per year) in “JACIE” centers compared to decreasing by a factor of 0.70 per 10 years (i.e. 3.5% per year) in “JACIE” centers. This difference in speed of improvement was statistically significantly in favor of the accredited centers as estimated by a multiplier HR (interaction test) for overall survival (HR 0.83; 0.71 to 0.97), relapse incidence (HR 0.71; 0.57 to 0.89) and relapse-free survival (HR 0.78; 0.67 to 0.91).

As a consequence, non-relapse mortality (HR 0.86; 0.73 to 1.03) and relapse incidence (HR 0.82; 0.70 to 0.97) were lower for the 22,008 patients who underwent allogeneic HSCT in “JACIE” centers, resulting in significantly higher adjusted overall survival (HR 0.86; 0.73 to 0.98) and relapse-free survival (HR 0.85; 0.75 to 0.95) (Table 2A). A cohort of patients who received an allogeneic HSCT between 2004 and 2006 illustrates the impact of accreditation on outcome (Figure 2A). Effects were detected in patients with a low or intermediate (not high) EBMT risk score, as illustrated by a subgroup of patients transplanted in a large center (Figure 2B).

Figure 2.“JACIE” accreditation status of the transplant team by November 2012 and outcome of patients transplanted between 1999 and 2006. (A) Kaplan-Meier estimates of overall survival of 17,655 patients with an allogeneic HSCT, transplanted in the years 2004–2006 in a center accredited (green line; n=8,983) or not (blue line; n=8,672) by 2012. The respective hazard ratios are presented in Table 2A. (B) Overall survival (OS) and non-relapse mortality (NRM) at 72 months by EBMT risk score for 17,243 patients transplanted with an allogeneic HSCT in a large center accredited by November 2012 (blue line) or not (red line). (C) Overall survival and non-relapse mortality at 72 months by EBMT risk score for 28,052 patients transplanted with an autologous HSCT in a large center accredited by November 2012 (blue line) or not (red line).

As in the analysis of “JACIE” effects depending on accreditation status at the time of transplantation, the data failed to show a significant effect of “JACIE” accreditation by 2012 on either reduction of mortality over time or on any of the four outcomes after autologous transplantation (n= 66,281; HR 1.03; 1.00 to 1.06; overall survival) (Figure 2C; Table 2B).

Center size and outcome

Center size (as calculated per calendar year of transplant and adjusted for type of conditioning) was significantly associated with all outcomes. Patients who underwent allogeneic HSCT in large centers had a lower non-relapse mortality (HR 0.95; 0.91 to 1.00) and relapse-free survival (HR 0.97; 0.93 to 1.0) resulting in a significantly better overall survival (HR 0.95; 0.92 to 0.98; per quartile) compared to patients transplanted in a smaller center, and adjusted for the accreditation effect and all other risk factors. Patients who underwent autologous HSCT showed analogous effects except for those on non-relapse mortality (HR 1; 0.97–1.03): per quartile increase in center size, the overall survival increased by a HR of 0.95 (0.94–0.97), the relapse-free survival by 0.95 (0.94–0.97) and the relapse incidence 0.94 (0.93–0.96).

Of note, the median follow-up of survivors was significantly longer among accredited centers (72 versus 61 months) and significantly longer for patients who had allogeneic HSCT compared to those who had an autologous HSCT. This difference may both denote the association between accreditation and improved follow-up, and lead to an underestimation of benefits associated with accreditation.

Discussion

These data provide a clear view of the potential impact of accreditation in medical practice in general and specifically in HSCT. Results became better in all centers over calendar time but they improved significantly faster and were more pronounced for patients transplanted in the context of accredited programs. As a consequence, non-relapse mortality and relapse incidence were lower, and relapse-free survival and overall survival were significantly better for patients having received their allogeneic HSCT in centers accredited for the quality management system “JACIE” by the year 2012. This difference in outcome was observed as early as at day 100 and continued up to 72 months after HSCT. The effects were substantial, systematic and clinically relevant with an overall improvement of 10–15%. The data suggest that accreditation as an indicator for quality driven work was the single most important contributing factor to the substantial improvement over time. More importantly, introduction of a quality management system can induce visible changes in a medical team, long before final accreditation. This fits with observations of quality management system work in industrial production.73 However, no such effects could be demonstrated for patients who underwent autologous HSCT.

The analysis revealed other new findings. Outcome was significantly and systematically influenced by center size, GNI/cap and calendar year, in addition to the known risk factors such as EBMT risk score, age and donor type. The improvement occurred stepwise, independently of the accreditation over calendar years. Outcome was better, the larger the centers, and the richer their respective country. The populations of patients changed over time and differed between centers with accredited and non-accredited programs as well as between small and large centers, making the analysis more complex. The only statistically significant interaction found, however, was between the accreditation process and the year of transplant for allogeneic HSCT. All other tested interactions, including center size, EBMT risk score, age and GNI/cap, were not statistically significant. Hence, our observations, based on clustered survival models of individual patients with identical characteristics, through stratification by disease and conditioning regimen, clustering by center, and modeling calendar year, risk score, age, donor relation and GNI/cap as covariates did respect the full diversity of the patient populations and the teams. The analysis showed that accreditation was associated with improved outcome for all patients undergoing allogeneic HSCT, pediatric or adult patients, in all diseases, in small and large centers and in countries with low or high GNI/cap.

Not all factors known to influence outcome after HSCT were included in the analysis, such as co-morbidity score, viral status or cytokine polymorphisms; no adjustment for modern HLA-typing of unrelated donors was made. Similarly, the accreditation process did not assess all potential factors influencing team performance.30 There was, however, no hint of any additional interaction that could explain the “JACIE” accreditation status as a simple surrogate marker for an unknown unrelated effect. It is therefore likely, with all the limitations of an observational study, that the findings are indeed sufficiently unbiased and robust; these results cannot be reduced to a simple center effect, learning curve, cumulative experience or case load.3531 It remains possible that accredited centers were more prone to quality work and that accreditation remains a surrogate marker of quality consciousness. Nevertheless, the data showed a close relationship between the individual steps of the accreditation process and the improvements; they showed a clear difference in speed of improvement over calendar time between accredited and non-accredited centers. These observations over a long time-span are evidence for a more causal than casual relationship.

The absence of a “JACIE effect” after autologous HSCT requires an explanation. The analysis was focused on survival, the strongest and most unambiguous endpoint; potential effects on quality of life, hospitalization time or costs were not evaluated. Follow-up was significantly shorter after autologous HSCT and in non-accredited centers (data not shown); more missing data might have obscured potential effects. Cell processing, a crucial step in autologous HSCT, was performed under a quality management system for autologous HSCT as systematically imposed by most competent authorities, independently of “JACIE” accreditation; this might have reduced potential effects. Transplant-related morbidity and mortality rates are significantly lower for autologous than for allogeneic HSCT, so that any change induced by the implementation of a quality management system may be more difficult to detect. Moreover, the time span of clinical care under direct supervision of the transplant team might have been too short to show a difference. Patient and risk assessment by the transplant team begins long before the transplant for patients with allogeneic HSCT and care by the allogeneic transplant team continues, in principle, lifelong.36 A quality management system includes multiple elements. It includes description of responsibilities, continuous quality improvement strategies and error management; it mandates standard operating procedures for selection of patients (and donors), transplant techniques, the stem cell product and follow-up; it stipulates that data collection and data analysis are integral parts of the therapy. Any quality management system is, therefore, more likely to manifest its benefit after the clinically more complex procedure of an allogeneic HSCT with its longer lasting link of patients to the transplant team.

Our observations support integration of a quality management system into complex medical therapeutic strategies, including solid organ transplantation.3837 The focus of regulatory aspects should no longer be on center size alone (minimal numbers of procedures being themselves requirements for “JACIE” accreditation) or on the sole use of center-specific outcome data. Last, the clear differences between autologous and allogeneic HSCT suggest the next steps to take. Quality management should probably no longer be restricted to just one phase of the treatment, the immediate transplant period, but cover the whole treatment program, from diagnosis to terminal care. The data also show the complexity of any analysis. A simple comparison of outcome of patients treated with accredited versus non-accredited programs no longer suffices; however, designing traditional randomized studies (“HSCT with versus without a quality management system”) appears unfeasible in this context. The professional organizations are challenged to provide the necessary framework and to stimulate outcome research as an academic necessity.39

In summary, these data document that the use of a clinical quality management system is associated with improved survival of patients undergoing one form of complex medical therapy: allogeneic HSCT. They support the concept of a quality management system as a driver for quality, hence better survival and suggest its broader application in other fields of clinical medicine.

Acknowledgments

The authors acknowledge the cooperation of the participating teams and their staff; the JACIE Accreditation Office in Barcelona, the JACIE Board and Executive Committee, the JACIE Medical Directors, the JACIE Accreditation Committee, all JACIE inspectors and EBMT registered transplant programs that worked hard to prepare and achieve JACIE accreditation; the EBMT presidents and the working party chairs; the EBMT statistical office in Leiden; the EBMT co-ordination offices in Barcelona, Paris and London, the Austrian Registry (ASCTR), the Czech BMT Registry, the French Registry (SFGM-TC), the German Registry (DRST), the Italian Registry (GITMO), the Dutch Registry (HOVON), the Spanish BMT Registry (GETH), the Swiss Registry (SBST), the Turkish BMT Registry and the British Registry (BSBMT).

Footnotes

• Funding This study was funded by the European Group for Blood and Marrow Transplantation (EBMT) and the European Leukemia Net (ELN). EBMT is supported by grants from the corporate members: Amgen Europe, ViroPharma Europe, Celegene International SARL, Genzyme Europe B.V., Gilead Sciences Europe Ltd., Miltenyl Biotec GmbH, Schering-Plough International Inc., Bristol Myers Squibb, CaridiaBCT Europe NV, Cephalon Europe, F. Hoffmann-La Roche Ltd, Fresenius Biotech GmbH, Therakos Inc., Alexion Europe, Chugai Sanofi – Aventis, Merck Sharp and Dohme, Novartis, Pfizer, Pierre Fabre Médicament. AG was in part supported by a grant from the Bangerter Foundation on outcome research.
• Authorship and Disclosures Information on authorship, contributions, and financial & other disclosures was provided by the authors and is available with the online version of this article at www.haematologica.org.

• Received August 8, 2013.
• Accepted January 27, 2014.