Comparing the clinical trial efficacy versus real-world effectiveness of treatments for multiple myeloma: a population-based study

Phase III randomized control trials (RCT) are the “gold standard” used to obtain marketing and regulatory approval for novel multiple myeloma (MM) treatments, inform patients about treatment outcomes, and inform treatment guidelines. Yet, numerous indirect real-world (RW) and RCT comparisons have shown that RW patient tend to have inferior outcomes compared to RCT patients. However, to date, no study has directly quantified the differences in outcomes between RW and RCT patients with MM treated with standard of care (SoC) therapies. Understanding and quantifying the difference in efficacy, the outcome in an “ideal” RCT setting, and effectiveness, the outcome in the “real life” clinical practice setting, is needed to contextualize the generalizability of RCT data to the general population. To fill this knowledge gap, we conducted a population-based cohort study to compare and quantify the difference in the RW effectiveness versus RCT efficacy of SoC MM regimens with respect to the progression-free survival (PFS) and overall survival (OS).

The RCT cohort was identified from registrational phase III RCT which led to the public reimbursement of SoC regimens in Ontario between January 1, 2013 to December 31, 2021. Regimens included lenalidomide/dexamethasone (Rd) bortezomib/Rd (VRd) in patients with transplant-ineligible newly diagnosed MM patients (TIE-NDMM). Relapsed refractory MM (RRMM) regimens included carfilzomib/Rd (KRd), carfilzomib/dexamethasone (Kd), daratumumab/Rd (DRd), daratumumab/bortezomib/dexamethasone (DVd), and pomalidomide/dexamethasone (Pd). The most recent published Kaplan-Meier PFS^{1 -7} and OS^1,2,7-11 curves were manually digitized using the WebPlotDigitizer software (version 4.6), then reconstructed using an established algorithm¹² to provide individual patient-level estimates of PFS and OS for the experimental arm in the RCT cohorts.

RW data was obtained using from Ontario’s ICES administrative database. Ontario has a universal, publicly funded healthcare system which provides access to chemotherapy, and the provincial administrative database captures virtually all health care encounters and has a loss to follow-up rate of 0.25%. Treatment data was accessed through the Ontario Drug Benefit database for regimens containing only oral medications and the Cancer Activity Level Reporting database for treatment regimens containing injected or infused medications. Patients diagnosed with MM between January 1, 2013 to December 31, 2020 and initiating treatment with SoC regimens either at diagnosis or relapse were included in this study. Provincial reimbursement criteria for SoC regimens mirrors the RCT inclusion criteria, with regards to the prior drug exposure and lines of therapy (see Online Supplementary Table S1). The data cut-off date was May 31, 2022.

We assessed the efficacy and effectiveness of RCT and RW data, respectively, by comparing the Kaplan-Meier survival curve estimates of PFS and OS. The RCT PFS was defined as the time from index regimen treatment to disease progression, death, or last follow-up, whichever occurred first. In RW administrative database, the progression date could not be accurately determined and so time to next treatment (TTNT) was used as a surrogate for RW PFS. TTNT was defined as the time from initiation of index regimen to initiation of subsequent MM treatment, death, or last follow-up. In both RCT PFS and RW TTNT definitions, patients remaining on the index regimen at last follow-up were censored. OS was defined as the time from initiation of the index treatment to death or end of follow-up. Meta-analyses using random effects models were used to compare the PFS and OS outcomes of RW versus RCT patients. Effect estimates for PFS and OS were summarized using hazard ratios (HR). The study was approved by the ethics committee of McMaster University.

Overall, 3,951 RW and 2,476 RCT MM patients, treated with seven SoC MM regimens, were included. Baseline characteristics of the RW and RCT cohorts are shown in Table 1. Overall, the RW cohort patients tended to be older than RCT patients. A minority of TIE-NDMM patients treated were initially started on a short course of single agent lenalidomide or bortezomib during the COVID-19 pandemic and then transitioned to the full triplet regimen. For RRMM SoC regimens, the time between MM diagnosis and index regimen treatment initiation was longer in the RCT versus RW cohorts. However, despite RW patients being treated with the SoC regimens earlier in their disease course, apart from RW patients treated with Pd, RW patients tended to have higher rates of previous lenalidomide and bortezomib exposure.

The RW had 51% increased risk of progression or death compared to RCT patients (pooled HR=1.51, 95% confidence interval [CI]: 1.03-2.21; P=0.034; Figure 1A). Six of the seven SoC regimens analyzed had a shorter mPFS in the RW cohort compared to the RCT cohort (Table 2; Online Supplementary Figure S1). The disparate PFS outcomes were more apparent in patients treated with RRMM regimens (pooled HR=1.66, 95% CI: 0.99-2.80; P=0.056, absolute decrease in mPFS ranged from 7.2-18.3 months in the RW cohort) as opposed to NDMM regimens (pooled HR=1.16, 95% CI: 1.03-1.31; P=0.012, absolute decrease in mPFS ranged from 3-8.2 months in the RW cohort). Even after excluding previously lenalidomide/bortezomib-exposed patients in the TIE-ND-MM VRd RW cohort, the RW cohort had a trend towards poorer PFS and poorer OS outcomes compared to the RCT cohort (PFS HR=1.12, 95% CI: 0.81-1.55; OS HR=2.037, 95% CI: 1.38-3.02). Similarly, RW patients had a worse OS compared to RCT patients treated with six of the seven regimens, with a 76% higher risk of death (pooled HR=1.76, 95% CI: 1.31-2.36; P<0.001; Figure 1A) and an absolute median OS ranging from 19.3-37.9 months lower than RCT patients. We then stratified outcomes age and baseline comorbidity index (see Online Supplementary Table S2). Older adults tended to have slightly longer mPFS but similar mOS (which may reflect slower transitions to next line therapy at time of relapse), while patients with more baseline comorbidities had shorter mPFS and mOS estimates. Overall, the mPFS and mOS were consistently lower in the RW versus RCT cohorts.

Table 1.Comparison of baseline characteristics and outcomes of patients treated with multiple myeloma regimens as part of standard of care in the real world versus and in the corresponding randomized clinical trial.

Figure 1.Meta-analysis of the progression-free survival and overall survival in the real-world and randomized clinical trial cohorts. (A) summarizes the progression-free survival (PFS) and (B) overall survival (OS) in real-world (RW) versus randomized clinical (RCT) patient cohorts with multiple myeloma (MM), stratified by regimens used in the newly diagnosed MM (NDMM) versus relapsed refractory MM (RRMM) treatment setting. Rd: lenalidomide and dexamethasone; VRd: bortezomib, lenalidomide and dexamethasone; Kd: carfilzomib and dexamethasone; KRd: carfilzomib and Rd; DVd: daratumumab, bortezomib and dexamethasone; DRd: daratumumab and Rd; Pd: pomalidomide and dexamethasone; CI: confidence interval.

Table 2.Summary of median progression-free survival and median overall survival in the real-world and randomized clinical trial cohorts.

This is the first study directly quantifying the significant difference in RCT efficacy and RW effectiveness of SoC MM treatments. Our study’s strengths include our data source - a large database comprehensively capturing treatment in a universal healthcare system with minimal loss to follow-up, with patients treated in both academic and community centers - thereby providing an accurate RW assessment of health outcomes.

The main contributors to the efficacy-effectiveness gap are likely differences in patient selection and the regimen administration or adherence between RW and RCT cohorts. It is well known that the stringent RCT inclusion criteria and mandatory drug washout periods often excludes patients with highly aggressive or proliferative disease. RW patients in this study had a shorter time from diagnosis to initiation of the index regimen and higher rates of prior drug exposure, suggesting they may have been more heavily pretreated compared to RCT patients which may explain why the efficacy-effectiveness gap was most apparent for RRMM regimens. RW patients in our study also tended to be older than RCT patients, and had a high comorbidity burden, and would likely not have met the stringent RCT inclusion criteria. Prior studies have similarly shown that up to 70% of RW patients would have been excluded from landmark RCT’s based on their baseline age, comorbidities, cytopenias, or organ function.^13-15 Lastly, RCT have strict protocols that require close patient monitoring and prespecified dose reductions based on reported adverse events. However, RW patients may have lower adherence or may have received lower doses of the SoC regimens which could have compromised outcomes.

Given limitations in our data availability within our administrative database, we could not assess how patient-level disease data (i.e., cytogenetic risk, baseline staging, prior treatment exposure and refractoriness) may have contributed to the efficacy-effectiveness gap. Another significant limitation was our use of TTNT as a surrogate for PFS in the RW cohort, as is often done in real-world observational studies. However, previous studies have shown that TTNT and PFS are comparable endpoints, and that TTNT may overestimate RW PFS due to delays in starting next-line therapy until significant biochemical or clinical progression. This is likely especially true in our study given the limited number of reimbursed treatment lines in our public-payer healthcare system. However, if treatment was switched due to intolerance or prior to meeting progression criteria, then TTNT may underestimate PFS. Lastly, our study reflects outcomes within the Canadian healthcare system where access to therapies is limited by public reimbursement. While our drug access is comparable to many other public healthcare systems in the developed world, our RW outcomes may not be as generalizable to patient populations with increased treatment accessibility.

In conclusion, this is one the largest population-level studies highlighting the significant efficacy-effectiveness gap between registrational RCT and RW usage of these regimens, with RW patients experiencing a 51% higher risk of progression or death, and a 76% higher risk of death compared to RCT patients. Future studies focusing on closing the efficacy-effectiveness gap may involve designing trials that better represent RW scenarios using pragmatic trial designs, or more inclusive eligibility criteria. Our data emphasize the importance of ongoing evaluation of RW data to further contextualize effectiveness of therapy and facilitate shared treatment decisions among patients and clinicians.

Footnotes

• Received May 1, 2024
• Accepted August 16, 2024

Correspondence

A. VISRAM - alisvisram@toh.ca

Disclosures

No conflicts of interest to disclose.

Funding

Acknowledgments

We thank IQVIA Solutions Canada Inc. for use of their Drug Information File. Parts of this material are based on data and/or information compiled and provided by CIHI, the Ontario Ministry of Health, and Ontario Health (OH). The analyses, conclusions, opinions and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred.