Prospective international validation of the Quality of Life in Myelodysplasia Scale (QUALMS)

Abstract

Disease-specific measures of quality of life can improve assessment of disease-related symptoms and psychosocial sequelae. We report on the development and validation of the Quality of Life in Myelodysplasia Scale (QUALMS), a 38-item assessment tool for patients with myelodysplastic syndromes (MDS). In 2014–2015, a multi-center cohort of patients with myelodysplasia completed the QUALMS, as well as the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire (QLQ-C30) and the Functional Assessment of Cancer Therapy Anemia Scale (FACT-An); a second administration was undertaken three to six months later. A total of 255 patients from the United States, Canada and Italy participated. Median age was 72 years, 56.1% were men, and the International Prognostic Scoring System distribution was 40.4% low, 42.0% intermediate-1, 13.3% intermediate-2 and 2.3% high. QUALMS scores ranged from 24 to 99 (higher scores are better), with a mean of 67.2 [standard deviation (SD)=15.2]. The measure was internally consistent (α=0.92), and moderately correlated with the multi-item QLQ-C30 scales and the FACT-An (r=−0.65 to 0.79; all P<0.001). Patients with hemoglobin of 8 g/dL or under scored lower than those with hemoglobin over 10 g/dL (61.8 vs. 71.1; P<0.001), and transfusion-dependent patients scored lower than transfusion-independent patients (62.4 vs. 69.7; P<0.01). Principal components analysis revealed “physical burden”, “benefit-finding”, and “emotional burden” subscales. There was good overall test-retest reliability among those with stable hemoglobin (r=0.81), and significant changes for patients hospitalized or with infections between administrations (both P<0.01). These data suggest the QUALMS is a valuable tool for assessing MDS-specific quality of life in the modern treatment era.

Introduction

The myelodysplastic syndromes (MDS) are a heterogeneous group of acquired hematopoietic stem cell disorders characterized by bone marrow failure and a tendency to transform to acute myeloid leukemia (AML). While supportive care with transfusions, hematopoietic growth factors and antimicrobial agents had long been the standard treatment approach,1 three disease-modifying therapies are now approved by the US Food and Drug Administration (FDA) for use in MDS, and many other agents are under investigation.2 Unfortunately, although hematopoietic cell transplantation (HCT) is potentially curative, it is not available to many patients due to their advanced age, comorbidities, or lack of an appropriate donor.3 The syndromes also have a variable course, with many patients living relatively symptom-free for many years, and only gradually developing problems such as fatigue, infections, and bleeding. This chronic nature of the syndromes conspires with the lack of curative options to make patients’ quality of life (QOL) a major focus of treatment decisions.4

Given these considerations, and the importance of QOL to patients with MDS,5 its rigorous measurement has been recognized as an MDS research imperative.86 Indeed, patients with MDS suffer from a wide variety of symptoms including fatigue, anxiety, insomnia, and dyspnea.9 It is essential to understand such patient-reported outcomes for their own value, and not just for their potential contribution to disease risk. Although an Internet-based survey of MDS patients’ QOL has been published,10 no MDS-specific QOL measure has been widely adopted for clinical or research use. Researchers aiming to assess the impact of MDS and its treatments18119 on QOL have most often used generic measures such as the Short-Form Health Survey (SF-36),19 or cancer-specific scales such as the European Organization for Research and Treatment of Cancer’s QLQ-C3020 and the Functional Assessment of Cancer Therapy-Anemia (FACT-An).21 While these measures are useful, they are not specific enough to contain all of the elements important to MDS-related QOL, and may contain several less relevant items.

In contrast to generic QOL questionnaires, disease-specific measures have the potential to more accurately reflect the full breadth of functional limitations and symptoms experienced by specific cancer populations.22 For example, recognition of the need for a disease-specific measure for another bone marrow stem cell disorder, myelofibrosis, led to the creation of the Myeloproliferative Neoplasm Symptom Assessment Form Total Symptom Score (MPN-SAF TSS),23 which has been critical to clinical research efforts for that disease and to regulatory approval of its first specific therapy, ruxolitinib.2524 In contrast, after many studies with generic instruments, the impact of many routine treatments on the QOL of MDS patients, such as erythropoiesis-stimulating agents26 and red cell transfusions,15 has yet to be clearly demonstrated.

We hypothesized that an MDS-specific measure of QOL would allow for a more relevant and complete assessment of the impact of interventions in both clinical and research settings. With that goal in mind, we sought the structured input of patients, caregivers, and health care providers to create and pilot a new MDS-specific measure of QOL: the Quality of Life in Myelodysplasia Scale (QUALMS). This instrument has now undergone prospective validation in an international cohort of patients with MDS, and we report on its validity, reliability and responsiveness.

Methods

Instrument development

In 2011, using the guidelines outlined by Guyatt,27 and working with the FDA recommendations for developing patient-reported outcome measures,28 we developed the QUALMS29 with structured input from MDS patients, caregivers, and clinicians (n=32) at Dana-Farber Cancer Institute (DFCI) in Boston, MA, USA. We piloted the instrument with additional DFCI MDS patients (n=20), making several changes, and producing a 38-question measure containing 33 core questions and 5 opt-out questions that took an average of 7.5 minutes to complete. Further development methods are detailed in the Online Supplementary Appendix.

Instrument validation: overview

In 2014–2015, we assessed the psychometric properties of the QUALMS in a cohort of patients with biopsy-proven MDS who were not involved in its development. Subjects came from five centers in the United States, Canada, and Italy (Table 1). All filled out the QUALMS, the QLQ-C30, and the FACT-An twice (at baseline and 3–6 months later) accompanied by reviews of their medical record. Additional study details are included in the Online Supplementary Appendix. We used the resulting data to assess the QUALMS’ internal consistency reliability, explore its underlying structure to identify potential subscales, and assess its concurrent validity, known groups validity, test-retest reliability (stability) and responsiveness. Each institution obtained study approval from its respective Institutional Review Board before enrolling patients, and all enrolled patients signed informed consent.

Table 1.Description of the QUALMS validation cohort.

Instrument validation: psychometric analysis

Descriptive statistics were performed using baseline data, followed by exploratory principal components analysis (PCA) with oblique rotation to identify the factor structure underlying the baseline QUALMS data and create subscales. We made an a priori decision to retain all factors that accounted for at least 5% of variance, if confirmed by the scree plot. We would then retain all questions with moderately high loading on each factor (r≥0.50) for subscales, and for the overall scale, retain all with modestly high loadings (r≥0.30) on at least one factor.30 We created subscale scores based on each of the factors. In the rare case that an item loaded moderately highly on more than one factor, we re-examined the inter-item correlations of the subscales, calculated internal consistency reliabilities, and discussed the theoretical implications of including the item in each of the factors before making a final decision about the subscale where the item would reside. Finally, we assessed if the subscales’ internal consistency improved with items removed; if it did, we planned to remove items accordingly.

We used correlation analyses to assess the concurrent validity of the QUALMS with other QOL measures that are theoretically related. Specifically, we correlated mean QUALMS scores with scores on the QLQ-C30 and on FACT-An, and used Fisher’s r to z test to examine differences among correlations. We identified groups known to differ on clinical markers (e.g. hemoglobin) and compared scores on the QUALMS to assess known-groups validity. This was completed using t-tests or analyses of variance (ANOVA) f tests. Next, utilizing baseline and follow-up data, we assessed the stability of the QUALMS by correlating the two scores. To assess responsiveness, we compared mean difference in QUALMS scores for patients with significant clinical events since baseline (bleeding, infection or hospitalization) to mean difference in scores for those without. Finally, we conducted exploratory validity analyses for the QUALMS subscales.

Results

Subjects

Two-hundred and fifty-five MDS patients (56% male) participated, from across five centers (Columbia 12%; Dana-Farber 20%; GIMEMA 36%; Moffitt 12%; Odette 20%). Patients were primarily white (95%), non-Hispanic (95%), and ranged in age from 28 to 92 years (mean=2, SD=0.8). The mean time elapsed between MDS diagnosis and enrolment was 3.6 years. Ninety-two percent of subjects were either fully active, or ambulatory but restricted in strenuous physical activity [Eastern Cooperative Oncology Group Performance Status (ECOG) scores31 of 0 or 1]. Twenty patients had psychiatric comorbidities (defined as depression or anxiety requiring psychiatric counseling or treatment),32 and 29 had a history of a solid malignancy at some point. Twenty-four patients had secondary MDS. Additional baseline characteristics are included in Table 1. Of note, 208 subjects (81.5%) completed a second QUALMS administration after a median interval of 4.3 months.

Descriptive analyses

Examination of individual QUALMS items indicated that none had floor or ceiling effects. A missing values analysis showed no identifiable pattern in missing values and indicated that across all 33 core items, fewer than 5% of responses were missing, and for 29 of 33 (88%), there were 2% or fewer with missing data. An analysis of the 5 potential opt-out questions revealed that the range of missing data or opt out was higher, 27% (“too tired to drive”) to 75% (“afraid of losing your job”). We thus retained only the 33 core QUALMS items for analyses after this step.

To score the QUALMS, answers for each question (all have 5-point Likert-type answers) were assigned a value with a potential range of 0 (worst) to 100 (best) as follows: Never=100; Rarely=75; Sometimes=50; Often=25 and Always=0. Four items were scored in the opposite direction such that Always=100 and Never=0. The QUALMS total score was calculated by averaging the scores on items 1–33, so the potential range of scores was 0 (worst) to 100 (best). Higher scores mean better QOL.

Internal consistency reliability analysis of the QUALMS using the 33 items revealed a Cronbach’s alpha of 0.92. Moreover, we found no further improvement to internal consistency with any items removed, so we retained all items. Overall QUALMS scores ranged from 24 to 99, with a mean score of 67.2 (SD=15.2). No significant differences were found in mean QUALMS scores of patients from the different MDS centers (P=0.09): Columbia 64.5 (±15.7); DFCI 66.5 (±14.5); GIMEMA 66.5 (±16.3); Moffitt 67.0 (±17.0); Odette 72.5 (±11.2).

Exploratory principal components analysis

Analysis of sampling adequacy indicated that the QUALMS questions were appropriate for factor analysis (Kaiser-Meyer-Olkin measure of sampling adequacy=0.88; Bartlett’s test of sphericity X(528)=2889; P<0.01). On the basis of the pre-specified criteria (see Figure 1 for scree plot) the exploratory principal components analysis was constrained to a 3-factor solution that explained 43% of the variance, including “Physical Burden” (30% variance; QUALMS-P), “Benefit Finding” (“silver linings” associated with disease; 7% variance; QUALMS-BF) and “Emotional Burden” (5.5% variance; QUALMS-E). All 33 core items had acceptable factor loadings (r≥0.30)30 on at least one factor and were thus retained for inclusion in the calculation of the overall scale score (Table 2). We also created subscale scores based on each of the factors.

Figure 1.Scree plot of exploratory principal components analysis: eigenvalues as a function of components. A scree plot is used in principal components analysis (PCA) to visually determine which underlying components explain most of the variability in the data. Generally, components are retained on the steep part of the curve, and once the line starts to straighten (after the “elbow”), it is felt that subsequent components do not explain much of the variability. For the QUALMS data, that point was seen after the 3^rd component (arrow).

Table 2.3-factor principal components analysis rotated structure matrix loadings and component correlation matrix used to derive the QUALMS subscales.*

Cronbach’s alphas for the final subscales were as follows: QUALMS-P a = 0.91; QUALMS-BF a = 0.62; QUALMS-E a = 0.84. The subscales’ internal consistency did not improve when items were removed, thus all items were retained. Correlation analyses revealed that the overall mean QUALMS had strong positive correlations with both the QUALMS-P (r=0.92, P<0.001) and the QUALMS-E (r=0.87, P<0.001), and a small but consistent positive correlation with the QUALMS-BF (r=0.17, P<0.05). The QUALMS-P and QUALMS-E were moderately correlated with each other (r=0.67, P<0.001), but not with the QUALMS-BF [r’s = 0.06 and 0.03, respectively, not significant (ns)].

Concurrent validity

The overall QUALMS score was moderately correlated with the global QLQ-C30 and its eight additional multi-item subscales (r’s=-0.65 to 0.68; P<0.01 for all) (Table 3), and had slightly stronger correlations with the FACT scores (e.g. r’s=0.74 to 0.79; P<0.01 for all).

Table 3.Correlations between overall QUALMS scores, EORTC QLQ30 and FACT-An.

Known groups validity

Patients who were transfusion-dependent had significantly lower overall QUALMS scores (worse QOL) than those who were not transfusion-dependent (Table 4). A similar pattern was seen comparing those who had ever had a transfusion with those who had not, and those who ever had treatment with those who had not. While variability in performance status was low, ECOG scores 2 or greater were highly associated with worse mean QUALMS scores compared to ECOG scores of 0 or 1 (52.2 vs. 68.0; P<0.001). Patients with Hb values greater than 10.0 g/dL had significantly higher scores compared to those who had values between 8.1 g/dL and 10.0 g/dL or 8.0 g/dL or lower. Higher scores were also found in patients who had lower marrow blasts and lower IPSS-R scores, but not for those with platelets greater than 50×10/L or ANC of 1×10/L or greater [an effect was seen at lower platelet (20×10/L) and ANC (0.5×10/L) thresholds; P<0.01 for both)]. Analyses using FACT-An scores as a dependent variable obtained mixed results. Although FACT-An total scores differed between patients with regard to transfusion use, dependence and hemoglobin (Hb) levels, they were not significantly discernible as a function of patients’ blast percentage, whether they ever had treatment, or their IPSS-R categories.

Table 4.Known groups analysis for the overall QUALMS and the QUALMS-P.*

Test-retest reliability

The overall QUALMS scores were relatively stable: r=0.76, P<0.001. Stability was strongest among those patients who showed no change in Hb from baseline (within ± 1pt) (r=0.81, P<0.001). Scores among patients whose Hb declined by 1 g/dL or more between baseline and follow up were less stable (r=0.60, P<0.05).

Responsiveness

t-test analyses supported responsiveness in overall QUALMS score for patients who experienced infection or hospitalization between baseline and follow up (P<0.01 for both), but not for those who experienced bleeding (Table 5).

Table 5.Responsiveness analysis for overall QUALMS scores.*

Preliminary validation of subscales

Concurrent validation analyses showed that QUALMS-P had moderately strong correlations with the QLQ-C30’s “Global Health” (r=0.67, P<0.001), “Physical Function” (r=−0.70, P<0.001) and “Fatigue” (r=−0.75, P<0.001) scores, along with the FACT-An (r=0.85, P<0.001). The QUALMS-E had moderate correlations with the QLQ-C30’s “Emotional Function” subscale (r=−0.58, P<0.001) and the FACT-An (r=0.57, P<0.001). The QUALMS-BF showed small, but consistent correlations with the EORTC QLQ-C30’s “Global Health” score (r=0.19, P<0.01) and the FACT-An (r=0.18, P<0.05). Finally, QUALMS-P scores significantly varied among patients who had different levels of transfusion exposure, dependence, treatment history, Hb, blast cells, IPSS scores, and baseline comorbidities (Table 4).

Discussion

We have developed a comprehensive instrument for capturing the critical QOL issues faced by patients with MDS. As a patient-reported outcome measure, the QUALMS demonstrates robust internal consistency, strong concurrent validity, and excellent differentiation between many known groups. It is reliable, and shows good responsiveness for patients who have undergone major clinical events between administrations. It is practical in that it takes less than ten minutes to complete, and contains clinically useful subscales.

We were surprised to find a factor emerge in our PCA that contained questions relating to so-called “benefit finding” (the QUALMS-BF). Benefit finding has been studied in other cancers,3633 where high levels may be associated with coping better and decreased levels of psychosocial stress; however, studies are not conclusive.37 Interestingly, unlike the overall QUALMS, or the QUALMS-P, the QUALMS-BF did not show strong correlations with the other QOL scales studied, arguing that this is a new dimension of MDS-related QOL that has not been previously captured. On the other hand, the QUALMS-BF was also less well-correlated with the overall QUALMS scores, which perhaps reflects a difficulty in assessing this domain given that there are only 3 contributing questions. Moreover, we do not know how benefit finding may change over the course of disease, whether it may peak soon after diagnosis, or if it may increase temporarily with each new treatment or disease-specific event.

While we acknowledge that more work is needed to determine precise thresholds for the overall QUALMS and its subscales, our data suggest that a clinically meaningful difference on the overall QUALMS may be between 5 and 10 points. We were able to distinguish between patients at baseline with differences in this range, and for those patients who had changes on their QUALMS scores at follow up, differences were also in this range. Moreover, a distribution-based method would argue that a meaningful difference would be a half standard deviation,38 which in the case of the QUALMS would be 7.6.

In the setting of a clinical trial, it is often desirable to minimize participant response burden when measuring QOL, especially if measured at many time points. We found that the QUALMS-P, a 14-item subscale focused on physical factors, had excellent internal consistency and performed well, distinguishing between many clinical known groups. In contrast, the 5 final QUALMS questions with opt-out options have not been studied beyond the development phase. Four of these (questions about sexual function, ability to work and drive, and take care of others) are arguably more relevant to younger patients with MDS (patients aged <65 years made up only 18% of our sample). Since little is known about the MDS-related QOL of this group, we suggest that these questions be retained and scored individually; they are clearly an area for further study.

We recognize several other areas that merit further investigation. First, while our study included five centers in three countries, we did not undertake a formal cross-cultural validation, and the instrument has only been translated into one other language besides English (Italian). Clearly more work is needed to characterize potential cultural and race-ethnic differences, and further international validation is ongoing. Second, while we found evidence of responsiveness, a 4-month observational interval is too short to fully assess changes in MDS-related QOL,39 and pre- and post-intervention or longer-term observational studies using the QUALMS will need to be performed. Third, our data were captured with in-person and paper administrations of the QUALMS. Although patients with MDS are in general elderly and may be less comfortable with contemporary technology, electronic and online versions of the QUALMS should be evaluated in the future.

Finally, as only 15.6% of patients in our cohort were in higher-risk IPSS categories, we note that further validation work will be necessary to investigate performance of our questionnaire in this higher risk population. This skew toward lower risk participants likely occurred because Intermediate-2 (INT-2) and High (HI) risk patients do not live as long, and are thus less likely to be captured in a study of our duration (enrolling for one year, with 6-month follow-up window). Indeed, in the cohort used to create the IPSS-R (n=7008, enrolling over many years, with median follow-up of approx. 4 years),40 there were only 23% INT-2 and HI risk patients, which supports the idea that most people living with MDS are in the risk categories that were more heavily represented in our analysis. Still, while we are confident that our study demonstrates that there will be differences in QUALMS scores as patients with MDS move between lower and higher risk disease states (and potentially back again), we acknowledge that more research is needed to characterize how the measure differs among those with higher-risk disease.

In conclusion, testing in a relatively large international cohort of MDS patients appears to show the QUALMS to be a valid measure of MDS-specific QOL. The measure and its subscales have the potential to be used to assess QOL throughout the MDS disease course. We envision the QUALMS as a valuable tool for use in clinical and research settings when evaluating MDS symptoms, making treatment decisions, and informing efficacy and effectiveness results from clinical trials and health services research.

Acknowledgments

The authors would like to thank Dr. Giovanni Caocci and Dr. Pasquale Niscola, respectively from the Hematology Unit, Department of Internal Medicine, University of Cagliari (Italy) and Department of Hematology, “S. Eugenio Hospital”, Rome, (Italy), and Dr. Michael Constantine, from the Department of Medical Oncology, Milford Hospital (USA), for their important contribution in enrolling patients for this study.

Footnotes

• Check the online version for the most updated information on this article, online supplements, and information on authorship & disclosures: www.haematologica.org/content/101/6/781
• FundingWe are grateful for funding from an Aplastic Anemia and MDS International Foundation grant, a Leukemia and Lymphoma Society Clinical Scholar Award, and a quality of life Grant of the Canadian Cancer Society in memory of James Tyrell (grant # 701837).

• Received December 1, 2015.
• Accepted February 29, 2016.