Center-level variation in accuracy of adverse event reporting in a clinical trial for pediatric acute myeloid leukemia: a report from the Children’s Oncology Group

While cure rates for pediatric acute myeloid leukemia (AML) range from 50–60% in Children’s Oncology Group Trials (COG)21 and 60%–70% in European cooperative groups,43 AML therapy is very intensive and causes substantial treatment-related morbidity. We recently showed that adverse events (AE) are under-reported on cooperative group clinical trials for de novo pediatric AML.5 Other investigators have also demonstrated heterogeneity in the types of AEs reported.76 Given the limited resources available on cooperative group trials,8 there is potential for significant variation between hospitals as resources and staffing for AE reporting are often impacted by hospital resources and priorities.9 Center-level variation in cancer treatment outcomes has been demonstrated based on hospital volume, with improved outcomes at hospitals with greater volumes.10 This study is an extension of our previously published work5 and sought to determine if there is center-level variation in AE reporting in order to better understand under-reporting of AEs in pediatric AML clinical trials. We hypothesized that there would be variation in AE reporting between hospitals and across AEs, and that hospitals that treat more children with AML would report AEs more robustly. Using gold standard chart abstraction data collected on 12 AEs at 14 hospitals, this study found that AE reporting sensitivity varied by hospital for anorexia (P<0.001), viridans group streptococcal (VGS) bacteremia (P=0.016), and pain (P=0.001) with a trend toward improved sensitivity at larger centers (anorexia: OR 1.7, P<0.001; pain: OR 1.5, P=0.022). This study demonstrates center-level variation in AE reporting and concludes that the association between hospital volume and reporting accuracy should be evaluated in larger cohorts.

Children’s Oncology Group (COG) trial AAML0531 randomized eligible patients to receive standard chemotherapy with or without gemtuzumab ozogamicin.1 As per standard COG procedures, grades 3 and higher non-hematologic AEs were collected on AAML0531 via manual review of the medical record by clinical research associates (CRAs), as noted in our prior publication.5 For the current study, two pediatric oncologists (T.P.M, M.K.) performed a retrospective medical record abstraction for all patients with available medical records at fourteen hospitals across the United States to evaluate the accuracy of AE reporting.5 The following grade 3–4 AEs were examined on each inpatient day for all chemotherapy courses: hypertension, hypotension, hypoxia, acute respiratory distress syndrome (ARDS), microbiologically-proven viridans group streptococcal (VGS) bacteremia, microbiologically-proven sterile site invasive fungal disease (IFD), anorexia, typhlitis, disseminated intravascular coagulation (DIC), pain, seizure and acute renal failure. AEs were selected to represent all organ systems and a range of clinically important AEs. Prior to initiation of chart abstraction, AE definitions were developed based on the Common Terminology Criteria for Adverse Events (CTCAE) version 3.

Patient demographic, disease and treatment characteristics were obtained from the COG AAML0531 database. As previously reported, the distributions of patient characteristics of abstracted and non-abstracted patients on AAML0531 were compared using χ or Wilcoxon rank sum tests.5 Specifically, Compared with all patients in AAML0531, chart abstraction patients were younger (median age, 8.4 years vs. 10.0 years; P=0.05), and a greater percentage were black (P=0.02) and had a higher WBC count at presentation (median WBC, 32.8 vs. 22.6; P=0.01). All enrolled patients were included in chart abstractions at centers participating in the Pediatric Health Information Systems Database (PHIS). PHIS data are restricted to free-standing pediatric hospitals.11 Chart data were used as the gold standard to determine sensitivity of trial AE reporting for each AE and by hospital.5 Fisher’s exact test or chi square test compared the sensitivity of AE reports across hospitals; χ test was used for AEs with larger numbers of courses with the AE. Logistic regression was used to determine if sensitivity was associated with AML patient volume at a hospital. All analyses used course-level data, and each course was considered a separate event based on prior research that showed no differences when generalized estimating equation models adjusted for within-subject and within-hospital correlations.5 A two-sided P-value of 0.05 was considered statistically significant. Statistical analyses were performed in STATA version 12 (College Station, TX).

As previously described, chart abstraction was performed on 204 patients (758 courses, 20.0% of all patients on AAML0531) at fourteen hospitals.5 The median number of patients per hospital was 18 (range 5 to 25). The average number of patients per year per hospital had a median of 3 (range 2 to 6). The median number of courses per hospital was 72 (range 18 to 102).

The sensitivity of AE reports varied widely (0% to 78.3%), as did the sensitivity at individual hospitals (0% to 100%) (Table 1). There were statistically significant differences in the sensitivity of clinical trial AE reporting between hospitals for anorexia (P<0.001), VGS (P=0.016), and pain (P=0.001) (Tables 1 and 2). Despite occurring often in chart data at all sites, pain was never reported at three hospitals, anorexia was never reported at four hospitals, and hypoxia was never reported at six hospitals. Some hospitals had higher sensitivity across all AEs, while other hospitals reported some AEs and did not report other AEs at all. For example, one hospital had sensitivity of at least 75% for hypotension and VGS, but less than 25% for hypoxia and pain. Another hospital had sensitivity greater than 0% for only one AE.

Table 1.Variation in sensitivity of reporting adverse events by hospital.

Table 2.Adverse event sensitivity by number of courses per hospital.

Figure 1 shows a trend toward higher sensitivity of trial AE reports at hospitals that administered larger numbers of chemotherapy courses. This is also demonstrated with the trend toward improved sensitivity for the majority of AEs at sites with more chemotherapy courses administered in Table 2. Logistic regression was performed for hypoxia, anorexia, VGS, and pain compared to the number of courses per site as these AEs each had more than 100 AEs in chart data. For every increase in 20 courses of chemotherapy at a hospital, there was a 1.7-fold increased rate of correctly-reported anorexia (OR 1.7, 95% CI 1.290–2.314, P<0.001) and 1.5-fold increased rate of correctly-reported pain (OR 1.5, 95% CI 1.062–2.155, P=0.022). There was a trend towards improved AE reporting for VGS (OR 1.6, 95% CI 0.986–2.690, P=0.057) and no association for hypoxia.

Figure 1.Relationship between the number of courses per hospital and sensitivity of all clinical trial adverse event reports by hospital.

This study demonstrates substantial variation between hospitals in sensitivity of AE reporting on a pediatric AML clinical trial. For the frequently identified AEs of pain, anorexia, and VGS, there were statistically significant differences in AE reporting sensitivity between hospitals. This indicates that not only are AEs under-reported on clinical trials,5 but also that reporting varies substantially between treating institutions.

The variation identified in AE reporting by institution is likely multifactorial and has concerning implications. Although data collection and analysis beyond the scope of this letter would be needed to identify these factors, several possibilities include the following: ambiguity in CTCAE definitions; variation in research staff availability, training, experience and workload compared to number of open trials; and variable familiarity with experimental agents. Since prior work has shown improved outcomes at hospitals treating more cancer patients,10 larger patient volume may also improve AE reporting accuracy. Given the large number of COG centers, it seems likely that this variation would result in inaccuracies and overall AE under-reporting rather than differential reporting by study arm, but further work is needed to confirm this assessment.

Of note, Good Clinical Practice (GCP) training at each site is a shared responsibility between the site and COG. Specifically, each site is responsible for ensuring that all GCP training requirements are met for each individual participating in the COG clinical trial. In addition, COG provides formalized training for clinical research associates (CRAs) in adverse event reporting. This training includes both general and trial specific components. Large phase III trial such as AAML0531 are typically not monitored by the study sponsor. However, there is regular and standardized auditing of data at each COG site. Notably, VGS was the only AE for which all hospitals had AE reports when an event had occurred in chart data. This indicates that the additional guidance provided for this targeted toxicity helped improve sensitivity across hospitals. Due to limited resources and limited research staff time available to devote to AE reporting,128 this result is not unexpected.5 Judicious use of targeted AE reporting may minimize the impact of variation in center-level reporting at least for specific AEs of particular clinical interest.

In conclusion, patterns of AE under-reporting must be identified so that interventions can be implemented to standardize AE reporting. Standardized systems and guidance on AE ascertainment combined with improvement of AE definitions could lead to more reproducible AE reporting and, ultimately, improve the accuracy of this important component of clinical trials.