Br

Published Ahead of Print on October 14, 2008 as 10.1200/JCO.2008.18.3996 The latest version is at http://jco.ascopubs.org/cgi/doi/10.1200/JCO.2008.18.3996

JOURNAL OF CLINICAL ONCOLOGY

E

D

I

T

O

R

I

A

L

Registries That Show Efficacy: Good, but Not Good Enough Mark N. Levine and Jim A. Julian, Department of Oncology, McMaster University; Juravinski Cancer Centre, Hamilton, Ontario, Canada

When readers of Journal of Clinical Oncology (JCO) scan the Table of Contents of an issue, they are likely to recognize designs of interventional research studies such as phase I trials, phase II trials, and randomized phase II and III trials. We suspect, however, that the reader has much less familiarity with research involving observational study designs, administrative databases, phase IV studies, or registries. We have also noticed that these latter terms are often used interchangeably. Our goal in the present commentary is not to provide a primer on epidemiology but to focus on registries and address such questions as “What are they?”, “What can they be used for?”, and “What are some of their limitations?” In recent years, there has been a marked increase in the number of medical registries and in publications using registry data. The word registry is derived from the Latin word registrum, which means list or catalogue. A medical registry can be defined as a systematic collection of a set of health and demographic data for patients with specific health characteristics held in a defined database for a predefined purpose.1-3 Registries were first used to describe disease incidence and as a resource for epidemiologic research. Subsequently, with advances in computer technology making it possible to store large amounts of data concerning treatment of patients with medical disorders, registries have been used to describe patterns of clinical care that generate inferences on quality of care and even effectiveness of therapy. They have even been used to assess safety of a drug after its approval by a regulatory agency after randomized trials (phase IV or postmarketing studies). Some registries contain data on all cases of a particular disease in a defined population such that the cases can be related to a population base. With this information, incidence rates can be calculated. Information on outcomes, such as remission and death, can be obtained if the patients are followed up routinely, either directly or through linkage with other registries or administrative databases (eg, cancer or death registries). There are also registries (such as hospitalbased or disease-specific registries) that are not population based. The intended purpose of a registry should always be prespecified and should define the necessary properties of the data to be collected. There are different types of data sources available that are collected.4-6 For example, administrative data are data that were originally collected for reasons other than research. Encounter data sets maintain a record of health care encounters and typically are maintained by payers to track reimbursement (eg, discharge database). Enrollment data allow the determination of a denominator population from which the encounter numerator is drawn (for example, census data).

The value of a registry greatly depends on the quality of its data. A primary concern is whether the data are accurate and complete. There can be many sources of data errors. Two examples are errors in programming and inaccurate transcription. To ensure the integrity of a registry’s data, a set of procedures should be in place both before data collection, to ensure the highest quality data at the time of collection, and after collection, to identify and correct sources of error.1,4 Examples of the former are a clear definition of data characteristics and study design and training of data collectors, and examples of the latter are data checks and site visits. One example of a registry that has been used as a platform for a number of research publications in JCO is the Surveillance, Epidemiology, and End Results Program (SEER) database.7 The primary purpose of this registry is to provide information on cancer incidence and survival. It covers approximately 26% of the US population. Since January 2006, there have been 16 papers published in JCO that have involved research using SEER. SEER has also been used to study the cost of cancer care at a population level.8 This database is large and comprehensive, with information collected on tumor characteristics including stage, demographic data, surgical intervention, and whether radiation therapy was administered. SEER includes follow-up for vital status and cause of death. Data are collected primarily from an institutional source and not from physicians’ offices, so that information on chemotherapy and hormonal therapy is under-reported. A quality control program is conducted each year by the National Cancer Institute to evaluate the quality and completeness of the SEER data. The strengths and limitations of the SEER registry in general were discussed in a 2003 editorial in JCO by Linda Harlan,9 and the limitations of making inferences on therapy using this registry were also highlighted in more recent editorials.10,11 A suggested guide to the reader when reading a registry article is presented in Table 1. Answering the questions in Table 1 will help in critically appraising the quality of the article. We will discuss some of the limitations of registries when they are used to compare interventions. First, the allocation of patients to the intervention is not random. Therefore, the intervention and comparison groups may differ in ways that affect the study outcome, potentially leading to biased overestimates of benefit (ie, selection bias). Second, follow-up is generally not as active or as standardized as in randomized trials; therefore, ascertainment of outcomes may be incomplete or inaccurate. Moreover, in some cases, outcomes may not

Journal of Clinical Oncology, Vol 26, 2008 DOI: 10.1200/JCO.2008.18.3996; published online ahead of print at www.jco.org on October 13, 2008

© 2008 by American Society of Clinical Oncology

Information downloaded from jco.ascopubs.org and provided by GENENTECH INC on October 14, 2008 from 192.12.78.245. Copyright © 20082008 by theby American SocietySociety of Clinicalof Oncology. All Oncology rights reserved. Copyright American Clinical

1

Levine and Julian

Table 1. Methodology Criteria for Critically Appraising the Quality of a Registry Study Criteria

Questions

Are the study results generalizable to my patients? Is the purpose of the registry clearly stated? Is the data in the registry of high quality? Are the outcome measures reasonable?

What is the population base of the registry? Is it well described? Are the patients highly selected? Can data from the registry answer the question being asked?

What is the patient follow-up? Are groups in the registry being compared, and is this potentially problematic?

Are procedures in place to ensure accuracy and completeness (eg, checks and audits)? Are objective criteria used? Is the assessment done in a blinded fashion? Is the assessment the same in groups being compared (potential for bias)? Is there missing data? Is the loss to follow-up stated? Do the types of patients participating in the comparison groups differ (potential for bias)? Does the information collected on the patients differ between groups (potential for bias)? Are there important factors that either have not been collected or have not been used in the analysis that can affect both group membership and outcome (confounder bias)? Is the analysis appropriate?

be assessed by individuals who are blinded to the intervention allocation, leading to further potential for bias. In other cases, follow-up is conducted passively through linkages to administrative databases. Because the coding of events in these data sets is not primarily for research purposes (and because coding is tied to other incentives such as reimbursement), these methods to ascertain outcomes, although powerful, may be prone to misclassification of outcomes as a result of coding errors and variations in coding practices. Third, because data collection for registries is often more passive than data collection in randomized trials, missing data may be a greater potential problem for registries. Finally, although registries are typically more generalizable to real-world practice because of their observational design, entry into a registry may not be as strictly monitored compared with randomized trials. This creates the potential for ineligible patients to enter the registry and may weaken the generalizability of findings obtained from analysis of registry data. To minimize these threats to validity, analyses of registry data should carefully select a comparison group that minimizes selection bias, describe differences in important prognostic factors between intervention and comparison groups, and statistically adjust for these differences.12-15 It is important to note, however, that even the most sophisticated statistical methods cannot correct for differences in unmeasured or unknown confounding factors. In addition, estimates of effectiveness may be highly sensitive to the analytic method used.16,17 Assessment of outcomes should be performed by individuals blinded to allocation where possible. This is often achieved through linkage to administrative data but, as discussed earlier, comes with its own set of limitations. To address these limitations, critical data elements from administrative databases can be validated through chart review. Finally, to optimize the quality of registry data (eg, minimize missing data and ensuring adherence to eligibility criteria), regular random audits of registry data can be performed. Three recent publications in JCO provide examples of the potential pit2

© 2008 by American Society of Clinical Oncology

falls that can occur when registry data are used to suggest that an intervention works.18-20 In a recent edition of JCO, Hillner et al18 reported the initial results from the National Oncologic PET Registry (NOPR). Considerable thought and effort went into the design, creation, and implementation of this registry, in which data are supplied by participating centers via the internet.21,22 The primary research goal was to assess the effect of positron emission tomography (PET) on referring physicians’ plans of intended patient management. Some procedures were implemented to ensure that the data were of high quality. There were a series of case report forms that were completed according to a schedule. Briefly, the referring physician requested the PET scan and completed a pre-PET form, which included the reason for the PET imaging, cancer type, performance status, and planned management if PET was not available. Patients were asked to provide consent to have their data included in the research database. Once the PET scan was completed, the PET report was uploaded to the database. The final step was the completion of a post-PET form by the referring physician. Fifteen priority areas for early assessment based on tumor type and indication were determined. The NOPR working group established a plan to analyze the data in three phases. In the current report, the first phase of the evaluation in which all cancers are aggregated together across all indications is presented. Hillner et al18 should be congratulated on their remarkable achievement. It is less than 2 years since the registry opened, and results on 34,000 patients registered in the first year have already been analyzed and published. Overall, physicians changed their intended management in 36.5% of patients after PET. Whether the initial indication for PET was diagnosis, initial staging, restaging, or suspected recurrence, there was a major change in management of the patient in approximately one third of the cases. At first pass, the large numbers and magnitude of the benefit associated with PET are impressive. However, an important question is, “How do the results apply to my patient in clinic?” In methodology jargon, this is referred to as generalizability. The results of the registry are presented aggregated by PET indication and not by tumor type, stage, or clinical scenario. We do not know what other imaging tests were performed (or not) before PET. There is no control group in the registry. So the question is, “PET changed management compared with what?” It is not known whether the same changes in management would have occurred with simpler, cheaper tests. This is an important question in many environments concerned about health care costs. In patients with planned biopsy before PET, biopsy was avoided in approximately 70%. The avoidance of a biopsy is important because it saves a patient from potential risk. However, it is possible that, in some cases, a biopsy would have been necessary and the PET-driven strategy may have been wrong. We do not know how often this occurred. Finally, in the NOPR, the referring physician indicated his or her intended change in plan, not what he or she actually did. There is abundant evidence from the literature that what physicians say they will do and what they actually do in practice is not always the same.23 Hence, it is possible that the 30% change in management is an overestimate of the true effect. Hillner et al plan to link their results with Centers for Medicare and Medicaid Services billing records to address this issue, and a more complete understanding of the meaning of these initial published data may then become more evident. In the accompanying editorial to the NOPR article, Larson24 describes the strengths of the study and also points out the limitation JOURNAL OF CLINICAL ONCOLOGY

Information downloaded from jco.ascopubs.org and provided by GENENTECH INC on October 14, 2008 from 192.12.78.245. Copyright © 2008 by the American Society of Clinical Oncology. All rights reserved.

Editorial

that the end point was based on intended patient management rather than actual management. He argues that the NOPR is a clinical trial and an alternative to the randomized trial, which he deems to be impractical in the setting of evaluating imaging technology.24 However, as discussed earlier, studies using observational data can be subject to unrecognized biases. Equipoise is the underpinning of the question addressed in any randomized trial: “I don’t know if PET improves patient management.” In the NOPR, it is possible that physicians who participated were already convinced that PET changes clinical management, and this could have influenced their selection of patients and their recording of change in management, resulting in a systematic overestimate of the magnitude of the benefit. With more than 20,000 PET patients enrolled onto the NOPR, there would have been sufficient patient numbers to carry out a series of carefully controlled trials in a number of indications. To underscore that randomized trials should be part of the process of technology assessment, the Ontario Ministry of Health and Long Term Care is funding randomized trials and prospective cohort trials to introduce PET into that province.25,26 Two studies based on registries are published in this issue of JCO. The BRiTE registry collected data on 1,953 patients with metastatic colon cancer who started chemotherapy plus bevacizumab between February 2004 and July 2005.19 The primary objective of BRiTE was to collect information on adverse events, and a secondary objective was to describe the effectiveness of bevacizumab in terms of progressionfree survival (PFS) and overall survival. The BRiTE population includes a broader range of patients than in the pivotal randomized trial.27 There was no formal assessment of effectiveness end points according to prespecified guidelines. The article by Grothey et al19 is a hypothesis-generating, unplanned analysis of the BRiTE trial that compares two subgroups of patients who experienced progression of their colon cancer—patients who continued on bevacizumab and those who did not. The survival of the patients who continued on bevacizumab was statistically significantly longer compared with the survival of those who did not. The authors conclude that “continued VEGF inhibition with bevacizumab beyond progressive disease could play an important role.” We would like to highlight a few important points. Yes, the registry did include a broader range of patients (including the elderly) and chemotherapy regimens than in the pivotal randomized trial.27 However, the characteristics of the patient population at the time of disease progression are unknown. Because time of disease progression was the start time for the analysis, it is important to know whether the overall disease burden of the two cohorts was similar at this time. Tumor burden could be reflected by sites of metastases, number of metastases, performance status, prothrombin time, and albumin. These characteristics of the two cohorts at the time of disease progression are not compared, and we do not know whether they differ in terms of important known or unknown confounders. Although the time from first metastasis, when patients were enrolled onto the registry, to progressive disease was the same for both cohorts, the time from first diagnosis of colon cancer to first metastasis for the two cohorts should be described to reassure that there is no underlying difference in the natural history of tumor growth between the two cohorts. The authors used a propensity score to adjust for potential imbalances between groups for important characteristics. However, there are limitations to this approach.17 Thus, on the basis of such a strong likelihood of bias from confounders, the results of the analysis can only be considered as explorwww.jco.org

atory in nature. The duration of bevacizumab therapy is an important question and can only be answered by a randomized trial, which has already been initiated (Southwest Oncology Group 0600); in this trial, patients experiencing progression on oxaliplatin-based chemotherapy plus bevacizumab in first-line therapy will be randomly assigned to either stopping bevacizumab or continuing it with secondline irinotecan-based chemotherapy. The Monoclonal Antibody Erbitux in a European Pre-License (MABEL) study in this issue of JCO can also be considered a registry.20 In this study, 1,147 patients with metastatic colorectal cancer who had recently experienced progression on irinotecan and who expressed epidermal growth factor receptor received cetuximab plus irinotecan. The goals for the study were to provide safety and efficacy data on cetuximab plus irinotecan and to confirm the findings of the Bowel Oncology with Cetuximab Antibody (BOND) trial in a community setting. In the BOND trial, 329 patients with epidermal growth factor receptor– expressing metastatic colorectal cancer that had progressed within the previous 3 months on irinotecan were randomly assigned to cetuximab (n ⫽ 111) or cetuximab plus irinotecan (n ⫽ 218) in the same dose and schedule as before random assignment.28 In the study by Grothey et al,19 two groups within the BRITE registry are compared. In contrast, in the study by Wilke et al,20 the MABEL cohort of patients is compared with patients in the BOND randomized trial.28 Is it reasonable to compare the results of this registry to those from a randomized trial? The first question we ask is, “Are the two groups comparable?” The median age in MABEL was 62 years compared with 59 years in BOND. In both studies, approximately 64% of patients were male, and 80% had had two or more prior chemotherapy regimens. In the MABEL study, 1% of patients had a Karnofsky performance score less than 80 compared with 12% of patients in BOND. The most common dose of the irinotecan regimen was 180 mg/m2 every 2 weeks in both MABEL and BOND. Thus, the answer to the question on the comparability of the two patient populations is probably yes. Our next question is, “Is it reasonable to compare the results of registry data with those from a randomized trial?” The answer to this question depends on the outcome measure. The eligibility criteria for a randomized trial can be restrictive. Accordingly, regulatory agencies are interested in additional safety information on a new agent after registration. Thus, a phase IV study can provide information on drug safety in a larger number of patients and in a broader range of patients. In addition, unrecognized toxicities can sometimes be identified. The MABEL investigators used standard criteria to document toxicity. It is reassuring that, as in BOND, the most common irinotecan-related toxicity was diarrhea, the most common cetuximab-related toxicity was acne-like rash, and the rates of these toxicities in the two cohorts were similar. The answer to the question of whether it is reasonable to compare MABEL with BOND for the outcomes of safety is yes. However, there are major limitations in performing cross-study comparisons for the end point of efficacy. The two studies assessed response and PFS using different time intervals; these outcomes were adjudicated by a committee blinded to treatment allocation in the BOND trial compared with the treating physician in MABEL. The median PFS in MABEL was 3.2 months compared with 4.1 months in the BOND trial. These two rates cannot be compared statistically but only in a qualitative sense. The MABEL investigators interpret the PFS results from the two studies to be similar. However, there is an almost 25% difference between the two rates. Given the short PFS of the © 2008 by American Society of Clinical Oncology

Information downloaded from jco.ascopubs.org and provided by GENENTECH INC on October 14, 2008 from 192.12.78.245. Copyright © 2008 by the American Society of Clinical Oncology. All rights reserved.

3

Levine and Julian

patient population and the imprecision in measurement of progression, it would seem that comparison of efficacy results between the registry trial and the randomized trial is limited and not a worthwhile endeavor. If the PFS for MABEL was 2.1 months, what would the investigators have concluded? Given this result, would they still recommend use of the irinotecan plus cetuximab combination? Finally, in an unplanned retrospective analysis, severe infusionrelated reactions were examined. The rate of such reactions was reduced in patients receiving antihistamine plus corticosteroid compared with patients who received antihistamine alone. The administration of corticosteroids was not controlled, and thus, the two patient groups could differ in potential important confounders, potentially leading to a biased comparison.14 Nonetheless, this potentially important observation requires validation in a larger prospective cohort. The randomized controlled trial provides the highest level of evidence for the comparison of alternative types of interventions in clinical practice.29 However, these trials often take a long time to complete, are expensive, and frequently included highly selected patient populations. For these reasons, there is constant interest in finding alternatives to randomized trials. Registries can provide useful information but should be performed according to high methodologic standards to ensure completeness and minimize bias. Care must be taken when they go beyond their original purpose and attempt to answer effectiveness questions. This point, made in a seminal article by David Byar2 more than a quarter of a century ago, is still important today. AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The author(s) indicated no potential conflicts of interest. AUTHOR CONTRIBUTIONS

Manuscript writing: Mark N. Levine, Jim A. Julian Final approval of manuscript: Mark N. Levine, Jim A. Julian REFERENCES 1. Arts DGT, de Keizer NF, Scheffer GJ: Defining and improving data quality in medical registries: A literature review, case study, and generic framework. J Am Med Inform Assoc 9:600-611, 2002 2. Byar DP: Why data bases should not replace randomized clinical trials. Biometrics 36:337-342, 1980 3. Mack MJ: Clinical trials versus registries in coronary revascularization: Which are more relevant? Curr Opin Cardiol 22:524-528, 2007 4. Pronovost P, Angus DC: Using large-scale databases to measure outcomes in critical care. Crit Care Clin 15:615-631, 1999 5. Pronovost PJ, Berenholtz SM, Dorman T, et al: Evidence-based medicine in anesthesiology. Anesth Analg 92:787-794, 2001 6. Rubenfeld GD, Angus DC, Pinsky MR, et al: Outcomes research in critical care: Results of the American Thoracic Society Critical Care Assembly Workshop on Outcomes Research. Am J Respir Crit Care Med 160:358-367, 1999 7. National Cancer Institute: Surveillance Epidemiology and End Results (SEER). http://seer.cancer.gov/ 8. Yabroff KR, Lamont EB, Mariotto A, et al: Cost of care for elderly cancer patients in the United States. J Natl Cancer Inst 100:630-641, 2008

9. Harlan LC, Hankey BF: The Surveillance, Epidemiology, and End-Results program database as a resource for conducting descriptive epidemiologic and clinical studies. J Clin Oncol 21:2232-2233, 2003 10. Cannistra SA: Gynecologic oncology or medical oncology: What’s in a name? J Clin Oncol 25:1157-1159, 2007 11. Schrag D: Enhancing cancer registry data to promote rational health system design. J Natl Cancer Inst 100:378-379, 2008 12. Feinstein A: The role of observational studies in the evaluation of therapy. Stat Med 3:341-345, 1984 13. Green S, Byar DP: Using observational data from registries to compare treatments: The fallacy of omnimetrics. Stat Med 3:361-373, 1984 14. Mamdani M, Sykora K, Li P, et al: Reader’s guide to critical appraisal of cohort studies: 2. Assessing potential for confounding. BMJ 330:960-962, 2005 15. Normand SL, Sykora K, Li P, et al: Readers guide to critical appraisal of cohort studies: 3. Analytical strategies to reduce confounding. BMJ 330:10211023, 2005 16. Stukel TA, Fisher ES, Wennberg DE, et al: Analysis of observational studies in the presence of treatment selection bias: Effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. JAMA 297:278-285, 2007 17. D’Agostino RB Jr, D’Agostino RB Sr: Estimating treatment effects using observational data. JAMA 297:314-316, 2007 18. Hillner BE, Siegel BA, Liu D, et al: Impact of positron emission tomography/ computed tomography and positron emission tomography (PET) alone on expected management of patients with cancer: Initial results from the National Oncologic PET Registry. J Clin Oncol 26:2155-2161, 2008 19. Grothey A, Sugrue MM, Purdie DM, et al: Bevacizumab beyond first progression is associated with prolonged overall survival in metastatic colorectal cancer: Results from a large observational cohort study (BRiTE). J Clin Oncol doi;10.1200/JCO.2008.16.3212 20. Wilke H, Glynne-Jones R, Thaler J, et al: Registrational study of cetuximab plus irinotecan in heavily pretreated metastatic colorectal cancer progressing on Irinotecan: MABEL. J Clin Oncol doi;10.1200/JCO.2008.16.3758 21. Hillner BE, Liu D, Coleman RE, et al: The National Oncologic PET Registry (NOPR): Design and analysis plan. J Nucl Med 48:1901-1908, 2007 22. Lindsay MJ, Siegel BA, Tunis SR, et al: The National Oncologic PET Registry: Expanded medicare coverage for PET under coverage with evidence development. AJR Am J Roentgenol 188:1109-1113, 2007 23. Siminoff LA, Fetting JH, Abeloff MD: Doctor-patient communication about breast cancer adjuvant therapy. J Clin Oncol 7:1192-1200, 1989 24. Larson SM: Practice-based evidence of the beneficial impact of positron emission tomography in clinical oncology. J Clin Oncol 26:2083-2084, 2008 25. Maziak D, Darling GE, Inculet RI, et al: A randomized controlled trial (RCT) of 18F-fluorodeoxyglucose (FDG) positron emission tomography (PET) versus conventional imaging (CI) in staging potentially resectable non-small cell lung cancer (NSCLC). J Clin Oncol 26; 397s, 2008 (suppl, abstr 7502) 26. Pritchard KI, Julian J, McCready D, et al: A prospective study evaluating 18F-fluorodeoxyglucose (18FDG) positron emission tomography (PET) in the assessment of axillary nodal spread in women undergoing sentinel lymph node biopsy (SLNB) for breast cancer. J Clin Oncol 26; 14s, 2008 (suppl, abstr 533) 27. Hurwitz H, Fehrenbacher L, Novotny W, et al: Bevacizumab plus irinotecan, fluorouracil, and leucovorin for metastatic colorectal cancer. N Engl J Med 350:2335-2342, 2004 28. Cunningham D, Humblet Y, Siena S, et al: Cetuximab monotherapy and cetuximab plus irinotecan in irinotecan-refractory metastatic colorectal cancer. N Engl J Med 351:337-345, 2004 29. Lachetti C, Guyatt G: Therapy and validity: Surprising results of randomized controlled trials, in Guyatt GH, Rennie D (eds): Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. Washington, DC, AMA Press, 2001, pp 247-265

■ ■ ■

4

© 2008 by American Society of Clinical Oncology

JOURNAL OF CLINICAL ONCOLOGY

Information downloaded from jco.ascopubs.org and provided by GENENTECH INC on October 14, 2008 from 192.12.78.245. Copyright © 2008 by the American Society of Clinical Oncology. All rights reserved.