Economic Evaluation In Critical Care

Economic evaluation of new therapies in critical illness Michael T. Coughlin, MA; Derek C. Angus, MD, MPH

The recent Food and Drug Administration approval of drotrecogin alfa (activated) and the potential of several other new therapies may represent the beginning of a breakthrough in the management of critical illness in the intensive care unit. However, their use in clinical practice will likely be dependent on a rigorous appraisal not only of their effects, but also of their costs. Novel therapies can no longer be judged simply by their effectiveness in treating illness, but must also be evaluated on an institutional and societal level on the basis of their cost. These considerations have important implications for the practicing intensivist, who will need to better understand the

T

he care of critically ill patients in a modern intensive care unit (ICU) results in a large societal burden in terms of both manpower and monetary cost. The high cost of critical care can largely be attributed to high overhead costs (e.g., need for experienced staff and expensive equipment), high resource utilization (pharmaceuticals, laboratory tests, and imaging procedures), and high demand for ICU services. With the continued increase in healthcare costs, there is an increasing need to establish whether new therapies are not only effective, but also cost-effective. Although this is true throughout medicine, the issue of costeffectiveness is especially important in critical care medicine. ICU costs in the United States exceed $150 billion, representing up to one third of all hospital costs (1). Furthermore, attempts to reduce ICU costs by other mechanisms, such as reduction in length of stay, have proved to be difficult (2). The concern over the financial effect of new therapies in the ICU is so intense that scrutiny begins even before therapies are approved by the Food and Drug Administration. Before ever gaining ap-

From the Clinical Research, Investigation, and Systems Modeling of Acute Illness (CRISMA) Laboratory, Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, PA. Copyright © 2003 by Lippincott Williams & Wilkins DOI: 10.1097/01.CCM.0000045033.69843.FD

Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)

conduct and design of economic evaluations, including their strengths and weaknesses. In this article, we review the rationale behind economic evaluations of new therapies and the alternative economic approaches available. We then discuss in more detail the elements contained in a cost-effectiveness analysis, the preferred approach to pharmacoeconomic evaluation today. (Crit Care Med 2003; 31[Suppl.]:S7–S16) KEY WORDS: pharmacoeconomics; cost-effectiveness; critical care; intensive care unit; outcomes research; new therapies; sepsis

proval, the antiendotoxin monoclonal antibody HA-1A stimulated considerable furor and debate—not only in the medical literature, but also in the national media (3–7)— over its anticipated cost. Currently, the Food and Drug Administration does not explicitly consider cost when evaluating new therapies. However, the high anticipated costs of new biologic agents considered in the treatment of sepsis and severe infections have placed pressure on the agency. It is perhaps as a consequence of this pressure that many recent antisepsis biologic therapies have been burdened with proving their ability to decrease mortality to gain Food and Drug Administration approval (8 –12). This burden is greater than that faced by many less expensive therapies (e.g., antibiotics). Even if the Food and Drug Administration approves a new therapy, access to the therapy is increasingly restricted by pharmacy and therapeutic committees. These committees are given the responsibility of reviewing hospital formularies to determine the value, or cost-effectiveness, of new products. With the formation of large hospital networks and managed care organizations, such review is becoming increasingly centralized, and the physician is gradually losing control over what drugs are available for the treatment of patients. Furthermore, the decision-making process by pharmacy and therapeutic committees is often not driven by in-depth economic analyses of

the likely effect of new therapies but, rather, is made simply on the basis of anticipated drug acquisition costs. Drug acquisition costs are important, but these must also be combined with data on efficacy and hospital costs to evaluate the total economic effect of new therapies. However, although access to new therapies is being tied increasingly to cost, and not simply effect, physicians continue to be skeptical and suspicious of cost analyses. In part, skepticism is prompted by the variable quality of earlier cost analyses. However, skepticism is also due to the lack of familiarity many physicians have with the general principles of health economics. Efforts by the US Public Health Service have attempted to set standards for the conduct of cost-effectiveness analyses in medicine (13– 15). More recently, the American Thoracic Society (ATS) established specific guidelines for the conduct of cost-effectiveness analyses in critical care on the basis of the US Public Health Service recommendations (16). These standards are likely to greatly improve the rigor of future costeffectiveness studies. However, if clinicians do not embrace the principles of costeffectiveness assessment, it is likely that such studies will continue to be viewed simply as ammunition for nonclinician administrators in a war to restrict physician autonomy and control clinical decisionmaking. In this review, we cover many of the principal aspects of cost-effectiveness analyses and consider how such studies S7

ought to be conducted on future therapies. Throughout this work, we will use a theoretical newly developed sepsis therapy as a working example. The goal of this review is to encourage all of us to familiarize ourselves with cost analyses and to consider cost-effectiveness analysis (CEA) as simply another tool in our evaluation toolset, to sit alongside case reports, animal studies, and randomized controlled trials (RCTs) of new drug therapies. For an in-depth discussion of economic analysis in health care, the reader is referred to the text by Gold et al (17).

ECONOMIC QUESTIONS IN HEALTH CARE One can distill all of health economics down to two main questions. The first asks, “Is a given therapy worth using when compared with alternatives?” For example, what is the worth of a new antisepsis agent that has recently been demonstrated in a large RCT to have some beneficial effect? We can consider that

worth represents some trade-off of cost and benefit (or effect). The second question is broader and asks, “Should a portion of available healthcare resources be allocated to a given therapy or program?” This is largely a social policy issue and requires considering the worth of new programs or therapies not only within a given disease, but also in comparison with other therapies in other diseases. For example, although a new therapy to fight sepsis might be deemed worthwhile in the treatment of sepsis, a state Medicaid agency might be forced to compare its value with that of a hepatitis B vaccination program for newborns or influenza vaccinations for the elderly. These types of choices become increasingly frequent in situations in which healthcare resources are limited and must be equitably distributed among multiple programs. In other words, we would wish to know of a new sepsis agent not only whether it is cost-effective with respect to standard management for sepsis, but also

whether we can afford it as a nation. There are a variety of different analytic approaches that address all or some part of these questions. Although the approaches seem similar and have similar names, there are key differences. To understand the strengths and weaknesses of each approach, we must first describe the various methodologies used to evaluate costs in health care. There are essentially four types of cost analysis: cost minimization, cost benefit, cost-effectiveness, and cost utility (Table 1). The fourth, cost-utility, is best viewed as a special case of cost-effectiveness. In addition, there are situations in which a CEA can produce a cost-minimization statement.

Cost-Minimization Analysis The cost studies perhaps most familiar to clinicians are focused solely on how much a drug costs to put on the pharmacy shelf. These studies are traditionally called cost-minimization studies. They are essentially drug acquisition cost

Table 1. Types of cost analyses

Type of Study

Numerator, Costs

Denominator, Outcome or Benefit

Cost minimization

Dollars

None

Cost benefita

Dollars

Dollars

Comment Antibiotic therapy for ICU patients at low risk of nosocomial pneumonia Singh et al.44 (2001) Drug acquisition costs for a 3-day course of ciprofloxin are $9,520 cheaper than average acquisition costs for unregulated antibiotic prescription Use of an aminoglycoside dosemonitoring program for burn patients with Gram-negative sepsis 45

Bootman et al. (1979) The dose-monitoring program led to $8.70 savings per dollar spent Cost effectiveness

Dollars

Specific measure of effectiveness (lives saved)

Cost utilitya

Dollars

A common utility metric (quality adjusted life years)

Thrombolysis for acute myocardial infarction Mark et al.46 (1995) Tissue plasminogen activator costs an additional $32,678 per additional life saved when compared with streptokinase Prophylaxis against recurrence of peptic esophageal strictures Stal et al.47 (1998) Omeprazole costs an additional $49,600 per additional QALY when compared with ranitidine

No estimate of consequences on other healthcare costs; clinical outcomes are assumed to be equivalent (i.e., no difference in subsequent pneumonia rate or mortality) even though a formal equivalence study was not conducted A key advantage is that all costs and effects are expressed in monetary units (dollars), facilitating assessment of worth; however, the key concern is that converting clinical effects, such as lives lost (or gained), into dollar amounts is controversial, somewhat arbitrary, and biased toward saving the lives of those with greater earning capacity Assesses change in both costs and effects but avoids controversy of converting clinical outcomes into dollar values; it is not clear whether “lives saved” are equivalent to other lives saved by other therapies in other diseases Cost per QALY allows comparison to other therapies used in other diseases; this is now the recommended approach.

ICU, intensive care unit; QALY, quality-adjusted life years. Because cost benefit and cost utility analyses produce a common metric, they can be used to compare studies that evaluate different outcomes. Reproduced with permission from Angus et al (16). a

S8

Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)

studies and are conducted frequently by hospital pharmacy departments. When different products are compared (e.g., two sulfonamides), each product is assumed to have equal efficacy and effect on all other aspects of treatment (although this may or may not be true). Effects such as shortened length of stay, reduced need for other therapies, and improved quality of life after illness are not considered in cost-minimization analyses. The preferred product is simply the one that costs the hospital less money per unit of treatment (e.g., per day of therapy or per dose). In addition, a cost-minimization analysis can result when a formal CEA (see below) with sophisticated assessment of all potential changes in costs and effects between two programs demonstrates no difference in effect. Nonetheless, in the situation where there is no difference in effect, there may be significant differences in cost between the programs. This result does not produce a cost-effectiveness ratio (because one would be dividing the change in costs by zero), but it does allow for accurate assessment of the true differences in cost between two programs with comparable treatment effects. This explicit evaluation of costs and effects, as opposed to an assumption of no difference in effect, is in contrast to traditional cost-minimization studies performed in the past.

Cost-Benefit Analysis The term cost-benefit is frequently confused with cost-effectiveness (18). In fact, a cost-benefit analysis is a very specific analysis, rarely conducted today, which expresses all costs and effects in monetary units. This means that a dollar value must be placed on all effects. For example, a life saved must be converted into a financial benefit. This conversion of life into monetary terms can be problematic and unintuitive. After conversion of all effects into monetary units, one then adds up all the costs (expressed in dollars) and subtracts them from all the benefits (effects, expressed in dollars). If the final total is negative, the costs outweigh the benefits, and vice versa. Although the final output is attractive in its simplicity, the manipulations required to convert all effects into dollar values are inevitably controversial. Because of the controversy concerning the values of effects, this type of analysis has largely Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)

fallen out of favor for pharmacoeconomic evaluations.

Cost-Effectiveness Analysis Perhaps the most misused phrase is the term cost-effectiveness. For example, many confuse the term with cost saving and effective. In fact, cost-effectiveness is simply a ratio of the net change in costs (dollars) associated with two different programs or therapies divided by the net change in effects (health outcome). The denominator represents the gain in health (life-years gained, number of additional survivors, or cases of disease averted), whereas the numerator reflects the cost (in dollars) of affecting the gain in health. Because the units used are different for the numerator and denominator, the typical cost-effectiveness expression will take the form such as cost (dollars) per year of life saved. Cost-effectiveness analyses are the dominant form of cost analysis today and were endorsed by both the US Public Health Service Panel on Cost-effectiveness in Health and Medicine (PCEHM) and more recently by the ATS as the principal method by which to assess the costs and effects of healthcare programs and therapies (14, 16). Deciding whether a therapy is costeffective is a subjective interpretation of the cost-effectiveness ratio. In other words, if $100,000 per year of life gained is deemed the cutoff for effectiveness, then a new therapy with a cost-effectiveness ratio of $82,000 per year of life gained is viewed as cost-effective. Although there is no absolute cutoff, there is general consensus that a level somewhere between $50,000 and $100,000 per year of life gained is deemed acceptable in

the United States today. Needless to say, however, the way in which both costs and effects are calculated can have profound effects on the resulting ratio, and herein lies much of the controversy over CEA. For reasons that will become clear later in this work, the typical CEA will require the collection of a significant amount of information on costs and effects, much of which may be gathered from widely differing sources. Interpretation of these different pieces of information is often difficult, and a decision analysis model is usually constructed that mimics the key clinical decisions and events. This model can most easily be represented by a tree in which each branch point is calibrated with a probability of occurrence and a cost. At its simplest, the tree will contain only branches for treatment allocation (e.g., new therapy or standard) and outcome (e.g., alive or dead). To calibrate such a tree, we would need to know only the probability of living or dying, depending on whether a given patient received the new therapy or not, and the average cost of care for survivors and nonsurvivors in the two treatment arms (Fig. 1). Alternatively, we may be interested in understanding the key events that drive either morbidity or cost (e.g., mechanical ventilation or hemodialysis). This could be important for a variety of reasons; there may be evidence that the study population has a far lower rate of mechanical ventilation than is expected for septic patients in general. Therefore, the extent to which differences in cost are the result of the number of patients undergoing mechanical ventilation may be important when estimating the cost-effectiveness of

Figure 1. Simple decision tree comparing outcome for patients treated with a new therapy vs. standard care. To calibrate the tree, we must estimate 1) the probability for a given patient to live or die, given whether he or she received the new therapy or not, and 2) the average costs associated with each of the four branches.

S9

the new therapy in the real world. Similarly, need for hemodialysis can be a significant driver of costs. A new therapy that reduces the need for mechanical ventilation or hemodialysis may be expensive, but the cost of the therapy can be offset by the reduced need for supportive care, and the therapy would hence be deemed cost-effective by CEA. Conversely, cost-benefit analysis would see only the expense of the new therapy without considering the reduction in supportive care. However, the addition of branch points for mechanical ventilation and hemodialysis expands the model dramatically, creating 16 branches to consider, and for each, we must know a patient’s likelihood of entering such an arm and the average costs (Fig. 2).

Special Case of CEA Cost-Utility Analysis. A cost utility analysis is a form of CEA in which the effects are converted into common units of utility. Typically, this approach might involve adjusting the number of years of survival for the quality of survival, where a person living for 1 yr with a quality of life score of 80% would be awarded 0.8 yrs of quality-adjusted survival. The advantage of this approach is that it allows comparison of different programs in different diseases. For example, using the number of quality-adjusted life years (as opposed simply to the number of lives saved), we can perhaps more equitably compare our sepsis therapy for critically ill adults with the hepatitis B vaccination program in newborns.

Methodologic Considerations in CEA Good CEA design requires consideration of many elements to both adequately explore the relationship between costs and effects and to determine the robustness of the conclusions and the comparability of the results with those of other studies. These elements are outlined in Table 2 with reference to both the PCEHM and ATS Guidelines and are discussed individually in more detail below.

Perspective The costs considered in a CEA can vary depending on whose perspective is taken. For example, consider the issue of early hospital discharge after childbirth. S10

Figure 2. Decision tree comparing outcomes for patients treated with a new therapy vs. standard care that incorporates the potential to undergo mechanical ventilation and receive hemodialysis. To calibrate the tree, we must estimate the probabilities and average costs for 16 separate trees. MV, mechanical ventilation; HD, hemodialysis.

The cost from the hospital’s or managed care organization’s perspective may be reduced by early discharge. However, from a societal perspective, the cost savings of the healthcare facility or organization may be offset by additional costs to the patient, such as extra time off work for the husband who must stay home to care for the new mother. Previously conducted cost studies have often been hampered by a lack of consistent perspective either within or among studies. The latter problem hampers comparison of results across studies, whereas the former threatens the validity of the study itself. The PCEHM and ATS recommend adoption of the societal perspective when costeffectiveness studies are conducted.

Outcomes (Effects) This is an exceedingly difficult problem for CEA for a variety of reasons. First, information on outcomes usually comes from RCTs, which often do not reflect the actual clinical practice of medicine. Conversely, the implications of a CEA are intended for real-world practice. In other words, a cost-effectiveness ratio is intended to capture the expected relationship between the costs incurred and the effects gained in actual practice. An RCT is usually designed to maximize the like-

lihood of finding an effect. As such, an RCT can represent a rather rarefied situation, which is quite distinct from the real world. For example, only specific patients may be selected, the dosage and timing of therapy will likely be optimized, and other aspects of care may be standardized and carefully controlled. The effect size generated under such rigorous situations is termed a therapy’s efficacy (or maximal effect). In the real world, the effect of a new therapy is likely diluted by less appropriate patient selection, changes in dosing and timing, and increased heterogeneity in other aspects of care. The effect of a new therapy under these real-world conditions is termed a therapy’s effectiveness. The more RCTs are refined, the further removed they are from the reality of using a therapy in clinical practice (19). Thus, the relationship between cost and effect in some RCTs becomes increasingly distorted. A cost analysis conducted by using the effect size generated from an RCT might better be termed a cost-efficacy study rather than a cost-effectiveness study. However, there are no clear guides on how to reduce the bias introduced by using efficacy data instead of effectiveness data. One possibility is to consider adding an open-label, open-enrollment arm to Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)

Table 2. Methodologic considerations in cost-effectiveness analysis Methodologic Problems

Aspect

Individual CEA

Comparing CEAs

Perspective

Not defined

Different

Outcomes, (effects)

Data are inadequate or difficult to evaluate

Different outcomes

Costs

Comparators, (standard care)

Discounting

Data are inadequate or difficult to evaluate

Choice distorts results

Inadequate representation of the effect of time

Different costs

—

Different rates

Uncertainty

Inadequate representation of uncertainty on results

—

Reporting

—

Not standard

PCEHM

ICU-Specific

Long-term follow-up is rare

Only hospital costs are usually measured; no international standard

Determining standard often difficult

Not usually done

Not usually done

Recommendations (Rationale)

Second ATS Workshop on Outcomes Research

Position

Societal (ethical, pragmatic)

Agree

QALYs (pragmatic, conventional)

Agree

Best-designed, least biased source (pragmatic) Costs to include: healthcare services, patient time, caregiving, non–health impacts (theoretical)

Agree

Include or exclude other disease costs and test in SA (theoretical, pragmatic, user needs, and accounting) Existing practice (conventional) If existing practice is suspect, consider bestavailable, viable low-cost, or “do nothing” (conventional) Discount costs and effects to present value (theoretical) Use a 3% discount rate (theoretical, pragmatic) SA essential; multiway SA preferred (user needs)

Include costs of other diseases (too hard to disentangle)

Reference case (user needs) Compare to available ratios (user needs) Journal and technical report (user needs)

Agree

Agree

Comment May be instances when provider perspective is useful Require better natural history of ICU conditions and modeling or longer follow-up; other outcomes may be useful depending on perspective Consider modeling reduced efficacy in sensitivity analysis Standard approach to measuring these costs not yet developed; estimating units of resource use and multiplying by standard costs probably most practical approach currently; detail with which resource use is tracked should be tailored to nature of intervention and likely effects on costs

Agree Agree

Many existing ICU practices may be ineffective or cost-ineffective; therefore, consider comparison to “best practice” rather than standard practice

Agree

Agree

Agree

Multiway sensitivity analyses probably essential given high likelihood that several key assumptions will be necessary to generate reference case from critical care trials But, also present “data-rich” case

Agree

Agree

Also file (e.g., on Internet) intended analysis plan prior to unblinding when concurrent with randomized clinical trial

CEA, cost-effectiveness analysis; ICU, intensive care unit; PCEHM, Panel on Cost-effectiveness in Healthcare and Medicine; ATS, American Thoracic Society; QALY, quality-adjusted life years; SA, sensitivity analysis. Reproduced with permission from Angus et al (16).

Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)

S11

RCTs on which a CEA is being conducted (20). However, this presents many logistic and ethical problems. The more accepted alternative is to expose the cost model to varying estimates of reduced effect from those seen in the RCT, during sensitivity analysis (see below). Another problem encountered when determining effect, or outcome, is that the outcome measure evaluated in the RCT may not be directly relevant in the cost analysis. The PCEHM recommends, and the ATS agrees, that quality-adjusted life years be used as the units of effect, or utility. However, most RCTs in critical care use short-term (day 28 or hospital) mortality as the primary end point, and still others use indices such as organ failure–free days as outcome measures (21). Although short-term survival likely correlates with long-term quality-adjusted survival, the relationship is not explicitly clear. Whether there is any relationship between organ failure–free days and longterm quality of life is even less clear. In fact, a recent study by Clermont et al. (22) showed that patients who develop acute organ dysfunction are at risk for poor long-term quality of life but that the risk is largely due to poor baseline health status and not directly to organ failure during critical illness. This problem is only slowly changing. Although the PCEHM recommends longterm outcome, a National Institutes of Health–sponsored workshop on sepsis studies recommended day 28 mortality (21). More recently, the UK MRC workshop still maintained that day 28 mortality was an appropriate primary end point but recommended follow-up to ⱖ90 days and, whenever possible, to ⱖ6 months (23). Furthermore, recent successful trials in sepsis reported mortality at widely varying time points—28 days (drotrecogin alfa [activated]) (24), 28 days and 1 yr (steroids) (25), and 60 days (early goaldirected therapy) (26)—and a recent study of acute respiratory distress syndrome (low tidal volume) (27) reported mortality to 180 days. Proponents of short-term outcome state that longer follow-up is too expensive and not necessarily related to the therapy being studied. Advocates of longer follow-up state that short-term survival, of indeterminate quality of life, and possibly with death a short time thereafter, is of little societal relevance (17). They further argue that the ability to prioritize healthcare spending on the basis of value requires that we compare the long-term value of alternaS12

tive programs for alternative disease processes. Many healthcare programs are administered, and/or have effects lasting over a long period of time, making longterm follow-up of patients enrolled in these types of programs essential. There is currently relatively little long-term follow-up information on ICU patients. However, the available evidence does suggest that there is considerable mortality and morbidity beyond hospital discharge, supporting the notion that we should consider longer follow-up (28 – 30). Quartin et al. (28) showed that continuing mortality occurs in sepsis patients for many months after discharge from the hospital. Studies exploring quality of life after ICU care have yielded conflicting results, but certainly several suggest a considerable diminution of quality of life, which seems to be sustained over time. Thus, until more evidence is available, studies of new therapies in the treatment of sepsis on which CEAs are intended should have some mechanism (e.g., a subset study or parallel cohort) to incorporate mortality follow-up for 6 –12 months with an accompanying quality of life assessment.

Costs Which Costs Should We Include? Which costs should be included? Debates over this subject can be so contentious as to resemble debates over whether to give colloids or crystalloids to hypotensive patients. The subject is further complicated by economic terms such as direct vs. indirect costs and tangible vs. intangible costs. We will attempt to avoid using too many accounting terms and to suggest alternative ways to understand this issue. Let us go back to the cost-effectiveness ratio. It is a ratio of net costs divided by net effects. Thus, regardless of whether the costs of any given element seem important, if they are distributed equally in both comparison groups, the net difference will be zero, and we therefore need not worry about them. Alternatively stated, we need consider only those costs that we believe to be relevant and likely to differ between the treatment and control groups. As an example, the PCEHM believed that the intangible costs of pain and suffering were relevant costs that should be measured in CEA, but we have never measured such costs in any critical-care CEA. Therefore, if a new therapy is unlikely to cause either more

or less pain, then we can continue to ignore such intangible costs, even although they are considered relevant. The caveat here is that we have now made an important assumption—no difference in pain—which may or may not be true. We have of course glossed over the term relevant. Which costs are relevant? All costs to society could be considered relevant when conducting a CEA from the societal perspective. Using this perspective, one could argue that the costs of lost wages while a patient is sick are relevant. In response to this issue, the PCEHM recognized that there are no correct answers. However, to promote standardization of CEA methodologies, they recommend inclusion of all healthrelated costs, and the ATS concurs that this is the current best approach. This includes intangible costs of pain and suffering and travel costs. They also recommended including opportunity costs and suggested that lost wages, not only as a postdischarge consequence of the illness, but also during hospitalization, represent an example of an opportunity cost. Direct application of these guidelines to critical care is not easy. However, one way to consider them is to think about a healthcare system without the new therapy vs. a health care system with the new therapy. We then need to include all possible cost elements that could differ between these two healthcare worlds. How Should Costs Be Included? Not all costs included in a CEA are necessarily measured empirically. The CEA is a model often calibrated by estimates. Some of these estimates come from measurements. For example, the estimate of differences in the mortality rate between a drug and placebo often is derived from the effect size in an RCT. Other estimates can be based on expert opinion or on some combination of measurement and opinion. For example, the cost of the actual therapy is usually unknown because the therapy is often not yet approved, and no pricing decision has yet been made by the company that manufactures the therapy. One is therefore forced to estimate on the basis of an educated best guess, perhaps with knowledge of the preliminary pricing from the company. Although one might be alarmed at this notion of educated guesswork, it is important to appreciate that estimates can be wildly erroneous yet have only a minimal effect on the cost-effectiveness ratio. To test how sensitive a CEA ratio is to various estimates in the cost model, the comCrit Care Med 2003 Vol. 31, No. 1 (Suppl.)

pleted CEA model is exposed to a rigorous sensitivity analysis (see below). In this way, we can decide to include many costs in a CEA, yet measure specifically only some portion of that total. As long as these estimated costs have little effect on the overall final CEA conclusions, the strategy regarding which costs to measure and which to estimate can be considered robust. For How Long Do We Measure Costs? When the cost of a therapy is computed, the duration of the costs attributed to therapy must also be considered. For example, if our new therapy allows more people to leave the ICU but causes a higher prevalence of renal failure requiring long-term dialysis, should all the costs of dialysis be attributed to that therapy? The answer is yes. Although most intensivists do not accept the concept of blaming the therapy received in the ICU for incurring longterm cost, it is difficult to argue to the contrary. In producing a survivor, one must also take responsibility for the cost of maintaining survival, which means following these costs for a significant length of time. Furthermore, if chronic renal failure leads to a lower quality of life, the new therapy will be doubly penalized, both for the cost of the dialysis program and for the decrement in effect (reduced quality-adjusted survival). How to Measure Costs? For those costs that we choose to measure, we must decide what represents true cost. When we consider hospital costs, as an example, true costs are generally assumed to be those generated by formal cost-accounting mechanisms. For a complete blood count, the costs include the wage rate for and time spent by the employee who drew the blood, the cost of the tube, some tiny amortized fraction of the cost of the equipment upon which the test is run, and so on. Needless to say, detailed information such as this is rarely available as part of a CEA. Another approach is to collect hospital charges and adjust them by the hospital- or department-specific cost/charge ratios. The relationship between charges and costs has long been a source of skepticism for physicians. However, recent work by Shwartz et al. (32), comparing department-specific cost/charge ratio–adjusted charges with estimates generated from a formal costaccounting system, found good correlation when assessing patients in groups. Agreement was much poorer when comparing individual patients and when using hospital-specific ratios. CEAs rely on average Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)

grouped estimates of costs, and therefore department-specific cost/charge ratios seem adequate for estimating hospital costs. Other proxy measures of cost, such as the Therapeutic Intervention Scoring System or length of stay, can also be used (33, 34). As stated previously, their value will depend on how sensitive the conclusions are to variations in the relationship between these proxy measures and true costs.

Defining Standard Care (Comparators) When comparing a new therapy, the choice of comparator, or standard therapy, is also critical. For example, the costeffectiveness ratio of a 1-yr cervical cancer screening program is quite different when compared with 2- or 3-yr programs. Similarly, a tissue thromboplastin activator has a different cost-effectiveness ratio when compared with standard acute myocardial infarction therapy with no thrombolytic therapy as opposed to standard therapy with streptokinase. The PCEHM recommends that the control therapy used for comparative purposes be the least expensive available standard therapy. However, this view is currently changing in the field of critical care. For example, in the treatment of sepsis, should standard care include early goaldirected therapy, steroids, or drotrecogin alfa (activated), even although these may be expensive? If so, do we consider all treatments, or just one, to be standard therapy? The ATS guidelines recommend that standard practice is not always best practice and that best practice should be the comparator of choice in critical care.

Discounting (Time) Discounting costs due to time is another important factor to consider when conducting a CEA. When we borrow money, we must pay it back with interest. This is because money is worth more now than it will be later. Therefore, $10 is more valuable now than $10 delivered at a rate of $1 per year for the next 10 yrs. Thus, to pay back $10 that we just received over the next 10 yrs, we would be required to pay back ⬎$1 per year. Because world economic growth is occurring at approximately 3%, the PCEHM has recommended that all costs be discounted at a 3% rate per annum, and the ATS agrees with this recommendation.

But what about effects: should they also be discounted? Are ten people living for 1 yr more valuable than one person living for 10 yrs? Although this issue may seem inhumane, consideration of this point is crucial. Discounting costs without discounting effects will incur the Keeler-Cretin procrastination paradox, wherein we would forever favor healthcare programs that take place some time in the future (35). This situation would have us forever putting off until tomorrow that which could be done today. Therefore, we also discount effects at 3%, the same rate as costs.

Robustness and Uncertainty When we perform a RCT, our primary conclusion is a statement of effect: did the new therapy change the outcome of interest? Although it is highly likely that the outcome rates will be different (rarely would the mortality rates in both arms be identical), we rely on statistical significance to tell us whether the observed difference is due to a true effect of the therapy and not to chance alone. We traditionally infer statistical significance when the p value is ⬍.05. In other words, we are 95% certain that the observed difference did not arise by chance alone. If we are interested only in the single dimension effect, then we care only about which therapy arm is better, not how much better. It is important to appreciate, however, that the p value does not confirm the magnitude of effect. Consider an example in which a new antisepsis strategy therapy has a mortality rate of 35%, as opposed to a placebo rate of 40%, with a p ⬍ .05. This does not mean that 5 lives are saved per 100 persons treated. Rather, it tells us that our best estimate is that five lives are saved. If we presume a binomial distribution around the mortality rates, we can generate confidence intervals around the two estimates. These confidence intervals might now tell us that new therapy saves from 1 to 9 lives per 100 persons treated, but they cannot tell us where the true value falls within that range. The p value simply confirms the likelihood that lives are saved by the new therapy, not how many lives. In CEA, however, we must quantitate the magnitude of effect (and cost) so that we can generate a ratio. The general principle is to first take our best point estimates of cost and effect to generate a base case. Thereafter, we vary all our measures S13

and estimates across their range of probabilities (e.g., 95% confidence interval) to determine the extent to which the costeffectiveness ratio varies. This is a sensitivity analysis (SA) and can be performed either with one or multiple variables simultaneously. In one respect, the sensitivity analysis can be considered analogous to the p value in that it allows us to explore the robustness of our conclusions. In other words, if, despite varying several or all variables across their stochastic distributions, there is minimal change in the final ratio, then one can have considerable confidence in the costeffectiveness ratio estimate. Another aspect of the sensitivity analysis is that it can be used to determine which model estimates must be the most accurate. For example, the cost-effectiveness ratio may be exquisitely sensitive to the estimate of ICU costs but relatively insensitive to the expected costs of postdischarge healthcare resource use. In this situation, one might need to measure ICU costs very carefully yet rely only on approximate estimates of postdischarge resource use. A comprehensive sensitivity analysis can, in fact, be considered more powerful than a p value in that it can be used to graphically show all of the uncertainties inherent in the underlying assumptions of the CEA model.

Reporting and the PCEHM Reference Case The PCEHM also recommended that all future CEAs produce a reference case in which the cost-effectiveness ratio is generated by a standardized approach to estimating and measuring each of the important elements of the CEA (Table 2). This includes the perspective chosen, determination of costs and effects, study time horizon, and measurement of uncertainty and sensitivity analyses. Use of this standardized approach allows for comparison of CEA results across studies. Comparison of reference cases between CEAs allows us to make inferences about the cost-effectiveness of a new sepsis therapy vs. a therapy used to fight breast cancer. It allows us to compare apples with oranges and not just apples with apples. Figure 3 shows comparisons of costs for a variety of treatments in different diseases. These include both interventions against specific disease states (e.g., myocardial infarction and stroke) (36, 37) and interventions designed to prevent injury or illness (e.g., airbags) S14

Figure 3. League table showing the range of cost effectiveness ratios for a variety of medical or preventive interventions (36 – 43). The vertical lines show the 20,000/quality-adjusted life years (QALY) and 100,000/QALY levels. Shading of the horizontal lines shows level of cost effectiveness: good (light gray), marginal (dark gray), and poor (black). The range of values within an intervention indicates differences in conditions or assumptions included in the model. tPA, tissue plasminogen activator; AMI, acute myocardial infarction; CABG, coronary artery bypass graft; 2V, two vessel.

(38). The ATS guidelines also recommend the generation of a reference case and further recommend the presentation of a data-rich case from the results of the RCT. This case would be generated by using a minimal number of model assumptions and the maximum amount of available data from the trial.

CEA AND HEALTHCARE POLICY Decision making based on the results of a CEA is founded on the idea of social utilitarianism. This value is in turn based on the assumptions that a) good is determined by consequences at the community level, these consequences being the sum of individual utilities (health and happiness); b) all utilities are equal within the metric used to measure them; and c) loss of benefit to some is balanced by benefit to others. As a simple example, consider the decision to fund a childhood immunization program rather than a radical chemotherapy program to treat a rare cancer. This decision assumes that spending resources on immunizations will maximize the community’s utility more than money spent on treating a rare cancer. Social utilitarianism acts to maximize the health and happiness (utility) of the community; consequently, utility leads to the maximum efficiency in the use of healthcare resources for the community’s benefit. CEA is designed to re-

sult in a prioritized list of community benefits for given cost outlays. However, although CEA can inform us about where to spend money to improve utility, it cannot inform about how much money should be spent improving health care overall. The overall goal of CEAs is to supply decision makers with information that can be used to choose between medical care options when all options are not financially feasible. If monies are unlimited, the relevant question becomes “what treatment options minimize patient morbidity and mortality?” and CEAs are unnecessary. If funds are limited, the question becomes “what is the best value?” and CEAs can help us to answer this question. This was the overall conclusion of the ATS. The workshop further recommended that critical care researchers use the PCEHM guidelines and conduct reference case CEAs as part of the evaluation of ICU interventions; these CEAs will allow more informative health care policy in the care of the critically ill.

CONCLUSIONS The conduct of a rigorous CEA is clearly challenging. There are many methodologic complexities to consider within the study, and the analysis is likely to be highly dependent on the availability and quality of information from other Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)

C

onsider cost-effec-

9.

tiveness analysis as simply another

tool in our evaluation toolset

10.

of new drug therapies.

studies (e.g., a phase III RCT). Much of the information required to conduct a thorough CEA may in fact be missing, and therefore the analysis will require sophisticated modeling techniques to explore the effects of various assumptions and uncertainties. For the reader to be confident in the results of a CEA, the analysis not only must be robust, but must also be perceived as such. Accomplishing this requires clear reporting on the part of the analyst and commitment on the part of the reader to embrace rigorously conducted cost-effectiveness analyses as a legitimate approach to determining the value of new therapies in the treatment of critical illness.

11.

12.

13.

14.

REFERENCES 15. 1. Halpern NA, Bettes L, Greenstein R: Federal and nationwide intensive care units and healthcare costs: 1986 –1992. Crit Care Med 1994; 22:2001–2007 2. Angus DC, Linde-Zwirble WT, Sirio CA, et al: The effect of managed care on ICU length of stay: Implications for Medicare. JAMA 1996; 276:1075–1082 3. Ziegler EJ, Fisher CJ Jr, Sprung CL, et al: Treatment of gram-negative bacteremia and septic shock with HA-1A human monoclonal antibody against endotoxin: A randomized, double-blind, placebo-controlled trial. N Engl J Med 1991; 324:429 – 436 4. Warren HS, Danner RL, Munford RS: Antiendotoxin monoclonal antibodies. N Engl J Med 1992; 326:1153–1157 5. Wenzel RP: Anti-endotoxin monoclonal antibodies: A second look. N Engl J Med 1992; 326:1151–1153 6. Winslow R: Centocor drug is said to need more testing. Wall Street Journal April 21, 1992; A3 7. Winslow R: Dose of reality: Costly biotech drugs will pose hard choices for US healthcare. Wall Street Journal November 5, 1991; A1 8. Abraham E, Wunderink RG, Silverman H, et al: Efficacy and safety of monoclonal antibody to human tumor necrosis factor alpha in patients with sepsis syndrome: A random-

Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)

16.

17.

18.

19.

20.

21.

ized, controlled, double-blind, multicenter clinical trial. JAMA 1995; 273:934 –941 Bone RC, Balk RA, Fein AM, et al: A second large controlled clinical study of E5, a monoclonal antibody to endotoxin: Results of a prospective, multicenter, randomized, controlled trial. Crit Care Med 1995; 23: 994 –1006 Dhainaut JF, Tenaillon A, Le Tulzo Y, et al: Platelet-activating factor receptor antagonist BN 52021 in the treatment of severe sepsis: A randomized, double-blind, placebo-controlled, multicenter clinical trial. Crit Care Med 1994; 22:1720 –1728 Fein AM, Bernard GR, Criner GJ, et al: Treatment of severe systemic inflammatory response syndrome and sepsis with a novel bradykinin antagonist, deltibant (CP-0127): Results of a randomized, double-blind, placebo-controlled trial. CP-0127 SIRS and Sepsis Study Group. JAMA 1997; 277:482– 487 Fisher CJ Jr, Dhainaut JF, Opal SM, et al: Recombinant human interleukin 1 receptor antagonist in the treatment of patients with sepsis syndrome: Results from a randomized, double-blind, placebo-controlled trial. JAMA 1994; 271:1836 –1843 Russell LB, Gold MR, Siegel JE, et al: The role of cost-effectiveness analysis in health and medicine: Panel on Cost-Effectiveness in Health and Medicine. JAMA 1996; 276: 1172–1177 Weinstein MC, Siegel JE, Gold MR, et al: Recommendations of the Panel on CostEffectiveness in Health and Medicine. JAMA 1996; 276:1253–1258 Siegel JE, Weinstein MC, Russell LB, et al: Recommendations for reporting cost-effectiveness analyses. JAMA 1996; 276: 1339 –1341 Angus DC, Rubenfeld GD, Roberts MS, et al: Understanding costs and cost-effectiveness in critical care: Report from the Second American Thoracic Society Workshop on Outcomes Research. Am J Respir Crit Care Med 2002; 165:540 –550 Gold MR, Russell LB, Seigel JE, et al: Costeffectiveness in Health and Medicine. New York, Oxford University Press, 1996 Doubilet P, Weinstein MC, McNeil BJ: Use and misuse of the term “cost effective” in medicine. N Engl J Med 1986; 314:253–256 Linden PK, Angus DC, Chelluri L, et al: The influence of clinical study design on costeffectiveness projections for the treatment of gram-negative sepsis with human antiendotoxin antibody. J Crit Care 1995; 10: 154 –164 Freemantle N, Drummond M: Should clinical trials with concurrent economic analyses be blinded? JAMA 1997; 277:63– 64 Dellinger RP: From the bench to the bedside: The future of sepsis research. Executive summary of an American College of Chest Physicians, National Institute of Allergy and Infectious Disease, and National Heart, Lung, and Blood Institute Workshop. Chest 1997; 111:744 –753

22. Clermont G, Angus DC, Linde-Zwirble WT, et al: Does acute organ dysfunction predict patient-centered outcomes? Chest 2002; 121: 1963–1971 23. Cohen J, Guyatt G, Bernard GR, et al: New strategies for clinical trials in patients with sepsis and septic shock. Crit Care Med 2001; 29:880 – 886 24. Bernard GR, Vincent JL, Laterre PF, et al: Efficacy and safety of recombinant human activated protein C for severe sepsis. N Engl J Med 2001; 344:699 –709 25. Annane D, Sebille V, Charpentier C, et al: Effect of treatment with low doses of hydrocortisone and fludrocortisone on mortality in patients with septic shock. JAMA 2002; 288: 862– 871 26. Rivers E, Nguyen B, Havstad S, et al: Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med 2001; 345:1368 –1377 27. Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome: The Acute Respiratory Distress Syndrome Network. N Engl J Med 2000; 342: 1301–1308 28. Quartin AA, Schein RM, Kett DH, et al: Magnitude and duration of the effect of sepsis on survival. JAMA 1997; 277:1058 –1063 29. Angus DC, Carlet J: Surviving intensive care: A report from the 2002 Brussels Roundtable. Intensive Care Med, In Press 30. Angus DC, Carlet J (Eds): Surviving Intensive Care: Update in Intensive Care and Emergency Medicine No. 39. Berlin, SpringerVerlag, 2002 31. Chelluri L, Grenvik AN, Silverman M: Intensive care for critically ill elderly: Mortality, costs, and quality of life. Review of the literature. Arch Intern Med 1995; 155:1013–1022 32. Shwartz M, Young DW, Siegrist R: The ratio of costs to charges: How good a basis for estimating costs? Inquiry 1995; 32:476 – 481 33. Keene AR, Cullen DJ: Therapeutic Intervention Scoring System: Update 1983. Crit Care Med 1983; 11:1–3 34. Rapoport J, Teres D, Lemeshow S, et al: A method for assessing the clinical performance and cost-effectiveness of intensive care units: A multicenter inception cohort study. Crit Care Med 1994; 22:1385–1391 35. Keeler EB, Cretin S: Discounting of lifesaving and other nonmonetary effects. Manage Science 1983; 29:300 –306 36. Kalish SC, Gurwitz J, Krumholz HM, et al: A cost-effectiveness model of thrombolytic therapy for acute myocardial infarction. J Gen Intern Med 1995; 10:321–330 37. Gage BF, Cardinalli AB, Albers GW, et al: Cost-effectiveness of warfarin and aspirin for prophylaxis of stroke in patients with nonvalvular atrial fibrillation. JAMA 1995; 274: 1839 –1845 38. Graham JD, Thompson KM, Goldie SJ, et al: The cost-effectiveness of air bags by seating position. JAMA 1997; 278:1418 –1425 39. Stinnett AA, Mittleman MA, Weinstein MC:

S15

The cost-effectiveness of dietary and pharmacologic therapy for cholesterol reduction in adults. In: Cost-effectiveness in Health and Medicine. Gold MR, Siegel JE, Russell LB, et al (Eds). New York, Oxford University Press, 1996, pp 349 –391 40. Hunink MG, Wong JB, Donaldson MC, et al: Revascularization for femoropopliteal disease: A decision and cost-effectiveness analysis. JAMA 1995; 274:165–171 41. Pliskin JS, Stason WB, Weinstein MC, et al: Coronary artery bypass graft surgery: Clinical decision making and cost-effectiveness analysis. Med Decis Making 1981; 1:10 –28

S16

42. Weinstein MC, Stason WB: Cost-effectiveness of coronary artery bypass surgery. Circulation 1982; 66:III56 –III66 43. Owens DK, Sanders GD, Harris RA, et al: Cost-effectiveness of implantable cardioverter defibrillators relative to amiodarone for prevention of sudden cardiac death. Ann Intern Med 1997; 126:1–12 44. Singh N, Rogers P, Atwood CW, et al: Shortcourse empiric antibiotic therapy for patients with pulmonary infiltrates in the intensive care unit: A proposed solution for indiscriminate antibiotic prescription. Am J Respir Crit Care Med 2000; 162:505–511 45. Bootman JL, Wertheimer AI, Zaske D, et al:

Individualizing gentamicin dosage regimens in burn patients with gram-negative septicemia: A cost-benefit analysis. J Pharm Sci 1979; 68:267–272 46. Mark DB, Hlatky MA, Califf RM, et al: Cost effectiveness of thrombolytic therapy with tissue plasminogen activator as compared with streptokinase for acute myocardial infarction. N Engl J Med 1995; 332: 1418 –1424 47. Stal JM, Gregor JC, Preiksaitis HG, et al: A cost-utility analysis comparing omeprazole with ranitidine in the maintenance therapy of peptic esophageal stricture. Can J Gastroenterol 1998; 12:43– 49

Crit Care Med 2003 Vol. 31, No. 1 (Suppl.)