Current issues of ACP Journal Club are published in Annals of Internal Medicine


Editorial

Economic evaluation of health care interventions: an economist's perspective

ACP J Club. 1996 Mar-April;124:A12. doi:10.7326/ACPJC-1996-124-2-A12



Twenty years ago, many persons still believed that the concept of a scarcity of resources did not apply to health care. Today, the concept of scarce health care resources is widely accepted by most clinicians. This change in attitude is reflected in the expanding medical literature on the economic evaluation of health care programs. Unfortunately, studies that have appraised the methods in these evaluations have reported low rates of compliance with recommended guidelines for economic evaluations (1-4). Doubts have been raised about the comparability and the generalizability of the results of these studies (5), leading to a call for further standardization of methods (6, 7). Therefore, it is not surprising that David Naylor, in his editorial in the January/February issue of ACP Journal Club (8), recommended that readers “… should maintain a healthy skepticism about the results of cost-effectiveness analysis and the usefulness of those results in purchasing and planning decisions.”

The purpose of this editorial is to address whether flaws in the economic methods themselves or the oversimplification and misapplication of these methods are responsible for the perceived (or real) lack of usefulness of this growing body of literature. The discipline of economics deals with maximizing some stated objective, for example, the production of health-related well-being among a given population through the choice of which health care procedures to provide to whom, given the resources available. When the discipline of economics is being chosen as the “mode of thinking” for resource allocation in health, the principles of the discipline must be followed.

The economics mode of thinking is based on the following fundamental concepts. First, resources are scarce. Second, when resources are scarce, choices must be made about how the available resources should be shared. But by choosing to use resources in a particular way, other opportunities to use those resources are abandoned. Thus, economics attempts to ensure that the benefits of a particular choice exceed the benefits of any alternative (i.e., the concept of opportunity cost). When the value of what we choose to do with available resources is at least as great as the opportunity cost, then the resources are being used in ways that maximize the benefits from the use of those resources. Only under this condition does the use of resources represent a position of economic efficiency.

Proponents of current guidelines for economic evaluations have accepted that maximization of benefits from available resources is the basis for economic evaluation of health care interventions. But many current evaluations and the standards for judging these evaluations do not embody sound economic principles for maximization of benefits given the available resources. This is a problem. For example, Mark and colleagues (9) concluded that “the cost effectiveness of treatment which accelerated t-PA [tissue plasminogen activator] rather than streptokinase compares favorably with that of other therapies whose added medical benefit for dollars spent is judged by society to be worthwhile.” In this case, the authors follow the common practice of determining whether a new program, which requires additional resources but produces additional benefits, should be implemented. This practice is done either in relation to some specific threshold level of the cost-effectiveness (CE) ratio (e.g., anything lower than $35 000 per QALY [quality-adjusted life-year] produced should be implemented) or on the basis of the program's rank in some “league table” of CE ratios of other programs or on the basis of the researchers' own judgment of the acceptability of the ratio.

This analysis fails from an economic perspective because it considers only the net increases in costs and in outcome, omitting the benefits forgone by the need to use additional resources for the proposed program. Mark and colleagues acknowledge that implementing the program “would cost the nation approximately $500 million each year,” but they do not identify where the money will come from and what the opportunity costs will be (i.e., benefits forgone) from transferring the money from existing programs to support the new programs of t-PA use. Therefore, this analysis does not address the maximization of benefits from fixed resources (i.e., economic efficiency). Nevertheless, Mark and colleagues conclude that their analysis “can inform the decision about whether this should become the standard of care in the United States.”

An attempt to support the use of CE ratios as a decision tool in a way that relates such use to economics is reported elsewhere (10). As is shown (11), the use of CE ratios in this way results in assuming a world that bears little if any relevance to the real world of allocating scarce health care resources for which economic rules are intended. For example, it is assumed that all health care interventions can be purchased in small incremental units (e.g., 1 h of a dialysis machine) and that the rate of output produced by all programs is constant, no matter how many increments are produced (e.g., if 1 dialysis machine would produce 50 additional QALYs per year, then one tenth of a dialysis machine would produce 5 additional QALYs). But these conditions are unlikely to be met in the real world, as even proponents of these decision rules acknowledge (5, 10).

It is interesting to note that different researchers have implicitly or explicitly chosen to pursue different courses of action. Some do not accept the discipline of economics as the basis for an economic evaluation of health care interventions. Economics is not the only discipline that deals with issues of resource allocation. For those who want to use other disciplines as the basis for such analyses, it seems fair to ask that they clearly identify the disciplinary base for their analyses and that they do not label them as economic evaluations. Otherwise, readers will be misled to believe that these analyses are consistent with the principles of the discipline of economics.

Among those who support the use of economic principles as the basis for such analyses, 2 schools can be identified: 1) those who believe that current economic methods are too complex for practical implementation and, hence, that strong assumptions are warranted to simplify the task and 2) those who believe that assumptions that oversimplify the real world threaten the validity of the methods and who thus are willing to accept a more complex execution.

As H.L. Mencken has said, “To every complex question there is a simple answer … and it is wrong.” A model is only as good as its assumptions, and the use of unrealistic assumptions (i.e., assumptions that are known to be empirically invalid) or strong assumptions (i.e., assumptions that have no normative appeal) will not help us solve real-life problems. This is not the economic way of dealing with real-life problems. Economics does provide us with valid (but not necessarily simple) tools to deal with questions of resource allocation. These methods may be data hungry or complex to execute, but they are there to be used by researchers and decision makers and have been successfully used (12).

As an alternative, faced with a real-world problem and a problem of feasibility of implementation of complex methods, one can decide to go the economics way of simplifying the task by choosing a second-best solution, for example, modifying the objective from one of optimization to one of unambiguous improvement (13, 14). The following is an example of a second-best solution applied to the case of t-PA versus streptokinase. The first step is to use a proper measure of outcome. The use of life-years saved is problematic because it does not capture the total effect of this treatment (e.g., the increased risk for stroke) and does not allow us to compare this intervention with other interventions that do not reduce mortality rates but do affect morbidity. The use of QALYs as an outcome measure provides a common unit of measurement, but there is no agreement about how to measure the weights for quality adjustment. This lack of agreement gives rise to the complaint, “a QALY is not a QALY is not a QALY” (15).

Instead of using an ambiguous measure (i.e., different methods resulting in different values) that is also incompatible with the requirements of the underlying economic theory (15), we need to find an unambiguous measure that is compatible with the underlying theory. An example is “willingness to pay” (WTP). This measure is not problem free (there is no problem-free measure), but if properly used (16, 17), it can provide us with a theoretically sound and unambiguous way of measuring the outcome. An example of an advantage of the WTP over the QALY is that WTPs allows you to capture the effect of the changes in the health of individual persons on others (for example, the effect of having a severe stroke on other members of the family).

Once we have an acceptable measure of outcome, we must consider how to determine whether a new intervention should be adopted. For a given budget, a necessary condition for implementation of a new intervention that results in improved outcome but also higher cost would be that at least one existing intervention (or a combination of interventions) will, if eliminated, generate resources sufficient to provide for the new program and reduce the community health-related well-being by less than the incremental gain in the community's health-related well-being that was caused by the implementation of the new intervention. This reallocation of health care resources can be seen as a “step in the right direction” because it results in an overall increase in the community's health-related well-being without using additional resources. It is only a step in the right direction, however, because more than 1 intervention (or set of interventions) might satisfy these conditions. But, as already said, a lack of data about all of these interventions is likely to prevent us from finding the optimal reallocation of resources in real life.

In the context of t-PA (assuming that it represents an overall improvement in the community's health-related well-being for increased costs), it might be practical to start with the relevant overall budget allocated to treat patients with myocardial infarction and see if at least 1 other intervention (or set of interventions) can be found that, if eliminated, satisfies both the conditions stated above. If we cannot find these interventions within this budget, then we can go to other budgets to try and identify the interventions. This policy of “first cleaning your own budget” should help in preventing departments from looking first for inefficiency in other areas without making sure that their own operation meets similar standards.

Finally, if all published studies that report the results of an economic evaluation were to follow the principles of the discipline of economics, would they provide relevant information for all local decision makers? This question has nothing to do with the validity of the methods offered by the discipline of economics. It deals with the issue of generalizability of the results of a study done in 1 place to decision makers elsewhere. Economics does not argue that empiric results from 1 study should be generalizable to all other settings. Further, attempts to force generalizability (e.g., mandating a unified social time preference for the world [18]) are not consistent with the principles of economics and are likely to result in invalid analyses.

In summary, economics provides useful methods to help sort out resource allocation to optimize benefits. When looking at economic evaluations, readers should pay attention to evaluations that follow sound economic principles.

Amiram Gafni, PhD
McMaster University
Hamilton, Ontario, Canada


References

1. Drummond MF. Studies in Economic Appraisal in Health Care. Oxford: Oxford University Press; 1980.

2. Drummond MF, Ludbrook A, Lowson K, Steel A. Studies in Economic Appraisal in Health Care. Vol 2. Oxford: Oxford University Press; 1986.

3. Warner KE, Hutton RC. Cost-benefit and cost-effectiveness analysis in health care: growth and composition of the literature. Med Care. 1980;18:1069-84.

4. Udvarhelyi S, Colditz GA, Rai A, Eptsein AM. Cost-effectiveness and cost-benefit analyses in the medical literature: are the methods being used correctly? Ann Intern Med. 1992;116:238-44.

5. Drummond MF, Torrance GW, Mason J. Cost-effectiveness league tables: more harm than good? Soc Sci Med. 1993;37:33-40.

6. Drummond MF, Brandt A, Luce BR, Rovira J. Standardizing economic evaluation methodologies in health care. Int J Technol Assess Health Care. 1993;9:26-36.

7. Luce BR, Simpson K. Methods of cost-effectiveness analysis: areas of consensus and debate. Clin Ther. 1995;17:109-25.

8. Naylor CD.Cost-effectiveness analysis: Are the outputs worth the inputs? ACP J Club. 1996 Jan-Feb;124:A12-4.

9. Mark DB, Hlatky MA, Califf RM, et al. Cost-effectiveness of thrombolytic therapy with tissue plasminogen activator as compared with streptokinase for acute myocardial infarction. N Engl J Med. 1995;332: 1418-24.

10. Johannesson M, Weinstein M. On the decision rules of cost-effectiveness analysis. J Health Econ. 1993;12:459-68.

11. Birch S, Gafni A. Changing the problems to fit the solution: Johannesson and Weinstein's (mis)application of economics to real world problems. J Health Econ. 1993;12:469-76.

12. Birch S, Gafni A. Cost-effectiveness/utility analysis: do current decision rules lead us to where we want to be? J Health Econ. 1992;11:279-96.

13. Gafni A, Birch S. Guidelines for the adoption of new technologies: a prescription for uncontrolled growth in expenditures and how to avoid the problem. Can Med Assoc J. 1993;148:913-7.

14. Birch S, Leake J, Lavis J. Economic issues in the development and use of practice guidelines: an application to resource allocation in dentistry. Community Dent Health. 1996;13:70-5.

15. Gafni A, Birch S. Preferences for outcomes in economic evaluation: an economic approach to addressing economic problems. Soc Sci Med. 1995;40:767-76.

16. Gafni A. Using willingness-to-pay as a measure of benefits. Med Care. 1991;29: 1246-52.

17. O'Brien B, Gafni A. When do the “dollars” make sense? Toward a conceptual framework for contingent valuation studies in health care. Med Decision Making. 1996;16:288-99.

18. Krahn M, Gafni A. Discounting in the economic evaluation of health care interventions. Med Care. 1993;31:403-18.