8 Cost-effectiveness analysis

Abstract

Cost-effectiveness analysis is a quantitative method of comparing the “return on investment” of a given policy (the desired results it produces, in relation to its cost), with other possible policies. This method makes it possible to estimate the effectiveness of a policy, i.e. its capacity to maximise a result criterion for each euro of public expenditure. It is useful for guiding public policy choices and the allocation of public expenditure within a given sector.

Keywords: Quantitative methods, cost-effectiveness, efficiency, effectiveness

I. What does this method consist of?

Cost-effectiveness analysis is a method of exploring the efficiency of a public policy, i.e. in colloquial terms determining its ‘return on investment’. It is a comparative method in which the intervention being evaluated is compared to several other options: existing policies, alternatives, etc. This comparison makes it possible to prioritise the different options and to determine which one allows for the optimisation of public expenditure, i.e. to obtain the best possible result for each euro invested. The prioritisation of the different options is based on a simple economic calculation, that of the “incremental cost effectiveness ratio”. This calculation relates the difference in costs to the difference in effectiveness between the intervention and its comparators.

Five steps are necessary to implement this evaluation method.

First, it is necessary to choose a perspective for the analysis and a target population. This consists of determining which perspective is adopted for the calculation: that of the public policy financer (payer)? of its beneficiaries? of society as a whole? It should be noted that the perspective often chosen in cost-effectiveness analyses is the societal perspective, because it takes into account the impact of the policy for all stakeholders (payers, beneficiaries, etc.). However, while this collective perspective is more interesting for the public decision-maker, it is also more difficult to implement because it implies a comprehensive measurement of costs and effectiveness criteria. The choice of the target population is often dictated by the objective of the public policy. For example, a public breast cancer prevention campaign targeting specific ages. It is often relevant to identify specific subgroups of people within this population based on, for example, their access to the intervention (e.g., access to health centres) or their exposure to other measures (e.g., access to privately funded preventive care).

Second, the scope of the costs associated with the public policy being evaluated must be determined. The costs considered in the evaluation are those linked to the deployment of the intervention, also known as “direct costs”: investment in infrastructure, equipment, salaries of staff dedicated to the intervention, information campaigns, etc. In the context of an evaluation covering several years, these costs are discounted to take account of inflation. Costs indirectly associated with the public policy can also be included in the evaluation. Indeed, if the policy has a strong impact on the domestic sphere, it is often appropriate to include this impact in the calculation of the efficiency ratio. For example, it can be expected that an increase in the generosity of public support for the autonomy of frail seniors will reduce absenteeism from work by family carers, who can substitute professional services for their care time.

The third step is to choose a criterion for the effectiveness of public policy. The choice of this criterion is decisive, as it must be “sensitive” enough to capture the impact of the policy. Two main categories of criteria are generally used in evaluations: outcome criteria and utility criteria. On the one hand, outcome criteria make it possible to measure the effectiveness of the policy being evaluated with numerical indicators: for example, unemployment durations for the evaluation of employment policies, mortality rates for health policies, school success rates for education policies, etc. On the other hand, the utility criteria make it possible to measure the impact of public policy on the well-being of households, measured using questionnaires. This is known as a cost-utility analysis, the objective of which is to measure whether the policy being evaluated optimises the average level of well-being of its beneficiaries.

Fourthly, it is necessary to determine the data sources to be mobilised for the evaluation and to estimate the impact of the public policy on the basis of these data. Evaluations use two main categories of databases: those from “randomised experiments” and so-called “real-life” data. Randomised experiments (see separate brief) involve comparing two different populations, one receiving the intervention being evaluated and the other receiving an alternative. The success of these experiments depends on the absence of “contamination” between the two arms being compared. Indeed, any interaction between the members of the two groups calls into question the evaluation of the effectiveness of the treatment, and therefore the entire cost-effectiveness evaluation. Typically, these studies cover a two-year period and include a few thousand participants. Evaluations based on real-life data consist of identifying ex-post in administrative databases the two populations included in the evaluation (always according to their exposure), and following their evolution over a longer period. They have the advantage of being exhaustive: sample size, number of years of follow-up, etc.

Finally, the cost-effectiveness analysis itself can be carried out. This analysis involves two steps: firstly, the average effects observed on the cost and effectiveness parameters must be estimated from the data, and secondly, the cost-effectiveness impact of the public policy must be modelled on the basis of the estimated effects. The estimation of the effects is carried out using econometric regressions, which make it possible to identify the average impact of the public policy in the sample of data used, and confidence intervals for this effect (upper and lower bounds which frame its average value). This estimation step is essential because it allows us to define whether the impact of the policy is significant or not, in other words, whether the policy is producing the expected results on average. The choice of estimation method is an essential aspect of this stage. This is followed by the modelling phase, in which the effect estimates obtained in the first step are used as inputs to the cost-effectiveness model. For example, different probabilities of emergency hospitalisations will be estimated as part of the evaluation of a falls prevention policy for the elderly. All possible impacts of the policy will be tested to take account of the uncertainty associated with its effects, i.e. the different possible values for the probabilities of hospitalisation. This is called “micro-simulation”.

II. How is this method useful for the evaluation of public policies?

Cost-effectiveness evaluation sheds light on the efficiency of a public policy, i.e. its capacity to maximise a result criterion for each euro of public expenditure. It is therefore a decision-making tool that makes it possible to answer many evaluative questions: Is it more cost-effective to implement policy A than policy B? What are the incremental costs and benefits of adopting a public policy? Which specific populations can benefit most from the deployment of this public policy? How can the allocation of public expenditure in a given sector (health, education, security etc.) be improved? Do the expected effects of a public policy exceed the costs of its implementation? Can the efficiency of a policy vary according to the profile of its beneficiaries, the population covered?

The answer to these questions can be given ex ante, in the context of an evaluation exploring whether it is wise to generalise the deployment of a programme implemented in a given geographical area, or ex post, in the context of an evaluation that determines whether a public policy has had the expected economic effects on a given population over a defined period of time.

The main advantage of this method is that it allows all possible economic effects of a public policy on a given population to be considered in a comprehensive manner. Its use therefore improves the transparency of public policy decision-making criteria. By using the results of a cost-effectiveness evaluation, the public decision-maker can arbitrate not only on the basis of economic criteria (costs), but also on the basis of the effectiveness of the results of public policy. In other words, the use of cost-effectiveness evaluation encourages decision-making that is not solely guided by considerations of public expenditure control, particularly when it uses efficiency criteria measured in terms of individual well-being. This method is also identified by France Stratégie as a central tool for comparing the efficiency of different public policies (Desplatz and Ferracci 2016).

III. Examples of the use of this method in the evaluation of health policies

Recently, numerous scientific studies have been carried out to evaluate the efficiency of policies to combat the COVID-19 pandemic. A systematic review of this literature identifies that the main control measures against COVID19 (testing, wearing of masks, social distancing, quarantines) were mostly efficient, i.e. their return on investment was high, and all the more so when the virus reproduction factor was high and they were introduced in combination (Vandepitte et al. 2021). This study nevertheless warns of the existence of country-specific factors (population density and structure, organisation of the health system, etc.) that explain the greater efficiency of these measures in different countries. It shows that the results of an efficiency analysis of a policy carried out in a particular country/context cannot be easily transposed to another country.

Cost-effectiveness assessment can also be used prospectively to guide public decision-making. For example, for more than ten years, the French National Authority for Health (HAS) has been using this evaluation method to determine the efficiency of health innovations and to inform negotiations on pricing and reimbursement of these innovations between the health industry and the Economic Committee for Health Products (CEPS). They are published in a transparent manner on the HAS website. These analyses are one of the main tools used by the CEPS to assess the expected impact of a decision to set the price of a medicine. Manufacturers are responsible for producing evaluation models for their innovations, following a methodological guide designed and updated by the HAS’s Economic and Public Health Evaluation Commission (CEESP). This guide details the criteria used to assess the quality of the analysis (HAS 2020). Once the analysis has been carried out, manufacturers submit an “efficiency report” describing the content of the model and its results. Efficiency opinions” issued by the EPHSC conclude on the impact of the efficiency of the introduction of a new treatment on the French market, or evaluate ex post the efficiency of a treatment after several months of use.

IV. What are the criteria for judging the quality of the mobilisation of this method?

The quality of this method depends on the quality of the data used to build the model, and the quality of the method of identifying the causal relationship to measure the effects of the policy. These two points are essential. Indeed, it is essential that the step of estimating the effects of the policy mobilises advanced identification methods (randomised trial, propensity scores, instrumental variables) which are often complex to implement. When these data are not available, it is necessary to mobilise data from the scientific literature to find comparable evaluations in other countries, and to use the data from these evaluations to “feed” the model. If such data are not available in the literature, qualitative interviews must be used, which may reduce the accuracy of the assumptions and the overall quality of the model.

V. What are the strengths and limitations of this method compared to others?

The main advantage of cost-effectiveness analysis is that it is a transparent and easily accessible policy-making tool. Indeed, the comparison of the efficiency of different programmes is carried out with the help of a graphical representation which allows for the easy identification of the most efficient measures, i.e. those with the most favourable cost-effectiveness ratio, as they are less expensive than a comparator for a higher efficiency. Moreover, these analysis methods are robust: they have been used for several decades in all areas of the economy (health, development, education, labour, etc.).

Nevertheless, this method has two main limitations. The first limitation is that this method does not really guide public decision making when the result of the evaluation shows that the policy is more effective but also costlier than another measure. This is often the case when evaluating the impact of a policy aimed at encouraging the deployment of an innovation on a market, which is often more expensive but also more effective than a comparator. The second limitation of this method is the sometimes high cost of implementing it. The implementation of a randomised trial requires a significant financial investment (a “small” experiment can cost several hundred thousand euros). Moreover, estimating the effect of the policy implies a long time follow-up, which is often disconnected from the political time. For this reason, the feasibility of efficiency evaluations often depends on the capacity of the evaluation body to conduct an experiment. In the past, this may have blocked the implementation of public measures. For example, the deployment of telemedicine in France, although desired by the public authorities, has long been partly blocked by the inability of market players and/or the health insurance system to conduct experiments that would allow conclusions to be drawn about the efficiency of these devices.

Some bibliographical references to go further

Desplatz, Rozenn. and Ferracci, Marc. 2016. “How to assess the impact of public policies? A guide for policy makers and practitioners.” Paris, France: France Stratégies.

HAS. 2020. Choix méthodologiques pour l’évaluation économique. Saint-Denis.

Vandepitte, Sophie. and Alleman, Tijs. and Nopens, Ingmar. and Baetens, Jan. and Coenen, Samuel. and De Smedt, Delphine. 2021. “Cost-Effectiveness of COVID-19 Policy Measures: A Systematic Review.” Value in Health 24(11): 1551‑69. https://doi.org/10.1016/j.jval.2021.05.013.