Evaluation of global health interventions. Advanced methods.
Edited by Valéry Ridde and Christian Dagenais

Evaluability study. An intervention to prevent drug use in schools in Quebec
Biessé Diakaridja Soura, Jean-Sébastien Fallu, Robert Bastien et Frédéric N. Brière

Conducting an evaluability Assessment (EA) is an important step in planning the actual evaluation of an intervention or program. It is an exercise that allows to quickly collect data on the implementation of a program in order to make decisions about possible changes or conduct a formal evaluation. In this actual case, EA was applied to a school-based substance abuse prevention program. The different steps to conduct an EA are described as well as the results and the lessons learned. In addition, the strengths and weaknesses of EA, as a pre-evaluative phase of a program, and the challenges of its implementation are presented.

The evaluation of sustainability. A results-based financing intervention in Mali
Mathieu Seppey et Valéry Ridde

Results-Based Financing (RBF) is a new type of health financing reform that is very popular in Africa. Several pilot projects are being implemented on this continent, and elsewhere, to provide better and more health care services. However, too little research has been done on the sustainability of this type of intervention in Africa. This chapter explains how this intervention was evaluated, through the concept of sustainability, and following a qualitative methodology. By reading this chapter, we will be able to better understand the research process that has been carried out thanks to its exhaustive presentation, from the first step (e.g. the completion of a review of the literature) to the last (e.g. the attribution of the level of sustainability of the intervention). A more reflexive section also allows readers to become more involved in the researcher’s experience by having a better understanding of the challenges encountered (both methodological and professional) and the lessons that he has learned.

Photovoice method
Lynda Rey, Wilfried Affodégon, Isabelle Viens, Hind Fathallah and Maria-Jose Arauz

In program evaluation, it is essential to use evaluation methods which are adapted to the intervention context, the evaluation objective and the target populations. However, this is quite challenging in the case of complex interventions and of marginalized communities. In this chapter, we are presenting a participatory evaluation method, called photovoice, using photos as a support for expression. This choice was particularly adapted to the youth and the indigenous communities targeted by a project of access to water, sanitation and hygiene in rural Mexico. This qualitative method was also selected for its creative and artistic characteristics, well suited to the evaluation objective : to assess the youth’s 
perceptions regarding the artistic activities they had participated in, as part of the Social Art for Behavior Change component of the project, to promote the adoption of adequate practices related to safe water and hygiene. The method’s strengths and weaknesses are outlined and the methodological process is described in detail. Findings from the evaluation are briefly presented, as well as lessons learned from the use of photovoice, in an effort to promote the application of this method which proved to be very useful and successful in the case studied.

Qualitative, computerized, participatory and inter-organizational evaluation. ÉQUIPO: a programme for women victims of violence in Bolivia
Mathieu Bujold et Jean-Alexandre Fortin

Development programs aim to produce change in communities concerned by complex socio-cultural issues e.g., women’s inequality, that evoke a plurality of meanings. Evaluation of these programs requires a systematic approach that promotes collaboration among stakeholders from different organizations. The three components of the ÉQUIPO method were devised to address these complex realities. The qualitative component aims to understand the processes of change from the points of view of those who experience complex situations. The participatory component, promoting the democratic participation of stakeholders, emphasizes that program evaluation is not an end in itself, but a means of improving social and collective conditions. The inter-organizational component stimulates exchanges between NGOs sharing similar missions. This method of learning in action was developed in 2012, during a pilot project designed to evaluate programs of Bolivian NGOs promoting equality between women and men. The objective of this chapter is not to present the results of the specific evaluation of these programs, but rather to describe the ÉQUIPO method, the context in which it emerged, and to discuss the repercussions and the challenges of its application from the perspectives of the co-evaluators who participated in this pilot project.

The photovoice method. An intervention with marginalized populations on access to drinking water, hygiene and sanitation in Mexico
Lynda Rey, Wilfried Affodégon, Isabelle Viens, Hind Fathallah et Maria-Jose Arauz

In program evaluation, it is essential to use evaluation methods which are adapted to the intervention context, the evaluation objective and the target populations. However, this is quite challenging in the case of complex interventions and of marginalized communities. In this chapter, we are presenting a participatory evaluation method, called photovoice, using photos as a support for expression. This choice was particularly adapted to the youth and the indigenous communities targeted by a project of access to water, sanitation and hygiene in rural Mexico. This qualitative method was also selected for its creative and artistic characteristics, well suited to the evaluation objective : to assess the youth’s perceptions regarding the artistic activities they had participated in, as part of the Social Art for Behavior Change component of the project, to promote the adoption of adequate practices related to safe water and hygiene. The method’s strengths and weaknesses are outlined and the methodological process is described in detail. Findings from the evaluation are briefly presented, as well as lessons learned from the use of photovoice, in an effort to promote the application of this method which proved to be very useful and successful in the case studied.

The analysis of an action research. Combination of approaches in the health sector in Burkina Faso
Aka Bony Roger Sylvestre, Valéry Ridde et Ludovic Queuille

Since 2011, action research has been conducted as part of a project to subsidize access to care for children under five, pregnant and lactating women in Sebba health district in Burkina Faso. It aims to test an innovation, the reimbursement of health care benefits delivered free of charge to patients in order to solve the difficulties inherent to the project of grant of access to care met by the health workers and the subsidizing NGO. As a difficulty he had in other workload for the health staff, the costs of production of data collection tools. The preliminary results of this action research being conclusive, it appeared necessary to analyze the approach used to implement it. To achieve this, an external analysis combined with a reflective analysis of the stakeholders made it possible to draw the appropriate lessons that made it possible to guide other action research. The success of such an analysis has gone through i) the development of an analysis grid with factual elements to carry out the external analysis, ii) a good preparation of the reflexive analysis and iii ) Stakeholder involvement in the analysis process from the beginning.

Mixed systematic reviews. Interventions using results-based funding
Quan Nha Hong, Anne-Marie Turcotte-Tremblay et Pierre Pluye

Systematic mixed studies reviews are literature reviews using a systematic approach to combine quantitative, qualitative and mixed methods studies. They are growing in popularity owing to their potential to provide a deep understanding of complex health interventions and problems. In this chapter, eight steps of systematic mixed studies reviews are presented: (a) formulate a review question, (b) define eligibility criteria, (c) identify sources of information, (d) select relevant studies, (e) appraise the quality of studies, (f) extract data and (g) synthesize extracted data from included studies. These steps are illustrated using an example of systematic mixed studies review on unintended consequences of performance-based financing intervention. Also, to provide guidance on how to combine qualitative and quantitative evidence, four synthesis designs are presented: (a) data-based convergent synthesis design, (b) results-based convergent synthesis design, (c) parallel-results convergent synthesis design, and (d) sequential synthesis design. This chapter ends with some challenges that can be encountered when conducting a systematic mixed studies review.

Integration in mixed methods. Conceptual framework for the integration of qualitative and quantitative phases, results and data
Pierre Pluye

Mixed methods (MM) are increasingly popular. In performing MM, researchers integrate qualitative (QUAL) and quantitative (QUAN) methods with respect to worldviews, methodologies and designs, research questions, data collection and analysis techniques, and study results. Several integration strategies have been proposed, but their conceptualization is usually design-driven, or fragmented, or not empirically tested. This is challenging for planning and conducting MM studies, and for training graduate students. Based on the methodological literature, this chapter presents a conceptual framework including types of integration, practical strategies, and possible combinations of strategies for integrating QUAL and QUAN phases, results and data. In the next chapter, this framework is tested using a review of 2015 MM studies with a method-detailed description. This supports the framework, illustrates multiple types of combinations, and provides global health examples. These two companion chapters contribute to advance methodological knowledge on MM via (a) a call for better reporting MM studies in scientific articles, and (b) a tested conceptualisation comprising three types of integration and nine specific strategies, which explain current and future possibilities for combining strategies to integrate QUAL and QUAN phases, results, and data.

The practice of integration in mixed methods. The multiple combinations of integration strategies
Pierre Pluye, Enrique Garcìa Bengoechea, David Li Tang, Vera Granikov

Mixed methods (MM) are increasingly popular. In performing MM, researchers integrate qualitative (QUAL) and quantitative (QUAN) methods with respect to worldviews, methodologies and designs, research questions, data collection and analysis techniques, and study results. Several integration strategies have been proposed, but their conceptualization is usually design-driven, or fragmented, or not empirically tested. This is challenging for planning and conducting MM studies, and for training graduate students. Based on the methodological literature, the previous chapter presented a conceptual framework including types of integration, practical strategies, and possible combinations of strategies for integrating QUAL and QUAN phases, results and data. In the present chapter, this framework is tested using a review of 2015 MM studies with a method-detailed description. This supports the framework, illustrates multiple types of combinations, and provides global health examples. These two companion chapters contribute to advance methodological knowledge on MM via (a) a call for better reporting MM studies in scientific articles, and (b) a tested conceptualisation comprising three types of integration and nine specific strategies, which explain current and future possibilities for combining strategies to integrate QUAL and QUAN phases, results, and data.

Quasi-experimental methods. The effect of the minimum legal age on alcohol consumption among young people in the United States
Tarik Benmarhnia et Daniel Fuller

Quasi-experimental methods have been developed to evaluate ‘natural experiments,’ an alternative design to randomized controlled trials that relies on the notion of the counterfactual to estimate causal effects. The objective of this chapter is to describe quasi-experimental methods, apply them empirically to an example, and provide all elements of the evaluation, allowing the reader to reproduce these analyses (in R and Stata) in their given context. The main objective of this type of policy evaluation is to attribute a causal effect of an intervention on a health indicator. We apply these methods to estimate the impact of a policy raising minimum legal drinking on alcohol consumption among young people. Two types of quasi-experimental methods are used in this evaluation: the Difference in Differences (DD) method and the discontinuous regression (RD) approach. We show that these approaches can be good solutions to evaluate policies when randomized controlled trials are impossible. These evaluation tools can be used to ensure that policies contribute to the improvement of population health, to inform and decide whether to maintain, modify or discontinue existing interventions and to ensure public resources are used effectively.

Cluster randomized trials. Maternal and child health
Alexandre Dumont

Cluster-randomised trials are used to assess the impact of health interventions with a very high level of evidenceexpected. While cluster randomized trials offer several advantages in low-resource countries compared to individual trials, they present some methodological issues because the unit of analysis is different from the unit of randomization and intervention. These challenges, both in terms of the intervention implementation and the statistical analysis, are presented based on a trial conducted in Mali and Senegal (QUARITE trial) to reduce maternal and neonatal mortality in hospitals. We will see how the difference-in-differences approach measures the effect of the intervention taking into account the secular trends, which often plays favorably on health indicators outside of any external intervention. The appropriate statistical models are presented in order to adjust the effect of the intervention on baseline case-mix and on the cluster effect which is specific to this type of trial.

Equity measures. Free obstetric care interventions
Tarik Benmarhnia et Britt McKinnon

Health equity is considered a priority for public health interventions and policies, but the concept of an equitable intervention varies in definition. We define an equitable intervention as one that has heterogeneity in its effects, and provides greater benefits for subgroups or territories of a population that have been defined as vulnerable. In this chapter, we assess the impact of an intervention establishing free obstetric care (childbirth costs excluded) on increasing equity and access to care. We notably present the concentration index and the Theil Index. Simulated data from population surveys is used to discuss important issues regarding measurement of inequalities and different indicators that can be used in evaluating the equitability of an intervention. Lastly, two evaluation strategies are discussed: a simple evaluation approach to illustrate the application of inequality measurement methods and an approach based on quasi-experimental designs. These tools to assess public health equity is important to ensure that interventions meet objectives to reduce inequalities and prevent them from contributing to increasing health disparities.

Cost-effectiveness analysis. An HIV/AIDS care decentralization intervention in Shiselweni, Swaziland
Guillaume Jouquet

The economic evaluation described here was finalized in 2013 and is part of a more comprehensive evaluation conducted by four independent evaluators and sponsored by Médecins sans Frontières (MSF) for the purpose of documenting past experiences and advocacy. The evaluated program started in 2007 in the Shiselweni region of Swaziland in partnership with MSF and the Ministry of Public Health. This AIDS treatment program had the specificity of being decentralized and integrated at the level of primary health centers. This was an innovative approach in Swaziland where AIDS treatment was centralized at the secondary health center level. Decentralized approaches to AIDS treatment in developing countries have been the subject of several studies showing their effectiveness. The purpose of this evaluation was to add an economic component. We have therefore conducted a retrospective cost effectiveness study from the service provider perspective with calculation of the incremental cost-effectiveness ratio (ICER). The main challenges for an evaluation such as this one reside 1) in the access to the financial data of the service providers 2) to the measure of the effectiveness gain compared to the alternative approach and isolating possible confounding factors.

Spatial analysis. Community intervention to control the Aedes aegypti mosquito in Burkina Faso
Emmanuel Bonnet, France Samiratou Ouédraogo et Diane Saré

This chapter present a spatial analysis of an intervention which aims at assessing whether similar values of an outcome are more likely to form clusters in a study area. This method called local indicators of spatial association (LISA) enable to identify areas where values are spatialy dependant with one another. LISA has been used to evaluate the effect of a community-based intervention which aims at controlling the mosquito responsable for spreading Dengue, Aedes aegypti in two neighborhoods of Ouagadougou the capital city of Burkina Faso. The final evalutation of the intervention using this method showed that larva clusters disappeared in the neighborhood which received the intervention while they persisted in the control neighborhood.

Analysis of implementation processes. A complex intervention in Burkina Faso: results-based financing
Valéry Ridde et Anne-Marie Turcotte-Tremblay

More research on implementation is needed to understand how interventions work in their real context and how they evolve over time. The objective of this chapter is to present the process of analyzing the implementation of a complex intervention, called performance-based financing, in Burkina Faso. We conducted a multiple and contrasted case study (18 healthcare facilities) with several levels of analysis. The empirical evidence comes from observations, informal and formal interviews and documents. To facilitate the analysis of a large amount of data, we used a pyramid process whereby each team member analyzed and synthesized data at a different level (healthcare facility, district, general). Thus, the research assistants participated in the whole process of analysis. The organization of a workshop on the utilization of a free qualitative analysis software allowed the development of a common understanding of the analysis process. The results of the study highlight the importance of opening the “black box” to understand the implementation of interventions in global health and to shed light on their adaptations according to the contexts and the social actors.

Evaluation of the fidelity of implementation. A project to distribute Arctic charr to pregnant women in Nunavik
Lara Gautier, Catherine M. Pirkle, Christopher Furgal et Michel Lucas

In 2011, the Nunavik Regional Board of Health and Social Services began supporting the Arctic Char Distribution Project (AC/DP) for pregnant women. This initiative promoted consumption of a traditional Inuit food— the fish Arctic char — for pregnant women living in villages of Nunavik, an area in northern Quebec (Canada) inhabited predominantly by people of Inuit ethnicity. This intervention was intended to reduce exposure to environmental contaminants and improve nutriritional status and food security. The project’s implementation was assessed based on data collected from background documentation, field notes and qualitative interviews with project recipients and implementers. Themes emerging from the data were discussed in the light of the framework for implementation fidelity developed by Carroll et al in 2007. Pregnant women fully embraced the initiative for its cultural appropriateness. However, project implementation was incomplete because it did not cover all intended geographic areas, and there was a recurring inconsistency in the supply and distribution of the fish. In addition, the initiative has been inconsistently funded. This work highlights the extent to which project complexity can impede successful implementation, particularly in terms of communication and coordination.

Assessment of fidelity and adaptation. Implementing global health interventions
Dennis Pérez, Marta Castro et Pierre Lefèvre

This chapter is about the evaluation of the implementation fidelity and adaptation while building evidence for global health interventions. This issue is particularly relevant for adaptive interventions. Fidelity or the degree to which an intervention is implemented as intended by its developers is an implementation outcome that is particularly meant to ensure that the intervention maintains its intended effects. Adaptation, on the opposite is the process of bringing changes to the original design of an intervention by its implementers or users. Five dimensions have been forwarded to measure fidelity: adherence, dose, quality of delivery, participant responsiveness, and program differentiation. However, there has been little research or practical advice on how to adapt an intervention to maintain its effective ingredients and mechanisms. Interventions can be a blend of both fidelity and adaptation; thus, evaluation must carefully examine both aspects. We confronted our empirical research with existing literature on fidelity. As a result, we considered that the framework for implementation fidelity proposed by Carroll and colleagues responds to our concern to assess adaptation in the context of fidelity. We proposed modifications to the framework to accommodate adaptable global health interventions with practical implications for the evaluation of fidelity and adaptation.

Realistic evaluation. The adoption of a public health policy in Benin
Jean-Paul Dossou et Bruno Marchal

Realist Evaluation (RE) identifies the causal mechanisms explaining the results of a given intervention in different contexts. Based on the evaluation of the implementation of a health policy in West Africa, this chapter illustrates the different stages of the RE cycle. In the first stage, the research question is formulated. The added value of the RE approach is optimal on questions such as « why, how, for whom, and in what contexts does an intervention work or not? ». The second step is the formulation of the initial program theory, which depends on the nature of the intervention being studied, the objectives, and the nature of the main outputs expected from the evaluation, the resources available and the evaluator’s familiarity with the object of evaluation The Data collection is method neutral and can be based on a particular form of qualitative interview called « realist interview ». In practice, data collection must adapt to key informants, their perceptions and expectations vis-à-vis the evaluators to avoid the tendency for systematic confirmation of the evaluator theory by the interviewees. The realist analyst actively identifies the « Intervention-Context-Actor-Mechanism-Outcome » configurations. In practice, retroductive analysis is used to explicit mechanisms that are implicit by nature. A table to track changes to the initial program theory facilitates the final step of the synthesis and the reporting of the RE.