23 Outcome Harvesting

Abstract

Outcome harvesting is a qualitative approach to ex post evaluation of social change results. Rather than testing for a specific impact of the intervention at hand, it consists in, in collaboration with stakeholders: broadly making sense and collecting evidence on outcomes, investigating how these were produced and whether and how the intervention may have played a role in this process, before substantiating these outcomes with external sources. Outcome harvesting is a utilisation-focused approach: it aims at producing knowledge for action. It is particularly useful in the case of complex interventions. In the case of complex interventions, when the effects of an intervention are previously not known or identified, or when the intervention has been significantly modified since its inception.

Keywords: Qualitative methods, case study, observable changes, outcome statements, harvest, impact

I. What does this approach consist of?

Outcome harvesting is a qualitative approach to monitoring and evaluation in which harvesters, meaning evaluators, identify, formulate, verify, analyse and interpret outcomes. Outcomes are defined as observable changes in the behaviour of individuals or collective actors such as groups, communities, organisations, institutions. Outcomes are changes in the practices, relationships, policies, actions, and activities of these individuals or actors. An outcome could be the increased enrolment of women in vocational training or the use of an innovative management practice by a farmers organisation. Outcomes can be positive or negative, intended or unintended, expected or unexpected. They are not to be confused with impacts, or the ultimate consequences of the intervention in wellbeing dimensions of the targeted beneficiaries, such as health or education. In outcome harvesting terms, impacts (e.g. in health, education) cannot be achieved without behaviour change (e.g. application of feeding practices, training attendance). Outcome harvesting therefore focuses on these behaviour changes.

Outcome harvesting provides evidence on what outcomes were achieved and how an intervention contributed, and the meaning of these outcomes in light of the evaluation questions. Outcome harvesting does not focus on the progress or achievement of intended, expected or planned outcomes of the intervention, but collects evidence on what has been achieved and works backwards to identify whether and how the intervention contributed to these changes. In the harvest, all the changes that actually happened are collected, which allows to capture also unintended outcomes, positive and negative. The evaluator facilitates their identification and the search for evidence of how they were achieved, and of the contribution of the intervention. Outcome harvesting is utilisation-focused: its purpose is to serve uses of the evaluation findings by the intended users, meaning those who will make decisions based on these findings. The focus is on learning from the evaluation in order to take action.

The key element of outcome harvesting is outcome statements that describe the change, who made it, when and where, what was the plausible contribution of the intervention’s activities, strategies and outputs to each change, and the significance of the change. As an example: since 2015, the agricultural department of Senegal has been issuing a newsletter presenting meteorological forecasts and targeted agricultural recommendations using a language and format adapted to farmers and the wider public, democratising access to scientific knowledge. The intervention contributed to this change through a study on farmer’s preferred communication channels and formats and a series of scientific communication trainings.

Outcome harvesting is carried out in six steps, through which outcome statements are refined and evidenced. The first step is to design the harvest in order to respond to the intended uses of the findings, defined by their primary users. The second step is to review documentation to identify and formulate draft outcome statements. The third step is to engage with the people who know how the intervention contributed to these changes, who review and refine the outcome statements with the evaluator until a set of precise statements is identified. Clearly defining what has changed, the contribution of the intervention and the significance of the change allows us to bound which outcomes are taken forward for the assessment and which are not. The idea is to strive for fewer verifiable outcomes for which to collect evidence in the next step. The fourth step is the substantiation of the outcomes with external sources that are independent from the intervention but knowledgeable about the outcome and can validate the contribution of the intervention. Substantiation allows us to verify the accuracy of outcomes, but also to enrich understanding of the change and the contribution of other actors or interventions. Other changes linked to the intervention that the primary users had not identified can emerge during the substantiation step. The fifth step is to analyse and interpret the outcome statements, systematising the evidence to answer the evaluation questions defined in the design step. The sixth step is about supporting users to use findings for their intended uses.

The 6 steps of outcome harvesting

1. Design the harvest

2. Review documentation, draft outcome statements

3. Engage with stakeholders

4. Substantiate the outcomes with external sources

5. Analyse and interpret outcome statements

6. Support use of findings by stakeholders

II. How is this approach useful for policy evaluation?

Outcome harvesting is a retrospective or ex post evaluation method, but it can also be used to monitor progress during an intervention. It is particularly useful for complex interventions where the programming context is unpredictable, uncertain, dynamic, and planned courses of actions are likely to change. Outcome harvesting is especially appropriate to: 1) monitor and evaluate interventions on emerging challenges for which little information and evidence exists; 2) generate evidence of what changes were achieved by an intervention that did not pre-define changes or had very general pre-defined changes; 3) provide evidence on changes generated directly and indirectly, intended or unintended; 4) evaluate an intervention that has changed significantly from what was originally planned.

This approach can be used to monitor an ongoing intervention by periodically and systematically collecting information and learning on the social changes achieved and the intervention’s contribution to them. It can be combined with qualitative or quantitative methods that address other evaluation questions. For instance, it can complement findings of methods that quantify the well-being related impacts to which have led the changes harvested.

Outcome harvesting focuses on the contribution of the intervention’s actions to social change, or the plausible and logical relationship between intervention and change, rather than the exact part attributable to this action. Unlike other contribution-based approaches (see separate chapter on contribution analysis), outcome harvesting does not depart from investigating the cause-effect link to intended outcomes. On the contrary, it collects “all” the observed results and then works backwards to reconstruct the contribution of the intervention to these changes. Indeed, the main interest of outcome harvesting lies in its ability to account for the dynamics of social change by adopting an open and broad approach in identifying these changes, even when they are unexpected or unintended. For this reason, it is particularly suited to the evaluation of complex interventions where uncertainty, dynamic contexts and adaptation are common.

III. An example of the use of this approach: the scaling of a weather and climate information policy in Senegal

The following presents a case study using outcome harvesting within a broader evaluation approach to assess outcomes of the scaling of Weather and Climate Information Services (WCS) in Senegal and how research actions contributed to these outcomes (Blundo-Canto et al. 2021). WCS are the production, translation, transformation, transmission, access and use of scientific information on weather and climate to support decision-making. In Senegal, the dissemination of weather and climate forecasts along with recommendations for economic sectors and actors expanded from pilot research projects to a national-level strategy over two decades. The evaluation of the outcomes of this scaling process had an accountability and a learning objective, and was based on three components. The reconstruction of the history of the innovation (Douthwaite et Ashby 2005): the detailed timeline and interconnection of events, factors, actions and actors that marked the scaling of WCS. The Outcome harvesting (Wilson-Grau 2018): the assessment of changes in observable practices, relationships, policies, actions, and activities of the actors involved in and affected by the scaling of WCS, and the contribution of research partnerships to these outcomes. The analysis of the impact pathway (Douthwaite et al. 2003): the causal chain leading from research actions of the National Agency of Civil Aviation and Meteorology (ANACIM by its French acronym) and its partners to the outcomes identified and their perceived medium and long-term effects.

The six steps of the outcome harvesting approach guided the implementation of the three components. In the design step, evaluators discussed with the leading change agent (ANACIM) the design of the harvesting, identifying documentary sources and key actors that would need to participate in the formulation step. In step two, existing documentation was reviewed to reconstruct the history of the innovation and pre-identify outcomes that could be linked to the scaling process, which were discussed with the change agent. In the formulation step, a workshop with 16 representatives of national and local actors involved in the scaling process was organised to reconstruct its key events, actors, actions, and contextual factors. The workshop allowed us to identify other actors involved in the process and some additional outcomes. The outcomes identified in the previous three steps were systematised and formulated, and subsequently substantiated through individual interviews with 44 knowledgeable informants. These informants were independent from the leading change actor, the ANACIM. Nonetheless, the particularity of the policy process studied, a scaling up from a pilot project to a nationally-wide action involving many actors and sectors made so that some of the participants of the workshop were interviewed in the substantiation process to validate and provide evidence of the outcomes for which they were knowledgeable informants. With the evidence from the substantiation step, outcomes were refined as well as the contribution of the research actions of the ANACIM and its partners along with other actors and factors. In order to support use, three restitution workshops at the national and local level were carried out, in which results of the outcome harvesting were presented and discussed by participants, as well as possible ways forward based on the knowledge generated.

The key findings of the approach combining outcome harvesting, innovation histories and impact pathway analysis can be summarised as follows. In the past two decades, Weather and Climate Services have been serving as key policy instruments to tackle increased rainfall variability and extreme climate events that affect vulnerable rural communities in the West African Sahel. The innovation history starts in the 1980s when, following a devastating drought, the Agriculture-Hydrology-Meteorology regional centre created the first multidisciplinary working group to facilitate the development of WCS, their interpretation, and their dissemination and uptake. In the 2000s, the Senegalese ANACIM partnered with national and international research actors to set up pilot decentralised multidisciplinary working groups to facilitate uptake of forecasts and recommendations at local level. Climate information was combined with agricultural advice in a language that used local concepts, habits and practices to make this information actionable by farmers. The multidisciplinary working groups met regularly during the rainy season to discuss transmission of forecasts and recommendations. Individuals that were influential in their farming communities were trained on forecasting concepts and interpretation in order to increase uptake by other farmers. In the years leading to 2018, WCS and multidisciplinary working groups were sequentially scaled up to most departments in Senegal.

The outcome harvesting allowed us to identify how climate information was incorporated in sectoral and national adaptation plans, strategies and programmes, as well as in the coordination of actions of multiple actors at the local level. It also allowed us to identify other sectors beyond agriculture, including fisheries, energy and water resources protection that were using WCS, showing that the outcomes generated crossed institutional, sectoral and governance boundaries. Beyond the actions of the ANACIM and its partners, this process was supported by a favourable global funding environment. By combining outcome harvesting with impact pathway analysis, it emerged that the scaling of WCS to new users, sectors and uses happened through five axes: 1) continuous improvement of WCS, 2) emergence and consolidation of WCS facilitators, 3) inclusion of WCS in action planning, 4) active mobilisation to sustain the scaling of WCS, and 5) empowerment of actors to take up new roles. The factors underlying the scaling process can be summarised as intended actions by the research partners, including capacity strengthening, knowledge sharing and action platforms, and creation of interaction opportunities; national and international financial support; and an enabling political environment. The continuous improvement of WCS through feedback from its users reinforced the scaling process, resulting in increased access to WCS for the population. The outcome harvesting also allowed to capture challenges raised by the expansion of WCS as new users and uses emerged. There is a growing demand for improved quality and finer-grain WCS that are delivered at the right time to make decisions, which requires significant investment. Issues of trust in who delivers the information, how it is produced, and how to interpret it can be an obstacle as WCS reaches more users but capacity building campaigns do not. Public-private partnerships could play an important role, but at present, the involvement of the private sector in delivering WCS is limited. The results of the evaluation were used for accountability of the research partners involved, for the production of scientific knowledge on the scaling of these policy instruments, and to discuss key challenges that the actors involved in WCS production, dissemination and use need to overcome.

IV. What are the criteria for judging the quality of the mobilisation of this approach?

Outcome harvesting focuses on evaluating outcomes, not impacts. It does not focus on counting beneficiaries or measuring the well-being effects they experience. It is important to make this explicit when judging the quality of this method. Its purpose is to identify intended, unintended, expected or unexpected changes in practices, relationships, policies, actions, or activities and assess the contribution of an intervention to these changes. Such an evaluation focuses on the use of findings by intended users for their intended uses, therefore its quality needs to be judged in terms of these intended uses (e.g. accountability or learning for adaptive management). For instance, the substantiation step is important when the purpose is accountability and final assessment but can be overlooked or carried out in a lighter way when the purpose is internal learning or monitoring.

To judge outcome statements, outcome harvesting uses the SMART criteria: each statement needs to be Specific, sufficiently detailed to be appreciated by any reader; Measurable, providing verifiable quantitative and qualitative information; Achieved, a plausible link between the contribution of the intervention and the outcome can be established; Relevant, the outcome is significant in light of the intervention’s purpose; Timely, the outcome occurred close to the time the evaluation is carried out, even if the contribution of the intervention happened a significant time before.

V. What are the strengths and limitations of this approach compared to others?

The key feature that makes outcome harvesting stand out compared to other evaluation methods is its focus on achieved outcomes independently of whether they had been planned or not, allowing to capture unintended or unexpected outcomes, both positive and negative. The method provides a systematic and structured way to identify these changes and to work backwards to determine whether and how the intervention contributed to them. Outcome harvesting produces quantitative and qualitative data to describe outcomes. However, it does not provide a quantitative assessment of these outcomes. Rather, it informs on the processes and strategies that have led to a quantitative outcome measured with other methods. It cannot be used for impact measurement. Applications of outcome harvesting that do not carry out the substantiation step, are better used for internal learning than for accountability as they do not include validation from independent and knowledgeable sources.

As other utilisation-focused approaches, outcome harvesting focuses on making the evaluation useful for its users and for the intended uses of the evaluation findings. The intended uses can be learning, decision-making, planning, accountability, informing partners, and so on, depending on what is agreed with the primary users at the design stage. This choice guides how the method is applied and the weight given to the substantiation of social change outcomes and the intervention’s contribution. The extent to which the contribution of the intervention is assessed in the harvest will be higher when the use is accountability at the end of an intervention than when the intended use is learning for adaptive management during the intervention.

Some bibliographical references to go further

This methodological note draws significantly from:

Wilson-Grau, Ricardo. 2018. Outcome Harvesting: Principles, Steps, and Evaluation Applications. IAP.

Blundo-Canto, Genowefa. and Andrieu, Nadine. and Soule Adam, Nawalyath. and Ndiaye, Ousmane. and Chiputwa, Brian. 2021. “Scaling Weather and Climate Services for Agriculture in Senegal: Evaluating Systemic but Overlooked Effects”. Climate Services, 22 (April): 100216. https://doi.org/10.1016/j.cliser.2021.100216.

Additional references mentioned:

Douthwaite, Boru. and Ashby, Jacqueline. 2005. “Innovation Histories: A Method for Learning from Experience”. The Institutional Learning and Change (ILAC) Initiative, 4. https://hdl.handle.net/10568/70176.

Douthwaite, Boru. and Kuby, Thomas. and van de Fliert, Elske. and Schulz, Steffen. 2003. “Impact pathway evaluation: an approach for achieving and attributing impact in complex systems”. Agricultural Systems, Learning for the future: Innovative approaches to evaluating agricultural research, 78(2): 243‑65. https://doi.org/10.1016/S0308-521X(03)00128-8.