Outcome harvesting – an excellent tool for evaluating projects in complexity!

‘Outcome harvesting is an excellent tool for evaluating projects, that operate in a complex environment and that do not have a pre-defined strategy!’   This was one of the comments from one of the participants, who participated in a workshop conducted by Ricardo Wilson Grau, one of the founding fathers of the Outcome Harvesting method.  The workshop was hosted on the 23^rd June, 2015 in The Hague by the Nedworc Association and OXFAM Novib,   participants were Nedworc consultants and MEL-officers of OXFAM Novib.

Outcome harvesting is an evaluation approach, involving all stakeholders in a project,  in a program context where there are no clear relations between cause and effect.  Especially projects that involve lobby and advocacy,  fast changing contexts and unpredictable situations can benefit from the Outcome harvesting approach. In 2013, UNDP selected Outcome Harvesting as one of the 11 major Monitoring and Evaluation innovations. There is a toolkit guide and several case studies on how Outcome Harvesting is applied. Read more at:  http://betterevaluation.org/plan/approach/outcome_harvesting

Ricardo Wilson - Grau

The method has been developed by Ricardo Wilson Grau and his colleagues. They were inspired by the Outcome Mapping and the Utilization-Focused Evaluation.   Outcomes are defined as  change in behavior, relationships, actions & activities and  policies & practices of a  social actor (individual, groups & communities, institutions, organizations). The Outcome Harvesting method collects evidence on what has changed (who, what, when, where and how). then, working backwards, determines whether and how an intervention has contributed to these changes.  The evaluation focuses on effectiveness, rather than on efficiency, which is done in projects with casual relationships with activities, results and goals. The approach is composed of 6 steps;

Source;  http://betterevaluation.org/plan/approach/outcome_harvesting

Step 1, Design the harvest;  determines the set-up of the evaluation, the data collection methods and the evaluation questions that will be asked;

Step 2,  Focuses on review of the documentation and evidence collected.  In this phase the most important Outcome is defined and the most crucial activity or event that contributed to this outcome is determined.

In step 3, Engage with informants more detailed questions are asked from the stakeholders in the project (see in the diagram above)  in knowing more details about the outcome and the contribution.  This can be done in a  face-2-face workshop setting with  the help of a facilitator having all the stakeholders involved. However, it can also be done by e-mail posing the questions to each of the stakeholders. This is also the phase where significance will be given to the outcome by the informants.

Step 4; Substantiate.  This is the phase were external stakeholders, who have not been involved in the project but who are authorities in domains of the project, are consulted. They comment and give input on the data and evidence, that has been collected in step 2 and step 3.

Step 5, Analyse, interpret and step 6, support use of findings complete the Outcome Harvesting Cycle.  This is phase where lessons, conclusions and recommendations are drawn. This is where the learning amongst the stakeholders take place and where recommendations for improvement can be formulated.

During the workshop conducted by Ricardo Wilson Grau, the group worked on a case study of a funding agency, promoting the advancement of women’s rights.  We noticed that often specific information (What,  when, where and how?) is lacking and you need to ‘dig deeper’ to complete the information.

Outcome Harvesting is characterized by the following strengths:

§ Corrects the common failure to search for unintended results.

§ Has verifiable harvested outcomes.

§ Uses a logical, accessible approach that makes it easy to engage informants.

§ Employs various means to collect data: face-to-face interviews or workshops, communication across distances (surveys, telephone, or email), and written documentation.

§ Ties the level of detail provided in the descriptions directly to the questions defined at the outset of the process; these descriptions may be as brief as a single sentence or as detailed as page or more of text, and may or may not include explanations of other variables.

Because of its nature, Outcome Harvesting also has certain limitations and challenges:

§ Skill and time are required to identify and formulate high-quality outcome descriptions.

From: Outcome Harvesting, Ricardo Wilson-Grau and Heather Britt (2012, revised 2013) link http://www.outcomemapping.ca/resource/outcome-harvesting

Outcome Harvesting in practice – the GPPAC case

In the second half of the workshop, Paul Kosterink working with The Global Partnership for the Prevention of Armed Conflict (GPPAC), explained on how Outcome Harvesting is applied for projects in their organisation.  They got assistance and advice from Ricardo Wilson Grau in building their Outcome Harvesting design.  GPPAC  is a member-led network of civil society organisations (CSOs) active in the field of conflict prevention and peacebuilding across the world, http://www.gppac.net/nl .  GPPAC is active in a number of coalitions and networks in preventing armed conflict. Most of their work is focused on lobbying and advocacy through the networks.  ‘The context in which we work is very complex’ explains Paul Kosterink.

Paul Kosterink

When we ask our partners to report, they have difficulty in specifying and writing down their results. Therefore, we ask our partners concrete questions on what changed and on what they achieved,  explains Paul Kosterink.    In  the environment we work, it is difficult to reach big results within a short time. Therefore,  we ask our partners to come up with simple and low-key outcomes, which can signify that things are changing.  For example a letter from a Ministry of Defense giving compliments for our work specifying on how we prevent conflict, can already be a significant Outcome, says Paul Kosterink.

Lessons learned

Participants expressed that the workshop was useful and helpful.  Some of the participants quoted;

·      If we would have linked Outcome Harvesting to our reporting system, it would have produced much better reports;

·         The method is appealing. I am motivated to do more study on how the method works, before I start applying it;
·         I will propose the Outcome Harvesting (OH) method as a tool for a developmental  evaluation for a proposal, that I am developing;
·         The Outcome Harvesting is an excellent method for projects operating in a complex environment, that do not have a pre-defined strategy;
·         Outcome Harvesting will save you a lot of resources in time, people and money. The evaluator does not need to collect all the outcomes, but can start working from the evidence that is collected in the reports;
·         Outcome Harvesting enables to ask more detailed and interesting questions for doing an evaluation.
·         Outcome Harvesting is not easy, but it is very interesting!

Simon Koolwijk and Erica Wortel

June, 2015