Blog based on the Focali network meeting 24th of April 2014, written by Focali researcher Anna Nordén . 

The urgent need to conserve forest carbon stocks and preserve biodiversity in order to maintain forest ecosystem functions and services has led to increased popularity in using different types of forest conservation interventions (projects, policies, programs). Interventions stretch from command-and-control, such as national parks, to market-based instruments, such as Payment for Ecosystem Services (PES), and from local to national (or even global) levels. 

Since conservation funds are often scarce it is of great importance to understand what works (and why and where) to make sure that funds are used cost-effectively. To be able to assess if an intervention achieves the intended outcome, a rigorous impact evaluation is needed. An impact evaluation tries to understand the causality between an intervention and an outcome by measuring the impact of the intervention. For instance, to evaluate mitigated carbon dioxide emissions due to a PES scheme. 

Based on this topic Focali held a network meeting the 24th of April 2014 regarding impact evaluation for forest conservation interventions and specifically we ask: can theory and practice meet?

My colleague Martin Persson  kicked off the meeting by presenting a background on impact evaluation theory and applications to forest conservation policies, including insights from the recent workshop ‘Evaluating Forest Conservation Initiatives: New Tools and Policy Needs’, held in Barcelona, December 2013. The challenge with measuring impact is to establish the counterfactual: we can only observe what happened with the intervention, not what would have happened without it. The methods therefore focus on the construction of the counterfactual: what would have happened without the intervention. The difference between the counterfactual and the outcome of the intervention is in academia referred to as “additionality”, while aid agencies rather refer to “attribution”, e.g., the change that can be attributed to the intervention.

There are several methods that can be used to construct the counterfactual and make a rigorous impact evaluation. The method used is in the end often determined by the context and the budget of each intervention. The table below presents the options for creating the counterfactual and some pros and cons with each option. 

Table by Anna Nordén

Before choosing the methods to be used, the first step in impact evaluations is the establishment of a theory of change. A theory of change provides logic and a framework to know what to evaluate and when. It also enables to account for confounding factors (both observable and unobservable) to be able to investigate the impact of an intervention on an outcome. The confounding factors are essentially an understanding of the contextual setting and to characterize potential selection biases that needs to be accounted for. For several interventions the goal may be multiple, for instance, a PES scheme could aim at both carbon dioxide emissions mitigation and poverty alleviation. In such case, separate theories of change should the established. 

Ida Hellmark, Advisor at the Evaluation Department at NORAD, presented some findings from a study that looks into the reasons why it has proven difficult to measure results in the Norwegian aid (including Norwegian International Climate and Forest Initiative). The reason for conducting this study was that none of the evaluations and studies commissioned by the Evaluation Department and finalized in 2011 could report sufficiently on results at the level of outcomes or impact. However, it is important to note that “no evidence of effect” does not mean “there was no effect”, but is rather an indication that no rigorous impact evaluations took place. The presentation by Björn Merkell, International coordinator at the Swedish Forest Agency, on the evaluation of Swedish forest policies confirms a similar trend of measuring inputs and outputs (e.g., investments in forest conservation, timber production, size of voluntary set areas and total forest cover) rather than outcomes (e.g., carbon dioxide emissions mitigation caused by a policy). This trend seems to be a general finding on evaluating conservation interventions;

MEA (2005): ”Few well-designed empirical analyses assess even the most common biodiversity conservation measures.”

Pattanayak et al. (2010): ”…we do not yet fully understand either the conditions under which PES has positive environmental and socioeconomic impacts…”

Center for Evidence-Based Conservation, Bangor¹: ”Despite the growing amount of scientific information produced by the research community, conservation practice and policy remain largely experience-based with limited evaluation of what works and what does not.”

Despite the fact that manuals for how to conduct both theory of change and rigorous impact evaluations exist, why is the practical implementation of impact evaluation so low? Several obstacles where discussed:

• Researchers and practitioners are unaware of the availability of impact evaluation tools.

• Lack of incentives ̶ generally believed that rigorous impact evaluations are expensive and/or difficult.

• Evaluating conservation initiatives may hold specific challenges (limits to randomization, spatial leakage, short-term nature of conservation interventions, data availability poor, etc.).

• Political economy reasons – “It pays to be ignorant” (Pritchett 2002, Policy Reform).

      o Lack of or competing priorities by leaders

      o Political decisions – Like Norway’s 1% of GDP to aid target which puts huge pressure to spend funds quickly.

• Internal vs. external validity – ‘naïve’ estimates from the right context may better predict impacts than rigorous impact evaluation results applied ‘out of context’ (Pritchett and Sandefur 2013, CDG WP–336). 

In line with the conclusion from NORADs study of its own system, to overcome these obstacles implementation of impact evaluations are needed at the initial phase of an intervention (project, policy or program). When properly conduced, impact evaluations provide the evidence needed in order to inform politicians and other stakeholders of what is working and what is not, in our search for a more sustainable development.

Impact evaluations are a crucial area both in research and for practitioners. As part of the Focali network, my colleagues and I hope to further explore and develop the use of rigorous impact evaluations of forest conservation interventions and we hope you will join us in this attempt and share your own experiences.

1 - www.cebc.bangor.ac.uk/cebcbackground.php (2014-04-20).

Some reading tips and links:

Ferraro P. J., Pattanayak S. K. (2006) Money for Nothing? A Call for Empirical Evaluation of Biodiversity Conservation Investments. PLoS Biol, 4, e105.

Miteva D. A., Pattanayak S. K., Ferraro P. J. (2012) Evaluation of biodiversity policy instruments: what works and what doesn’t? Oxford Review of Economic Policy, 28, 69-92.

Leeuw F., Vaessen, J. (2009). Impact Evaluations and Development: Nonie Guidance on Impact Evaluation. siteresources.worldbank.org/EXTOED/Resources/nonie_guidance.pdf

Roe, D., Grieg-Gran, M., Mohammed, E.Y. (2013). Assessing the social impacts of conservation policies: rigour versus practicality. IIEED Briefing, http://pubs.iied.org/17172IIED

International Initiative for Impact Evaluation, J-PAL (www.povertyactionlab.org)