Results Chains: A Tool For Measuring Effectiveness And Attributing Change To Conservation Programs

Results Chains: A Tool for Measuring Effectiveness and Attributing Change to Conservation Programs

Environmental Evaluators Forum Washington D.C. June 12, 2008

This Presentation 1. Background on the Use of Evaluation & Adaptive Management in the Field of Biodiversity Conservation 2. What Is a Results Chain 3. How to Develop Results Chains 4. How to Use Results Chains as a Framework for Attributing Change to Conservation Programs 5. Build Example Chains

This Presentation •

• • •

•

Background on the Use of Evaluation & Adaptive Management in the Field of Biodiversity Conservation What Is a Results Chain How to Develop Results Chains How to Use Results Chains as a Framework for Attributing Change to Conservation Programs Build Example Chains

Common Questions

• Are we achieving an impact? • Are we doing the right things? • Are we doing them well?

To Achieve Success, We Need To:

• Develop sound “theories of change” • Measure the results of and improve • •

effectiveness of actions Assess impact on ecosystems and species Convince managers, donors, and supporters that results are credible

How Can We Help Practitioners Meet These Challenges?

Adaptive Management

What is Adaptive Management? The integration of design, management, and monitoring to provide a framework for: • Testing assumptions • Adaptation • Learning

Foundations of Success Our Mission

To improve the practice of conservation through adaptive management – working with practitioners to systematically test assumptions, adapt, and learn.

Foundations of Success Our Goals Our goals involve achieving the three foundations of success:

• Foundation #1: • •

Define clear goals and practical measures of success. Foundation #2: Determine guiding principles for using conservation strategies and tools. Foundation #3: Develop and strengthen practitioners’ ability to do AM

The Conservation Measures Partnership: Leading Conservation Organizations

Core Members: Collaborating Members:

Funding Support:

The CMP Open Standards for the Practice of Conservation www.ConservationMeasures.org

[email protected]

Many Versions of Adaptive Management in Practice

CMP Open Standards

Results Chains – A Tool for Implementing the CMP Open Standards Results chains

How are the CMP Open Standards Structured?

Steps Sub-steps Description Outputs References

Example from the CMP Standards Step 1: Conceptualize Sub-step 1: Complete Situation Analysis Description: This standard asks you to complete the description of the context within which your project takes place… Outputs: • Identification and analysis of indirect threats and opportunities. • Assessment of stakeholders. • Initial conceptual model that illustrates cause and effect relationships among factors operating at your site. • Ground-truthing and revision of your model. References

Interchange Between CMP and Orgs AWF Heartland Conservation Process

AWF

CI

1. Priority Setting

2. Heartland Selection Initial Scoping

Learning and Adaptive Management

Performance And Impact Assessment

4. Heartland Strategy Implementation, Evaluation, and Adaptation Implementation of Priority Interventions

Conservation Target and Goal setting

Socio-economic Analysis

TNC

3. Heartland/Landscapelevel Planning

Implementation Planning

Threats and Opportunity Analysis

5. Scale-down Heartland operations

1. Define    

Initial team Scope & vision Targets Context & stakeholders

5. Share    

2. Design

WWF’s Conservation Project/Programme Cycle

WWF

Lessons Formal products Feedback & evaluation Learning culture

4. Analyze/Adapt    

Incoming data Results & assumptions Operational functions Plans & budgets

 Action plan: goals, objectives & activities  Monitoring plan  Operational plan

3. Implement    

Workplans & budgets Fund raising Capacity building Partnerships

CMP

Define the context Vision and scope Threats mapping Stakeholders

Review Progress and Revise Approach Evaluate impacts Adapt to changes Share lessons learned

WCS Implement Actions and Measure Effectiveness Work-plans and Budgets Implement actions Monitor progress

Design Approach and Measures of Success Select targets Create conceptual models Build conservation landscapes Monitoring frameworks

This Presentation •

• • •

•

Background on the Use of Evaluation & Adaptive Management in the Field of Biodiversity Conservation What Is a Results Chain How to Develop Results Chains How to Use Results Chains as a Framework for Attributing Change to Conservation Programs Build Example Chains

What is a Results Chain? A tool that clarifies assumptions about how conservation strategies contribute to reducing threats and achieving the conservation of targets

What is a Results Chain? The Basic Components of a Results Chain: Strategy

What is a Results Chain? The Basic Components of a Results Chain: Strategy Impact on Target Goal

What is a Results Chain? The Basic Components of a Results Chain: Strategy Result Objective

Result (Direct Threat )

Objective

Impact on Target Goal

Results Chain Terminology Logic Model Inputs

Activities

Outputs

Outcomes

Impacts

Results Chain Inputs

staff, time, money, other resources

Strategy

set of actions undertaken by a project

Process

Outputs

immediate products of project activities

Outcomes interim results (objectives) achieved by outputs

Outcomes interim results (objectives) achieved by outputs

Results

Impacts

desired end goals of the project

What is a Results Chain? A diagram of a series of “if…then” causal statements that: • Defines how a project team thinks a strategy will contribute to reducing a threat and conserving a target • Focuses on the achievement of results – not the implementation of activities • Is composed of assumptions that can be tested

Results Chains Implicit Assumptions: Strategy

?

?

Conservation target improved

Achieving Success SUCCESS!

Accurate Results Chain

used to design

Well executed project

leads to

used to design

Well executed project

does not lead to

Desired results

used to design

Poorly executed project

does not lead to

Desired results

used to design

Poorly executed project

does not lead to

Desired results

Desired results

THEORY FAILURE

Inaccurate Results Chain PROGRAM FAILURE

Accurate Results Chain TOTAL FAILURE!!!

Inaccurate Results Chain

This Presentation •

• • •

•

Background on the Use of Evaluation & Adaptive Management in the Field of Biodiversity Conservation What Is a Results Chain How to Develop Results Chains How to Use Results Chains as a Framework for Attributing Change to Conservation Programs Build Example Chains

How to Develop a Results Chain 1. Construct an initial results chain 2. Complete the links in the results chain 3. Verify that your results chain meets criteria of a good results chain

A Chain From the Model Allows You to… Traditions & consumer preferences

Demand for caviar

Harvesting for caviar High price for caviar

Need to generate income

Sturgeon Pollution from domestic sewage

Inadequate zoning regulations Limited gov’t capacity for land use planning Rapid urbanization

Need for electricity

Gov’t policies favorable to urban dev.

Global warming

Dams

Clearing for new home construction Population growth

Failure of rural economy

Scope: Volga River watershed

Cheap land

Demand for second homes

Volga River & tributaries

Riparian forest

Forest corridors

A Chain From the Model Allows You to… Media campaign

Traditions & consumer preferences

Demand for caviar

Harvesting for caviar High price for caviar

Need to generate income

Sturgeon Pollution from domestic sewage

Inadequate zoning regulations Limited gov’t capacity for land use planning Rapid urbanization

Need for electricity

Gov’t policies favorable to urban dev.

Global warming

Dams

Clearing for new home construction Population growth

Failure of rural economy

Scope: Volga River watershed

Cheap land

Demand for second homes

Volga River & tributaries

Riparian forest

Forest corridors

1. Construct an Initial Results Chain

Healthy sturgeon population

1. Construct an Initial Results Chain

Reduction in harvesting for caviar

Healthy sturgeon population

1. Construct an Initial Results Chain

Media campaign

?

Reduction in harvesting for caviar

Healthy sturgeon population

1. Construct an Initial Results Chain

Media campaign

Consumers prefer not to eat caviar

Decreased demand for caviar

Reduction in harvesting for caviar

Healthy sturgeon population

2. Complete the Links in the Results Chain

Media campaign

?

Consumers prefer not to eat caviar

Decreased demand for caviar

Reduction in harvesting for caviar

Healthy sturgeon population

2. Complete the Links in the Results Chain

Media campaign

Increased knowledge of importance of sturgeon

?

Consumers prefer not to eat caviar

Decreased demand for caviar

Reduction in harvesting for caviar

Healthy sturgeon population

2. Complete the Links in the Results Chain

Media campaign

Increased knowledge of importance of sturgeon

Increased interest in sturgeon conservation

Consumers prefer not to eat caviar

Decreased demand for caviar

Reduction in harvesting for caviar

Healthy sturgeon population

3. Review the Criteria for Good Results Chains

Media campaign

• •

Increased knowledge of importance of sturgeon

Increased interest in sturgeon conservation

Consumers prefer not to eat caviar

Decreased demand for caviar

Reduction in harvesting for caviar

Healthy sturgeon population

Results oriented: Boxes contain desired results (e.g., reduction of hunting), and not activities (e.g., conduct a study). Connected in a “causal” manner: There are clear connections of “if…then” between each pair of successive boxes.

3. Review the Criteria for Good Results Chains

Media campaign

•

Increased knowledge of importance of sturgeon

Increased interest in sturgeon conservation

Consumers prefer not to eat caviar

Decreased demand for caviar

Reduction in harvesting for caviar

Healthy sturgeon population

Demonstrates changes: Each box describes how you hope the relevant factor will change (e.g., improve, increase, or decrease).

3. Review the Criteria for Good Results Chains

Media campaign

Increased knowledge of importance of sturgeon

Increased interest in sturgeon conservation

Consumers prefer not to eat caviar

Decreased demand for caviar

Reduction in harvesting for caviar

Healthy sturgeon population

•

Relatively complete: There are sufficient boxes to construct logical connections but not so many that the chain becomes overly complex.

•

Simple: There is only one result per box.

How to Develop a Results Chain 1. Construct an Initial Results Chain Based on Your Conceptual Model 2. Complete the Links in the Results Chain 3. Verify that Your Results Chain Meets Criteria of a Good Results Chain

What is a Results Chain? A diagram of a series of “if…then” statements (“causal”) that: • Defines how a project team thinks a strategy will contribute to reducing a threat and conserving a target • Focuses on the achievement of results – not the execution of activities • Is composed of assumptions that can be tested

What is NOT a Results Chain? It is not an implementation flow diagram… Media campaign

Identify target audience

Produce educational materials

Distribute educational materials

Monitor & evaluate the campaign’s effectiveness

Healthy sturgeon population

Your Turn: Which of the Following is NOT a Results Chain? A.

Lobbying of government for stronger regulations

Identify key decision makers

Educate decision makers

Decision makers pass laws

Increased yields

B.

C.

Promotion of sustainable agriculture

Community capacity building for forest resource management

Farmers implement sustainable agriculture methods

Greater indigenous knowledge about rights

More permanent crops

More control of & vigilance over external actors

Research & develop regulations

Increased permanence of agricultural occupation

More illegal wood confiscated

No wildlife trade

Less conversion of forest to agriculture

Less illegal selective logging in indigenous communities

Jaguar populations increased

Coastal forests conserved

Miombo woodland conserved

Primary forest conserved

This Presentation •

• • •

•

Background on the Use of Evaluation & Adaptive Management in the Field of Biodiversity Conservation What Is a Results Chain How to Develop Results Chains How to Use Results Chains as a Framework for Attributing Change to Conservation Programs Build Example Chains

Steps to Attributing Change to Programs 1. Define the Program’s “Theory of Change” 2. Develop Key Results into Good Objectives 3. Define Indicators for Objectives and Goals 4. Measure Indicators 5. Define What Results are Directly vs. Indirectly Attributable to Program

Example from the Meso-American Reef

Mexico

Belize

Guatemala

Honduras

1. Define the Program’s Theory of Change

1. Define the Program’s Theory of Change

Promote Agricultural Best Mgmt Practices (BMPs)

Coral reefs

Obj Ag6, Ag7

Less agrochemical contamination in marine waters & organisms

Result

Seagrasses

Littoral zone

KEY Strategy

Mangroves

Threat Reduction Result

Target

1. Define the Program’s Theory of Change Obj Ag1

Obj Ag2 Obj Ag3

Promote Agricultural Best Mgmt Practices (BMPs)

Bioaccumulation study completed & shared with agroindustry

Companies agree to participate in program to reduce agrochemicals

Companies sign MOU

Companies accept BMP plan

Coral reefs

Obj Ag6, Ag7

Less agrochemical contamination in marine waters & organisms

Result

Seagrasses

Littoral zone

KEY Strategy

Mangroves

Threat Reduction Result

Target

1. Define the Program’s Theory of Change Obj Ag1

Obj Ag2 Obj Ag3

Promote Agricultural Best Mgmt Practices (BMPs)

Bioaccumulation study completed & shared with agroindustry

Companies agree to participate in program to reduce agrochemicals

Companies sign MOU

Companies accept BMP plan

Coral reefs

Obj Ag6, Ag7

Obj Ag4, Ag5

Companies implement BMPs

Result

Less toxic agrochemical contamination in streams & rivers

Mangroves

Seagrasses

Littoral zone

KEY Strategy

Less agrochemical toxicity in plantations

Less agrochemical contamination in marine waters & organisms

Threat Reduction Result

Target

1. Define the Program’s Theory of Change Obj Ag1

Obj Ag2 Obj Ag3

Promote Agricultural Best Mgmt Practices (BMPs)

Bioaccumulation study completed & shared with agroindustry

Companies agree to participate in program to reduce agrochemicals

Companies accept BMP plan

Companies sign MOU

Coral reefs

Obj Ag6, Ag7

Obj Ag4, Ag5

Companies implement BMPs

Less agrochemical toxicity in plantations Other companies accept BMPs

KEY Strategy

Result

Threat Reduction Result

Target

Less toxic agrochemical contamination in streams & rivers Industry adopts BMPs

Less agrochemical contamination in marine waters & organisms

Mangroves

Seagrasses

Littoral zone

2. Develop Key Results Into Good Objectives Ag5: By the end of 2012, reduce by 40% the total pesticide toxicity from fungicides, insecticides, and herbicides used on Promote Companies agree banana, citrusBioaccumulation and sugarcane inin highCompanies run-off Agricultural Best study completed to participate Mgmt Practices & shared with program to reduce sign MOU Mesoamerican Reef watershed areas (BMPs) agroindustry agrochemicals Obj Ag1

Obj Ag2

Obj Ag3

Companies accept BMP plan

Coral reefs

Obj Ag6, Ag7

Obj Ag4, Ag5

Companies implement BMPs

Less agrochemical toxicity in plantations Other companies accept BMPs

KEY Strategy

Result

Threat Reduction Result

Target

Less toxic agrochemical contamination in streams & rivers Industry adopts BMPs

Less agrochemical contamination in marine waters & organisms

Mangroves

Seagrasses

Littoral zone

3. Define Indicators for Objectives and Goals Ag5: By the end of 2012, reduce by 40% the total pesticide toxicity from fungicides, insecticides, and herbicides used on Promote Companies agree banana, citrusBioaccumulation and sugarcane inin highCompanies run-off Agricultural Best study completed to participate Mgmt Practices & shared with program to reduce sign MOU Mesoamerican Reef watershed areas (BMPs) agroindustry agrochemicals Obj Ag1

Obj Ag2

Obj Ag3

Companies accept BMP plan

Coral reefs

Obj Ag6, Ag7

Obj Ag4, Ag5

Companies implement BMPs

Less agrochemical toxicity in plantations Other companies accept BMPs

Less toxic agrochemical contamination in streams & rivers Industry adopts BMPs

KEY Strategy

Result

Threat Reduction Result

Target

Indicator: total pesticide toxicity from fungicides, insecticides, and herbicides used on banana, citrus and sugarcane

Less agrochemical contamination in marine waters & organisms

Mangroves

Seagrasses

Littoral zone

3. Define Ag1: By the endIndicators of 2008, developfor a listObjectives of agrochemicals to reduce… Obj Ag1

Obj Ag2

and Goals

Ag4: By the end of 2012, at least 16 companies are implementing BMP Companies Companies accept BMP programs sign MOU plan Obj Ag3

Promote Agricultural Best Mgmt Practices (BMPs)

Bioaccumulation study completed & shared with agroindustry

Companies agree to participate in program to reduce agrochemicals

Obj Ag6, Ag7

Obj Ag4, Ag5

Indicator: list developed

Companies implement BMPs

Less agrochemical toxicity in plantations

Less toxic agrochemical contamination in streams & rivers

Less agrochemical contamination in marine waters & organisms

Goal for CoralOther Reefs: Industry Goal for Coral Reefs: Indicator: # of companies By 2017, ataccept least 25%adopts of coral reefs in the BMPs BMPs companies implementing MAR (including all reef types) are KEY BMP programs Threat ecologically viable.* Reduction Target Result Strategy Result * ecologically viable = accretion rate >/= X, live coral coverage of 7-20%, healthy population of herbivores, “balanced” species composition, etc.

Coral reefs

Mangroves

Seagrasses

Littoral zone

4. Measure Indicators

4. Measure Indicators Indicator: total pesticide toxicity on banana plantations Possible Monitoring Strategies: • Pre-test / Post-test • Time-series • Comparison with strict control group • Comparison with comparison group

5. Define What Results are Directly vs. Indirectly Attributable to Program Obj Ag1

Obj Ag2 Obj Ag3

Promote Agricultural Best Mgmt Practices (BMPs)

Bioaccumulation study completed & shared with agroindustry

Companies agree to participate in program to reduce agrochemicals

Companies accept BMP plan

Companies sign MOU

Coral reefs

Obj Ag6, Ag7

Obj Ag4, Ag5

Results directly attributable to WWF

Companies implement BMPs

Less agrochemical toxicity in plantations Other companies accept BMPs

Less toxic agrochemical contamination in streams & rivers

Less agrochemical contamination in marine waters & organisms

Industry adopts BMPs

Result

Threat Reduction Result

Target

Results indirectly attributable to WWF

Seagrasses

Littoral zone

KEY Strategy

Mangroves

Impossible to attribute results solely to WWF

This Presentation •

• • •

•

Background on the Use of Evaluation & Adaptive Management in the Field of Biodiversity Conservation What Is a Results Chain How to Develop Results Chains How to Use Results Chains as a Framework for Attributing Change to Conservation Programs Build Example Chains