Logical Frame (pdm)

L

E

A

D

Log-frame Evaluation Application Design

The Users Guide to L-E-A-D Log-frame Evaluation Application Design Year 2004 version

- A New Approach for Policy and Program Evaluation -

FASID Foundation for Advanced Studies on International Development

March 2005 ©FASID

Preface A decade ago, the Foundation for Advanced Studies on International Development (FASID) developed PCM (Project Cycle Management) that has become the de-facto standard management tool of international development projects. Nowadays, evaluation is needed not only for a project, but also for a program or a policy under which multiple projects are conducted. In 2002, FASID established an inter-institutional study group, named “Study Group for Evaluation on Public Policy and Programs”, to develop a new evaluation method for the next generation to be applicable for policy and program levels. With intensive efforts of experts participating in the study group, a new method “LEAD” (Log-frame Evaluation Application Design) was developed. In April 2003, “Production Unit for LEAD Users Guide” (PULUG) was established by young members of the study group. Since then, the unit pursued the refinement of the new method to be more practical and useful, in order to enable the method to be more easy to use and thereby leading to significant improvements for the users. This Users Guide was written to extend the practical knowledge of “LEAD” to people who are interested in using this new evaluation method. The readers are assumed to be planners, practitioners and managing officers in international development. In addition, this booklet will provide a new insight for others working for various social programs, such as public officers in national or local government, and it may be useful even for officers in private companies who wish to comprehensively improve the evaluation of their own projects. This Users Guide is intended to use plain and clear expressions in explanation for every reader to understand, whilst it may still be difficult or ambiguous for some readers. Therefore, any comments, complaints and suggestions are welcomed as “the evaluation for improvement”. We hope that LEAD leads you to the improvement of your projects and programs.

International Development Research Institute, FASID ©FASID

Contents １．INTRODUCTION ..............................................................................................................................1 0-0. FROM “EVALUATION” TO “IMPROVEMENT” ......................................................................................2 0-1. ABOUT LEAD ...................................................................................................................................3 0-2. CONTENTS OF LEAD .......................................................................................................................4 0-3. STRUCTURE OF THE USERS GUIDE .................................................................................................6 0-4. CASE .................................................................................................................................................6 ２．LEAD IN PRACTICE........................................................................................................................8 1. INITIATION PHASE .................................................................................................................................9

1-1. Evaluation Scope Planning ........................................................................................................9 a) Outline ......................................................................................................................................................... 9 b) Output Sample: Scope of Evaluation ......................................................................................................... 10 c) Explanatory Notes:..................................................................................................................................... 11 1-1-1.Evaluation for Whom? ............................................................................................................. 11 1-1-2. Preparing “Scope of Evaluation” ........................................................................................... 11 1-1-3. Setting Evaluation Criteria ..................................................................................................... 11 1-1-4. Cases Which can not be Applied to Theory-Based Evaluation like LEAD...................... 12 1-1-5. Good Relationship with Stakeholders .................................................................................. 12 1-1-6. LEAD as a Tool of Capacity Building.................................................................................... 13

2-1. Hierarchy Tree Development .................................................................................................14 a) Outline ....................................................................................................................................................... 14 b) Output Sample: Hierarchy Tree ................................................................................................................. 15 c) Explanatory Notes:..................................................................................................................................... 16 2-1-1. Hierarchy Tree......................................................................................................................... 16 2-1-2. Building a Hierarchy Tree ...................................................................................................... 16 2-1-3. Tips for Building a Hierarchy Tree ........................................................................................ 16 2-1-4. Coverage of a Hierarchy Tree ............................................................................................... 17 2-1-5. The Project without Activities and the Activity without the Project Name ....................... 17

2. PREPARATION PHASE ........................................................................................................................18

2-2. Logic Tree Development ........................................................................................................18 a) Outline ....................................................................................................................................................... 18 b) Output Sample: Logic Tree ........................................................................................................................ 19 c) Explanatory Notes...................................................................................................................................... 20 2-2-1. Logic Tree ................................................................................................................................ 20

©FASID

2-2-2. Setting Overall Goal ............................................................................................................... 20 2-2-3. “Means-Ends” Relationship ................................................................................................... 21 2-2-4. Tips for Setting Intermediate Objectives .............................................................................. 22 2-2-5. Theory-based Evaluation by Building a Logic Tree............................................................ 22

２-3. Projects Mapping ..…………………………………..…………………………………………...........23 a) Outline ....................................................................................................................................................... 24

2-4. Log-frame Formation ..............................................................................................................24 a) Outline ....................................................................................................................................................... 24 b) Output Sample : Log-Frame ...................................................................................................................... 25 c) Explanatory Notes:..................................................................................................................................... 26 2-4-1. Log-frame................................................................................................................................. 26 2-4-2.Special features of Log-frame for LEAD ............................................................................... 27 2-4-3. Log-frame as a Tool for Management .................................................................................. 27 2-4-4. Converting a Logic Tree into a Log-frame ........................................................................... 28 2-4-5. Setting Indicators and Means of Verification....................................................................... 29 2-4-6. Setting Important Assumptions ............................................................................................. 31 2-4-7. Setting Inputs........................................................................................................................... 32

3. EVALUATION PHASE ...........................................................................................................................34

3-1. Evaluation Study Design .........................................................................................................34 a) Outline ....................................................................................................................................................... 34 b)Output Sample: Evaluation Grid................................................................................................................. 35 c) Explanatory Notes...................................................................................................................................... 37 3-1-1. Evaluation Purpose and Evaluation Criteria ....................................................................... 37 3-1-2. Program Level Evaluation...................................................................................................... 37 3-1-3. Designing Evaluation Study................................................................................................... 40 3-1-4. Setting the Methods of Data Collection ............................................................................... 41 3-1-5. Data Collection for Factor Analysis ...................................................................................... 42

3-2. Evaluation Study Implementation ...........................................................................................44 a) Outline ....................................................................................................................................................... 44 b) Output Sample: Evaluation Grid with Results ........................................................................................... 45 c) Explanatory Notes...................................................................................................................................... 47 3-2-1. Implementation of the Evaluation Study .............................................................................. 47 3-2-2. Question of Attribution ............................................................................................................ 47 3-2-3. Sorting the Results of the Evaluation Study ........................................................................ 49

3-3. Conclusions, Recommendations and Lessons Learned .........................................................50 a) Outline ....................................................................................................................................................... 50 b) Output Sample : Conclusions, Recommendations and Lessons Learned ................................................... 51 c) Explanatory Notes...................................................................................................................................... 52

©FASID

3-3-1. Drawing Conclusions.............................................................................................................. 52 3-3-2. Drawing Recommendations .................................................................................................. 52 3-3-3. Drawing Lessons Learned ..................................................................................................... 53 3-3-4. Preparing Presentation Materials ......................................................................................... 53

4. IMPROVEMENT PHASE .......................................................................................................................54

4-1. Feedback .................................................................................................................................54 a) Outline ....................................................................................................................................................... 54 b) Output Sample: Commitment to Improvement .......................................................................................... 55 c) Explanatory Notes...................................................................................................................................... 56 4-1-1. Commitment of Improvement................................................................................................ 56 4-1-2. A Participatory Approach in Making the Commitment of Improvement ........................... 57

4-2. Improvement Planning .............................................................................................................58 a) Outline ....................................................................................................................................................... 58 b) Output Sample: Improved Log-frame ........................................................................................................ 59 c) Explanatory Notes...................................................................................................................................... 61 4-2-1. Cases Where an Improvement of the Log-frame is Not Necessary ................................ 61 4-2-2. Workshop for Improving the Log-frame ............................................................................... 61 4-2-3. Ex-ante Evaluation of the Improved Log-frame .................................................................. 62

4-3. Improvement Implementation .................................................................................................63 a) Outline ....................................................................................................................................................... 63 b) Output Sample: Improvement Cycle.......................................................................................................... 64 c) Explanatory Notes...................................................................................................................................... 65 4-3-1. Progress Monitoring ............................................................................................................... 65 4-3-2.Monitoring System ................................................................................................................... 66 4-3-3. A Periodic Evaluation for Further Improvement .................................................................. 67

5. EPILOGUE: EVERLASTING CYCLE OF IMPROVEMENT ........................................................................68

©FASID

１．Introduction

©FASID

1

0-0. From “Evaluation” to “Improvement” By definition, evaluation is to ascertain the achievement of a project or a program (i.e. a group of projects or activities) from a particular valuating point of view. Project cycle has long been considered in the general concepts of project management as follows.

Planning

①

(Plan)

②

③ Evaluation

Implementation

(See)

(Do)

The project cycle begins with making a plan, and is followed by the implementation process to carry out the contents of the plan. In following, evaluation is conducted by comparing the results of the project with the original plan. The evaluation is connected to the improvement of plan making in the next cycle. The project cycle works best when a well-documented original plan is prepared at the planning stage. This condition is, however, difficult to be realized in reality. Moreover, “policy evaluation” or “program evaluation”, which evaluates a group of projects with similar overall goals, is one of the most popular but challenging issues in evaluation studies. However, there is not any standardized method to undertake it. In both cases, the critical point is how to make the proxy plan to be compared with the results. Thus, it calls for a new approach that combines the various projects (or activities) and integrates them into a body of a program that can be evaluated in a logical manner. In this case, the cycle is not commenced from the planning stage, but rather it starts from the evaluation process to improve the plan. This is called “the cycle of improvement.”

③

Cycle of Improvement

Planning (Plan)

②

④ Evaluation

Implementation

(See)

(Do)

① Integration (Combine)

Program A

©FASID

Program B

Program C

2

0-1. About LEAD LEAD stands for Log-frame Evaluation Application Design. This method is based on the PCM (Project Cycle Management) evaluation method that is the de-facto standardized method for evaluating international development projects of Japan and other donors. LEAD employs its functions to be applicable for the program-level evaluation.

This method is applicable for the following needs.

-

To evaluate various types of projects implemented independently, but having similar overall goals as a comprehensive program.

-

To evaluate multiple projects with similar overall goals conducted by various agencies as a comprehensive program.

-

To link the results of evaluation with improvement of management.

©FASID

3

0-2. Contents of LEAD LEAD consists of several steps in four phases as follows.

Phase

1. Initiation Phase

Step

1-1. Evaluation Scope Planning

Issues

to

be

addressed For

what,

and

to

what extent will it be evaluated?

2-1. Hierarchy Tree Development 2. Preparation Phase

2-2. Logic Tree Development 2-3. Projects Mapping 2-4. Log-frame Formulation

What is the structure and content of the target program?

3-1. Evaluation Study Design 3-2. Evaluation Study Implementation 3. Evaluation Phase

3-3. Conclusions, Recommendations and

What are the results of the evaluation?

Lessons Learned

4. Improvement Phase

4-1. Feedback 4-2. Improvement Planning 4-3. Improvement Implementation

What should be improved? How is it to be improved?

Each step produces output as follows.

©FASID

4

Road Map：LEAD Method, Process and Outputs

LEAD Steps 1-1 Scope of Evaluation

2-1 Hierarchy Tree

2-2 Logic Tree

2-3 Projects Mapping

2-4 Log-frame

3-1 Evaluation Design

Outputs

Scope of Evaluation

LEAD Hierarchy Tree

LEAD Logic Tree

Projects Map

LEAD Log-frame

Evaluation Grid

3-2 Implementation of Evaluation Research

3-3 Conclusion, Recommendation and Lessons Learned

4-1 Feedback

4-2 Planning of Improvement with Log-frame

Evaluation Result

Conclusion, Recommendation and Lessons Learned

Commitment of Improvement

Improved Log-frame

Action Plan for Implementation

Continued Improvement

©FASID

4-3 Implementation of

Based on the Cycle of

Improvement Plan with Log-frame

Improvement

5

0-3. Structure of the Users Guide The LEAD Users Guide explains each step in three parts, those of “Outline”, “Output Sample” and “Explanatory Notes”. a) “Outline” clarifies the main parts: objectives, procedures and output, on one page. b) “Output Sample” shows an example of output from an imaginary case. c) “Explanatory Notes” covers important points in a more detailed explanation.

0-4. Case This guide employs an imaginary case as follows, and to be used for “Output Samples”.

Case: Evaluation of Assistance of Country J for Improvement of Second-Grade Education in the Republic of Amelia. The Republic of Amelia is located on the west coast of the African continent. Country J has been assisting the development of Amelia since its independence in the 1960s, when Amelia had been considered as one of the least developed countries. Recently, Amelia has been witnessing the significant growth of agriculture as the core sector of the national economy, with the exports of its agricultural products having been increasing., This has now drawn Amelia away from the group of poorest countries. Along with national economic growth, however, the economic disparity between the rich and the poor has been aggravated, and the trend seems set to continue without any intervention. As one of the main factors for the disparity, the educational disparity is considered the most serious and fundamental factor, according to the results of some social studies that have been conducted. Although more than 90% of Amelians have graduated from the elementary schools on average, the secondary school education is not well extended yet in Amelia since the graduation rate against the total population is still less than 40% on average. In order to improve this situation, Country J has been implementing three projects and a scholarship program in the capital city of Amelia for a decade.

List of Project Name (Department of MOFA of Country J) ・Rehabilitation Project of Secondary School Facilities.

（Dept. of Grant Aid） ・Training Project of Secondary School Teachers

（Dept of Technical Assistance） ・Development Project of Secondary Education Curriculum

©FASID

6

（Dept of Technical Assistance） ・Scholarship Program for Secondary Education

（Dept of International Communication）

These projects are conducted independently by various departments of the Ministry of Foreign Affairs of Country J, with no comprehensive evaluation yet being carried out. In order to improve effectiveness and efficiency of country J’s assistance for Amelia, MOFA calls for a comprehensive evaluation from a broader point of view. In addition, Country M, one of the other main donors in this field, has been conducting a “Project for Capacity Building of Education Administrators”. In the comprehensive evaluation, this project needs to be considered as one of the influential factors.

©FASID

7

２．LEAD in Practice

©FASID

8

1. Initiation Phase

1-1. Evaluation Scope Planning a) Outline

Objective:

To comprehend the needs of users (or clients) of evaluation in order to clarify the purpose and the scope of evaluation Procedures:

1.

Clarify the clients for whom evaluation is conducted, to whom the results will be fed back. (Who will be the user of evaluation?)

2.

Set the purpose of the evaluation based on the needs of the evaluation.

3.

Determine the scope (area) of the evaluation in order to deal with the needs.

4.

Consider the main stakeholders, such as evaluators, clients, supporters, etc.

5.

Consider the documents needed for the evaluation research.

6.

Put the above into a “Scope of Evaluation” that will be shared with stakeholders.

Output:

Scope of Evaluation

©FASID

9

b) Output Sample: Scope of Evaluation

Scope of Evaluation Evaluation Purpose

To evaluate assistance of Country J to Amelia’s secondary

(What is the purpose of

education from a broader point of view, in order to redesign the

evaluation?)

coordination of projects into an effective and efficient program.

Projects and Programs Concerned

・ Rehabilitation Project of Secondary School Facilities.

(What will be evaluated?)

・ Development Project of Secondary Education Curriculum

・ Training Project of Secondary School Teachers. ・ Scholarship Program for Secondary Education

Evaluation Criteria

・ Effectiveness (including impact)

(What viewpoints or criteria will be

・ Efficiency Etc.

focused in evaluation?)

Needs of Clients and Users (Who will be the direct

・ Management officers of Country J’s aid （Needs: to improve the assistance program of secondary education in Amelia.

beneficiaries of evaluation?)

Stakeholders/ Supporters (Information sources?)

Evaluation Team (Who conducts the evaluation?)

・ ・ ・ ・

Management officers of Country J’s aid Project staffs of concerned projects Relevant agencies of secondary education in Amelia Ministry of Foreign Affairs of Amelia, etc.

・ Ms. A (Leader) ・ Mr. B (Education expert) ・ Mr. C (Coordinator)

Schedule of Evaluation

Period

(How long is evaluation conducted

Progress Report: Monthly (to clients)

for?)

Final Report

Reporting and Feedback

In addition to monthly reporting, the final report will be prepared

(How will the results be fed back?)

and submitted to stakeholders, including the clients.

Post-evaluation commitment

By reviewing the evaluation results, Country J will draft a

(How will the evaluation be

Amelia, which will be implemented after reaching a consensus

utilized?)

with relevant organizations in Amelia.

Budget

No more than 30 thousand USD.

©FASID

: June to December, year 20xx. : December, year 20xx

redesigned assistance program for secondary education in

10

c) Explanatory Notes: 1-1-1.Evaluation for Whom? The implementation of evaluation also requires the concept of result-based management. Evaluation needs to achieve expected results, which should be clarified prior to the commencement of the evaluation, by asking “what is the purpose of the evaluation?”, or “whose needs does this evaluation expect to satisfy?”. Without clarification of such basic issues, the evaluation tends to be “evaluation just for evaluation sake”, and which is pointless. Evaluation is conducted because of a context in which there are stakeholders who need the evaluation to be undertaken for specific reasons. The evaluator should comprehend the needs of the clients or the users of the evaluation results, in order to conduct effective and efficient evaluation.

1-1-2. Preparing “Scope of Evaluation” Before beginning an evaluation, a Scope of Evaluation needs to be prepared. This is not only for the evaluator who confirms the scope of works, but also for stakeholders (including clients or users) in order to understand the evaluation’s expected outcomes and limitations. By clarifying this in the printed document, the understanding of the evaluation can be accurate. When it is necessary to change the contents of the evaluation because of, for instance, a lack of information sources, the change can be properly done through the Scope of Evaluation, which works as a compass. Sharing the Scope of Evaluation among stakeholders enables more support and collaboration for the evaluation to be obtained. It is important to note that the shown Scope of Evaluation is just one example template. The evaluators should configure their own Scope of Evaluation by adding or deleting components according to the needs of their specific evaluation. Any other topics that need to be evaluated should be clarified in detail and confirmed with the stakeholders.

1-1-3. Setting Evaluation Criteria The evaluation criteria are particular points of view with which an evaluation is conducted. DAC (Development Assistance Committee) under the OECD recommends five criteria for the evaluation of international cooperation projects/ programs. Many international cooperation agencies employ all or some of the five criteria for their projects and programs. ¾

Efficiency.................................. Project has produced outputs with least inputs?

¾

Effectiveness ............................ Project effectively achieved the project purpose?

¾

Impact ...................................... Project accrued positive or negative impacts?

¾

Relevance ................................. Project is still relevant to the needs of stakeholders?

¾

Sustainability........................... Project’s outcomes can be sustained?

©FASID

11

There may be other criteria such as “urgency”, “equity as a public project”, “coherence with other projects”, “validity and coverage of beneficiaries”, etc. The evaluation criteria can be different to meet the purpose or the needs of evaluation. Evaluation criteria is selected (even just tentatively) and clarified in the Scope of Evaluation, which should be presented to the stakeholders, including the clients, before the start of the evaluation. In reality, the greater the number of evaluation criteria, the more time and cost is expended. Including all of the evaluation criteria is wasteful, not only in the matter of efficiency, but also in the quality, since analysis with too many tasks tends to be flawed. On the other hand, selecting evaluation criteria only by the consideration of time and cost is not appropriate in order to satisfy the needs of evaluation. In practice, it is a matter of balancing the evaluation needs with the available resources.

1-1-4. Cases which can not be Applied to Theory-based Evaluation like LEAD There are some needs that can not be satisfied by LEAD. For example, the need to evaluate a model project for poverty alleviation implemented in several places all over the world, or the need to evaluate the several impacts which road construction would accrue, are not very applicable to LEAD. LEAD is relying on the program theory (i.e. Logic Tree) that utilizes the logical relationship between “Means” and “Ends”. If the “Means-Ends” relationship can not be developed with activities of target projects (or programs), then evaluation by LEAD can not be proceeded. A model project of poverty alleviation that is implemented at dispersed places can not be “combined” into a single logical diagram (a so called “Logic Tree”), so it is impossible to apply a theory-based evaluation method like LEAD. With a multiple impacts evaluation on road construction, applying LEAD is not recommended since it is not necessary to form a complex logical model for evaluation. Although it is true that advantages of LEAD are not fully mobilized in these cases, this Users Guide covers various topics that may be very useful to all types of evaluation.

1-1-5. Good Relationship with Stakeholders In evaluation, establishment of a good relationship with stakeholders is a very important issue. It is the project (or program) implementer who will receive and turn recommendations from evaluation into actions. The recommendations will not be implemented when the project implementers have the feeling of resistance towards the evaluator, much like the relationship between the criminal and the judge or between the artist and the critique.

Even in collecting the information, if the evaluator’s

attitude is over critical or judgmental, the evaluator will not be able to obtain the real information from stakeholders, especially the implementers. The collaboration of stakeholders, including the implementer, is indispensable for conducting LEAD, which needs the formation of a logical diagram of target projects (or programs) at the preparation

©FASID

12

phase with these stakeholders. Thus, LEAD is a cooperative evaluation method rather than a judgmental or criticizing one. In other words, LEAD is “formative rather than summative.” The evaluator must fully understand this characteristic of LEAD, and they must make the best effort possible for forming a good relationship with all the stakeholders. It is, however, a matter of balance. An evaluator that accepts all of the demands and information from the stakeholders will lose the credibility in the end. The evaluator must be independent and preserve the mind of objectivity and fairness.

1-1-6. LEAD as a Tool of Capacity Building In case that the evaluation is a program level evaluation concerned with several projects by multiple agencies, it is necessary to clarify the coordinating agency who realises the recommendations from the evaluation. If the coordinating agency is the local government, the evaluation team should contain officers from the local government from the commencement of the evaluation. The participation of the officers should be integrated into the evaluation, i.e. it is not just by the name of the team or the attendance of periodical meetings. The evaluation should be designed in accordance with the understanding of the participating officers. It is no one but the participating officers who actually improve the projects with their coordinating capabilities. Coordination of multiple projects needs the competent capacity of understanding and analysing of the projects. Thus, the participation in LEAD is very effective capacity building for the coordinating officers. Planning and implementing of LEAD are led actually by the evaluator (or the evaluation team), but involvement of officers working for the implementing agency or a coordinating agency will enable the recommendations of the LEAD evaluation to be more feasible and effective. By participating in LEAD from the preparation phase, participating officers will learn the strengths and weaknesses of the target projects (programs) at full length. Thus, the recommendations and lessons drawn from such a participatory evaluation will be more concrete for realisation.

©FASID

13

2. Preparation Phase 2-1. Hierarchy Tree Development a) Outline

Objective:

To describe the whole picture of target projects (including activities) by stratifying them into a hierarchical tree.

Procedures:

1. List the name of all projects (or programs) to be evaluated. 2. List all of the main activities of the projects. 3. Arrange (or stratify) the projects into a hierarchical tree according to the way of the institutional arrangement or as expressed in reports (e.g. by implementing agencies or administrative departments). 4. Confirm the Hierarchy Tree with all stakeholders to form a consensus of understanding.

Output:

Hierarchy Tree

©FASID

14

b) Output Sample: Hierarchy Tree

Hierarchy Tree

Agency J's assistance

Activities

Dept. of Grand Aid

Projects

Rehabilitation Project of Secondary School Facilities

Responsible Department

Construction of a building

Provision of Toilet

Provision of a water tank

Trainig on maintenance and management of facilities

Training Project of Secondary School Teachers

Procurement of teaching materials

Issue a news letter on female education

Research on the promotion of femnale educaiton

Education for female education

Implementation of a training program

Training for Teachers

Training female teachers

Scholarship Program for Secondary Education

Development Project of Secondary Education Curriculum

Female education support

・Omitted・

Public relations for female educaiton

©FASID

Dept. of International Communication

Dept. of Teachnical Assistance

Training for teachers

Planning of training program for teachers

・Omitted・

Development of a guidebook on teaching methods

Curriculum Development

・Omitted・

Development of Curriculum & Syllabus

Improvement of textbook program

Development of Teaching Materials

Development of a guidebook on teaching materials

Organization of a committee of textbook development

・Omitted・

Provision of free textbooks

・Omitted・

15

Provision of scholarship

Development & maintenance of database

Developme of manual on financi managemen

c) Explanatory Notes: 2-1-1. Hierarchy Tree It is difficult to understand the whole picture of the projects to be evaluated when the numbers and areas of the projects are many. In some cases, the project name exists but the activities are not clear or not actually conducted. In other cases, the evaluation must cover some schemes like a scholarship program that does not include proactive activities or regulations like legal punishment to prevent crimes. The LEAD method needs to construct the logical “means-ends” relationship from these sorts of things before proceeding to the phase of evaluation study. It is, however, very difficult to form the logical relationship directly from these “raw” materials, and it is somewhat dangerous in the sense that some important components may be lost when forming the rigorous logic at the beginning. Thus, the preliminary stage is just to construct a diagram by following the existing institutional approach, or the existing hierarchical order. The diagram is called “Hierarchy Tree ”. Building a Hierarchy Tree is considered the preparation for the next stage of building a Logic Tree. For stakeholders, a Hierarchy Tree would be more familiar than a Logic Tree and a Log-frame, therefore the Hierarchy Tree would help their future understanding of the Logic Tree and Log-frame that are developed later.

2-1-2. Building a Hierarchy Tree It is recommended to build a Hierarchy Tree cooperatively with the main stakeholders based on relevant documents and opinions of other stakeholders. It is begun with establishment of a database into a table with a spread sheet software such as “Excel”. Database contains “Name of agency”, “Name of department”, “Name of project”, “Main contents”, “Budget”, and so on. The data is put in order of each column in a table, because it is easy to change it into a hierarchy tree. While a table-style is handful as a database, the tree-style like “a hierarchy tree” is more powerful in presentation. To draw the tree, it is recommended to use the drawing functions in a spreadsheet software program, or in a drawing software program such as “Inspiration”. A more effective method of building a Hierarchy Tree is undertaking a participatory hands-on workshop using paper cards on which the name of the projects and activities are described. The Hierarchy Tree is then constructed by arranging these cards on a large sheet of paper on a wall. Such a participatory workshop is helpful in building a consensus among the stakeholders.

2-1-3. Tips for Building a Hierarchy Tree As indicated in the name, a Hierarchy Tree stratifies the projects to be An increase in agricultural production in the Country of C

evaluated according to the way of the institutional arrangement. One of the most common forms of arrangement is “by department” or “by type of scheme” (e.g. technical cooperation or grant aid). The hierarchy (or

Department A

Department B

stratification) can be made into multiple layers. For example, the first layer Program A

Program B

Program C

is “by country”, the second layer is “by organization”, the third layer is “by

©FASID

16

Program D

scheme”, the fourth layer is “by project name”, and the lowest layer is “by activities”. It is, however, not recommended to build a complex diagram that has too many layers. The purpose is to build the most recognizable diagram that can be understood easily by the main stakeholders. Thus, participation of the main stakeholders in building a Hierarchy Tree is important.

2-1-4. Coverage of a Hierarchy Tree When there is any project or activity other than the target projects that influence the common objectives, it is recommended to include them into a Hierarchy Tree. This is because such information will be a useful reference when building a Logic Tree in the next stage, and when considering the important assumptions in the Log-frame. In our case, the project conducted by Country M is considered as one of the important external components, influencing the achievement of the outcome of the target projects. Therefore, the Hierarchy Tree includes the Country M’s project within it.

2-1-5. The project without Activities and the Activity without the Project Name In developing a Hierarchy Tree, you may find a project that exists on paper but no actions are taken at all, or you may find some activities that are not formed as a project. Moreover, there is an institutional regulation, such as law with punishment, or a scheme like a scholarship program that does not need activities, but that does influence the objectives. In making a Hierarchy Tree, such things must be put in just as they are. For example, activities without any project name are put in without a name. In this stage of building a Hierarchy Tree, it is not necessary to make a logical arrangement. Instead, one should try to express the structure of subjects in a naturally and institutionally accepted way. It is the preliminary stage, and so the Hierarchy Tree can be an unorganized one as long as it reflects the reality.

Describe it as it is!

©FASID

17

2. Preparation Phase

2-2. Logic Tree Development a) Outline

Objective:

To clarify the logical structure of the target projects as a whole program by arranging their activities along the relationship of “Means-Ends”.

Procedures:

1. List all activities from the Hierarchy Tree and place them in the bottom line. 2. Set an Overall Goal (a policy or program level goal) by referring to documents and opinions of stakeholders. 3. Consider Intermediate Objectives which bridge between activities and the Overall Goal in a logical sequence. 4. Put activities with the same (or very similar) Intermediate Objectives into groups. 5. Iterate the above processes to complete the Logic Tree in which “Activities – Intermediate Objectives” and “Intermediate Objectives – Overall Goal” can be expressed by the logical relationship of “Means-Ends”. 6. Confirm the Logic Tree with the consensus of stakeholders.

Output:

Logic Tree

©FASID

18

b) Output Sample: “Logic Tree”

Logic Tree J's Assistance

Enrollment of middle school education is improved in Amelia

Overall Goal

Intermediate Objectives

Activities

Running regular classes

Construction Provision of Provision of Training on of building bathroom a water tank maintenance

©FASID

Omitted

・Omitted・

Procurement of teaching material

Improve a middle school curriculum

Modification Making a Making a Development of curriculum guidebook on guidebook of of teaching and syllabus teaching teaching materials methods material development

Teacher's teaching methods improve

Organizaiton of Committee Provision of Training for Training for Planning of free for the teachers top managers a training program for improvement textbooks teachers of textbooks

・Omitted・

Acceptance of trainees

Students' morale toward schooling increases

Provision of Provision of ・Omitted・ free scholarship textbooks

Equal educational opportunity for both boys and girls

Support for female education

19

Training for female education

・Omitted・

c) Explanatory Notes 2-2-1. Logic Tree The Logic Tree expresses the logical structure of projects and activities to be incorporated into a proxy program by reconstructing the logics of the components. Establishing a Logic Tree is necessary to conduct a comprehensive evaluation. With the Logic Tree the following aspects can be evaluated.

-

Holistic effects of various projects as a comprehensive program

-

Relationships among projects and activities

-

Single effects of projects, including the indirect effects

It is the evaluator who is responsible for developing a Logic Tree but the building process must be participatory with stakeholders, as with the development of a Hierarchy Tree.

2-2-2. Setting Overall Goal Overall Goal is the common goal for which all related projects and activities are conducted. It is set at the top of a Logic Tree, so that there is only one Overall Goal. If there is no Overall Goal identified in advance, the evaluator sets a new Overall Goal. In setting this Goal , care should be taken when deciding how to express it. It is NOT the summary of what the projects (or program) will do, but it expresses a positive change or impact on the beneficiaries that the projects (or program) are intended to accrue. Bad and good examples of an Overall Goal in evaluating traffic-related projects are:

(Bad Example) ×

Traffic infrastructure is upgraded.

(Good Example) ○

Traffic congestion is reduced.

Overall Goal

In this case, the ’bad’ example is just describing what the projects do, and does not say what the purpose of the traffic infrastructure development is for. Thus, it should be written in such a way as that stated in the ‘good’ example, as this describes the outcome (or impact) of the development projects. It can be said that the traffic infrastructure is not only for vehicle traffic, but also for pedestrians as well. In this case, the expression of the Overall Goal could be “The city becomes a place where the citizens feel able to be transported more easily”. This expression is a wider concept compared with the original one. If an expression is too vague, some definitions may be added to the Overall Goal. In practice, some clues for setting the Overall Goal can be found within the main goals of each project, which is set above the direct project purpose. Referring to the needs and the purpose of

©FASID

20

evaluation, or meeting with stakeholders such as the program officers and representative beneficiaries, will also give good clues. It is important to determine the Overall Goal by reaching a consensus among stakeholders as explained in the previous steps.

2-2-3. “Means-Ends” Relationship The Logic Tree consists of components such as the Overall Goal, Intermediate Objectives and Activities, among which the relationships are constructed in the logics of “Means-Ends”. As stated below, an Activity “to teach mathematics to a student” is one of the Means for the End (objective) that “the student can calculate”. For achieving this objective, there may be other means, such as “to provide a textbook of mathematics”. Moreover, the statement that “the student can calculate” can be one of the means for a higher objective such as “the student can promote to the next grade”.

Like this, a Logic Tree can be expanded flexibly.

The student can promote to the next

End

The student can calculate

End

↑

↑

Means

Means To teach mathematics

To provide a textbook

The above Logic Tree may not be accepted by some readers. For example, it can be criticized that “to provide a textbook” is not a direct means and that, instead, this should be “the student to study by himself”. In practice, all Logic Trees can be true for the developers who create the tree with their own logic. In other words, there is no “perfect” Logic Tree that everyone in the world would agree to as being the ultimate model. A Logic Tree should of course be developed in an objective and rigorous way. It is, however, impossible to form such a perfect model, because human beings cannot escape from subjectivity. The state of objectivity results from a reorientation of the subjective opinions of human –beings, in reality. In LEAD, the Logic Tree is a basis of following the evaluation, so it is necessary to make a consensus on the Logic Tree with the stakeholders, including the groups for whom the recommendations will be made. Thus, the evaluator should be good at not only creating logic, but also dealing with the likely diverse opinions of stakeholders in order to reach a consensus.

©FASID

21

2-2-4. Tips for Setting Intermediate Objectives The Logic Tree consists of three layers such as the Overall Goal, Intermediate Objectives and Activities. In practice, however, it is difficult to set Intermediate Objectives to be filled between the Overall Goal and Activities at first. The following procedures are recommended. For projects that have their own project purposes, just assume the project purposes as candidates for the Intermediate Objectives, rather than decomposing them into activities. After setting the tentative Intermediate Objectives directly from the project purpose, extract unnecessary activities which can not be connected to Intermediate Objectives, and consider whether their purposes can become Intermediate Objectives or not. If there are just activities or schemes without clear-cut purposes, consider what they are intended to achieve, then set the Intermediate Objectives. When all activities are set under Intermediate Objectives, check if there are any Intermediate Objectives that are the same, or very similar. If you find any, then group them under a common Intermediate Objective. Where there are a lot of projects and activities, input the information into a spreadsheet program such as Excel. Put in the project names and main activities, and make a rough categorization of the activities into several groups. By using the sorting function of the software, put the activities into groups. Then consider the common objective of each group as Intermediate Objectives.

2-2-5. Theory-based Evaluation by Building a Logic Tree The establishing process of a Logic Tree is considered as evaluation, since this process can distinguish unnecessary activities and illogical projects with unclear purposes. In evaluation, it is very important to know which activities (or projects) are not necessary to achieve the Overall Goal. You may be faced with an extreme situation in which you can not construct a Logic Tree since the purposes of the activities are too diverse to be considered as a program. Continuing to the next step by setting an irrational Logic Tree is a waste of time and resources. In this case, the evaluator can recommend fast tracking the reforming of the program, jumping to the improvement phase to start reforming the program as a new plan, rather than doing a wasteful evaluation study with such an illogical Logic Tree.

Initiation Phase Preparation Phase

Scope of Evaluation

Hierarchy Tree Logic Tree Log-frame

Research and Analysis Phase

No logic at all…. Need for re-planning.

Evaluation Study Design Evaluation Study Implementation Conclusion, Recommendations and Lessons

Improvement Phase ©FASID

Feedback Improvement Planning Implementation of Improvement Plan

22

2. Preparation Phase 2-3. Projects Mapping a) Outline

Objective:

To make geographical map that shows areas where are covered by each project. Procedures:

1. Collect information in terms of areas where activities are implemented or positive impact is expected to be reached by each project. 2. Draw above areas on the map by different colour. 3. Confirm the projects Map with the consensus of stakeholders.

Output:

Projects Map

©FASID

23

2-4. Log-frame Formation a) Outline

Objective:

To form the Log-frame which describes the main components of the program based on the Logic Tree. Procedures:

1. Transfer the components in the Logic Tree into corresponding cells of “Narrative Summary” of the Log-frame. 2. Set external factors influencing the program into “Important Assumptions” of the Log-frame. 3. Set quantifiable indicators for Overall Goal and Intermediate Objectives. 4. Set “the Means of Indicators” 5. Set “Inputs” for enabling Activities 6. Confirm the Log-frame with stakeholders

Output:

Log-frame (Logical Framework)

©FASID

24

b) Output Sample: Log-frame Log-frame of Country J’s assistance for secondary education for Amelia Target Area: The Capital of Amelia Narrative Summary (of Objectively Verifiable Program/Policy) Indicators Super Goal： ・ ・

Poverty in the capital of Amelia is reduced.

Overall Goal： ・ ・

The secondary school enrollment in the capital of Amelia is improved. ・

Intermediate Objectives： 1.

2. 3. 4. 5.

The number of school facilities with sufficient educational function is increased. School curriculum is improved. Teachers’ skills and knowledge are improved. Students’ morale toward schooling increases. Accessibility of girls’ schooling is improved.

Means of Verification

The population below the poverty line is less than 10% in the capital of Amelia in the year XXXX.

・

The rate of students receiving secondary education becomes more than 80% in the city in the year XXXX. The ratio of graduation/entrance of secondary students becomes more than 1 to 10 in the year XXXX.

・

・

Major Important Assumptions

National statistics from the Ministry of Social and Health. UNDP Statistics Year Book. National statistics on education derived from the Ministry of Education.

・

・

1-1. More than 90% of schools in the city meet the national school facility standard level A.

1-1

Results of audit of the national standards of school facility.

2-1 An improved school curriculum is endorsed by the educational committee and the third party in the year XXXX. 2-2 More than 80% of schools in the city introduce an improved curriculum in the year XXXX.

2-1

Minutes of the Boards of Education.

2-2

Statistics on education (the Ministry of Education).

3-1

Statistics on education (the Ministry of Education).

4-1

Results of Questionnaire and Survey.

5-1

Results of Questionnaire and Survey.

・ ・

3-1 More than 70% of all teachers have a license with which they can teach the curriculum in the year XXXX. 4-1 More than 90% of all students in the district enjoy schooling in the year XXXX.

Government policies on secondary education are kept unchanged. Other planned poverty reduction programs are implemented.

Trained teachers continue working for the school. School buildings constructed by the project are adequately maintained.

5-1 Less than 10% of all women feel that a woman has a limited opportunity in the year XXXX.

Activities：

Inputs：

1. Provide school facility 1-1. Set up national standards for school buildings. Construct school facility 1-2. Equip the facility with bathroom 1-3. Equip the facility with a water tank 1-4. Training on how to manage and maintain the facility 1-5. Procure teaching materials 1-6. Audit the national facility standards

Project Name ・ Secondary school reconstruction project ・ Secondary school teacher training project ・ Secondary school curriculum development project ・ Secondary school scholarship program

・

Total Amounts: ooooooooo yen

Pre-conditions：

Major Inputs: ○○、

・

Social condition surrounding students do not worsen. M’s projects to improve secondary school education are implemented.

、××

Stakeholders: Amelia: the Ministry of Education, Ministry of Social Insurance, Ministry ofJustice Local Public Agency: Department of Secondary Education

(The rest is omitted)

©FASID

25

c) Explanatory Notes: 2-4-1. Log-frame Log-frame (or Logical framework) was originated in a management tool of USAID in order to express the summary of a development project. Log-frame consists of the main components of a project such as project purpose, outputs, activities inputs, quantifiable indicators, external risks, etc. In PCM, Log-frame is introduced as PDM (Project Design Matrix), which is created through a participatory workshop with stakeholders. PDM is utilized in monitoring and evaluation in a project cycle. Log-frame in LEAD is also a matrix table containing an overall goal, expected objectives, activities, inputs, indicators, risks, etc., of the program to be evaluated. It is formed to confirm the structure of the program prior to designing an evaluation study. The form of a Log-frame is depicted as follows.

Program Title： Date：

Target Area： Narrative Summary of

the

Program

Objectively

Means of

Important

Verifiable

Verification

Assumptions

Indicators Overall Goal： Objective

that

the

program

should

achieve within the program period

Indicators

for

Data sources from

Conditions

measuring the level

which indicators are

sustainability

of

derived

the

achievement

of

Overall Goal

effects.

Indicators

Objectives which should be achieved in

measuring the level

Data sources from

Conditions

order to achieve the Overall Goal

of

which indicators are

Overall

derived

besides

Intermediate

of

program

Intermediate Objectives:

achievement

for

for of

Objectives.

for Goal

that

of

Intermediate Objectives

Activities：

Inputs：

Summary of Actions to achieve the

Personnel, materials, equipment, facilities

Conditions

Intermediate Objectives

and funds required by the project

Intermediate

for

Objectives besides

that

of

Activities. Pre-Condition Conditions

that

must be fulfilled before

the

program starts

©FASID

26

2-4-2.Special features of Log-frame for LEAD The Log-frame of LEAD omits the top row of the matrix, compared with the general Log-frame used in project management. LEAD is developed for a program or policy level evaluation method, so that the core objective of the

Narrative Summary of Program/Polic y

than the Overall Goal, “Super Goal” should be inserted on the top row of the

Super Goal: Poverty reduction in Amelia is promoted Overall Goals:

matrix. For an example from the Amelia case, the Overall Goal of the

Intermediate

program (a group of projects) is “to increase the number of secondary

Objectives:

program is the Overall Goal level which is higher than the project purpose level in the general Log-frame. In cases where there is a higher objective

Objectively Verifiable Indicators

education graduates”. In the Amelia case, this Overall Goal is one of several program goals of the poverty alleviation policy, and the Super Goal is “to

Activities:

Inputs：

reduce the poverty population in Amelia”, which is located at the top of the matrix. On the other hand, there can be a case where the Logic Tree has several layers from the Overall goal to Activities. In this case, the new row (e.g. “Outputs”) can be added between Intermediate Objectives and Activities. The Log-frame can be modified flexibly for the needs of the evaluation. It is, however, recommended to avoid making too complex a Log-frame, that would reduce the understanding of the stakeholders and that would be difficult to be used in practice.

2-4-3. Log-frame as a Tool for Management The Log-frame contains important information of the program about its objectives and indicators, and which becomes essential for evaluation on the achievement of the program. The distinct parts in the Narrative Summary of the Log-frame, which segregate the program into “Overall Goal”, “Intermediate Objectives”, “Activities”, “Inputs” and “Important Assumptions”, enable the evaluator to pursue detailed factor analysis on the results of the achievement. While the Log-frame is used for evaluation, it is the Log-frame’s strength that means it can be used for program management in planning and implementation, too. PCM (Project Cycle Management) applies PDM (Project Design Matrix = Log-frame) in all the phases of project management, such as planning, implementation, and evaluation. By forming a Log-frame in evaluation, it can be used as a basis for the following phases of planning and implementation. Forming the Log-frame may be cumbersome if it is used only for evaluation. However, when considering that the evaluation within LEAD is a step for the next phase of improvement, the Log-frame formation is effective and efficient in the long-run. If the evaluation needs to be conducted just only for evaluation purposes, it is not necessary to use LEAD. LEAD is an evaluation method that is integrated intrinsically with the subsequent improvement of the program by using the Log-frame. Implementation with the Log-frame is discussed in section 4-3.

©FASID

27

2-4-4. Converting a Logic Tree into a Log-frame If a Logic Tree has been formed logically, it is technically easy to convert the Logic Tree into a Log-frame, since the Log-frame’s left column “Narrative Summary” consists of “Overall Goal”, “Intermediate Objectives” and "Activities” that are connected in a logical relationship of “Means-Ends”, just like the Logic Tree.

Narrative Summary (of Program/Poli cy)

Objectively Verifiable Indicators

Means of Verificatio n

Major Important Assumptions

Overall Goals (Program Goals)

Logic Tree

Intermediate Objectives Overall Goal

Objective A

Activities

・

Activities

（Sub-progr am Goals)）

Activities：

Objective B

Activities

Inputs：

Pre-conditio ns

Activities

Overall Goal (i.e. Program Goal) This is the goal (purpose) of the program (or a group of projects) to be evaluated, and it is recommended to set only one Overall Goal. It should describe, in a sentence, the concrete benefits or impacts given by the program to the beneficiaries (i.e. the target group). It is not a description of what the program will be doing, instead it should clarify what the program intends to achieve. What to do

（Bad Example）×

Overall Goal：Technical assistance for Rice production in village “A“ is conducted What to achieve

（Good Example）○

©FASID

Overall Goal：Rice production in village “A“ is increased.

28

・

Intermediate Objectives（i.e. Sub-program Goal） Intermediate Objectives are intermediate goals to achieve the Overall Goal. For one Overall Goal, several Intermediate Objectives are set. They are described in a single sentence, as with the Overall Goal. They are results from Activities. In other words, Activities should not be confused with Intermediate Objectives. Action

（Bad Example）×

Intermediate Objective：Training for farmers are conducted.

Result

（Good Example）

・

○

Intermediate Objective：Farmers apply the new techniques.

Activities Activities are descriptions of actions to achieve Intermediate Objectives. For one Intermediate Objective several Activities are set, which are described with a “to do” expression. It is recommended to put identical outline numbers (or reference codes) for activities.

Tips： In practice, it is recommended to set one Overall Goal, four-to-seven Intermediate Objectives, and around four Activities for each objective, to enable a better understanding. Putting codes to Intermediate Objectives and Activities will help further.

2-4-5. Setting Indicators and Means of Verification Objectively Verifiable Indicator (OVI) is a quantifiable degree to be achieved for each objective expressed in the Log-frame, such as the Overall Goal and Intermediate Objectives. For each objective, one or more OVI is set. The OVI is described in a sentence, which includes the following information.

Information

Example

・

When............................ “By September of year 2005”

・

Where........................... “In area A of country A”

・

Who/What .................... “Average income of farmers”

・

How much.................... “By 30% from the last year”

・

What to do ................... “To be increased.”

©FASID

By September 2005, 2005 年９月までに、A 国 A average income of farmers 地区の農民の平均所得が前 is increased by 30% from 年比で 30%増加する。 the last year.

29

By setting the OVI, vague expressions in the Narrative Summary in the Log-frame become clearer to enable an objective evaluation to be conducted. In the course of setting the OVI, contents of the Narrative Summary can be elaborated in its definition and modified if necessary. In practice, the following information will be useful in setting the OVI.

・

Planned indicators in existing policy papers or program documents.

・

Standards regulated in public (National or international standards set by the governments, UN, ISO, etc.).

・

Indicators used in similar projects.

・

Opinions of program officers, policy makers and other stakeholders.

・

Estimation from logical relationships expressed in the Log-frame.

Choosing the OVI requires utilising the following selection criteria. ・

“Objectivity” .......... It should be based on fact, and not influenced by personal beliefs or feelings.

・

“Direct”.................. The type of data should be exactly what should be measured. For example, the type of data for an objective ”Income from agriculture is increased” is “income from agriculture”. It is neither “agricultural production” nor “general income of households”.

・

“Adequate Level” .. The level of data should be exactly what should be measured. For example, the OVI of an objective “Agricultural technique is upgraded” is not “the number of training courses conducted”, since it is one of the activities for the objective. On the other hand, “the amount of agricultural product” is not the proper OVI, since it is the result of upgrading the agricultural techniques. Technically, there can never be any case where the same OVI is used in different levels of the Narrative Summary in the Log-frame.

・

“Coverage”............. The OVI should cover all of what should be measured. Sampling should be conducted without bias.

The cells in the right hand side of the OVI are for "Means of Verification", which indicate the source of information. Means of Verification states relevant documents, methods, sourcing organizations, locations and so on, in order to make the setting of indicators and their monitoring more efficient and effective.

When

considering

the

Means

of

Verification,

"availability",

"reliability"

and

"cost-performance" of information should be considered. When there is no available source of information, it is necessary to collect information by conducting researches by the evaluation team. It is, however, recommended to use the existing data as much as possible in practice, since information collecting itself consumes a lot of time and monetary resource. The selection of Means of Verification should proceed carefully, along with the selection of OVIs.

©FASID

30

2-4-6. Setting Important Assumptions "Important Assumptions" are the conditions which are necessary for the program achievement, but are not controllable by the program, and it is uncertain whether they will be realized or not. Important Assumptions are found from logics within the Narrative Summary of the Log-frame. As described below, Important Assumptions for "Activities" to "Intermediate Objectives" can be found by asking whether there is any condition to be realized besides that of the Activities for accomplishing the Intermediate Objectives.

Outcomes

Activities

Important Assumptions

For instance, the increase of agricultural production (as the Overall Goal) cannot be achieved only by the improvement of farmer’s skills (as an Intermediate Objective), as there also needs to be rain. In this case, "Amount of rainfall is enough" can be considered as one of the Important Assumptions. For another example, the upgrading of the teaching quality in a school cannot be achieved if the trained teachers leave the school, and therefore despite the fact that the training has been well conducted for the teachers. In this case, "The trained teachers continue to work in the school" can be thought of as one of the Important Assumptions. There are other issues such as economy, politics, environment, society, culture and so on, for considering Important Assumptions.

Important Assumption

Improved farmer’s skill

©FASID

31

When considering Important Assumptions, cross-cutting issues, which the OECD recommends, are of some help.

Cross-cutting Issues

Examples

・ Political ................................................. "The government keeps the policy of supporting secondary education." ・ Technical ............................................... "The electricity necessary for agriculture is provided." ・ Environment ......................................... "More than 1000mm of annual rainfall is available." ・ Social/ Cultural ..................................... "Not many villagers migrate away for seeking jobs." ・ Institutional .......................................... "Farmers' Cooperative Organizations continue to exist." ・ Economic ............................................... "Market price of rice is maintained."

Projects or activities that are included in the Logic Tree, but are not included in the Narrative Summary of the Log-frame, are also candidates for Important Assumptions. When there are other organization(s) to do projects influencing the program, the projects should be considered as one of the Important Assumptions. This will help in the later stages for analysing the attribution of the program. In the Amelia case, the project conducted by Country M can be considered as an Important Assumption. "Pre-conditions" are the conditions that should be realised before the commencement of the program. In practice, however, it is a rare case to need the pre-conditions because LEAD evaluation is conducted mostly for on-going programs.

2-4-7. Setting Inputs The "Inputs" cell consists mainly of human resources, facilities and materials, and other operating costs. If the inputs are diverse in various operating organizations, the inputs to be divided throughout the organizations should be so described. This will help the later analysis of the contribution level of each organization. In practice, the evaluator determines the method of setting the Inputs, but it is recommended to clarify at least the main components of inputs (e.g. types and amounts of human resources, facilities, materials and other costs) by each organization. When analysing the efficiency of the program, it is recommended to put these inputs into each group of Activities or to enumerate the inputs into monetary values.

©FASID

32

Well Done! You have successfully achieved the halfway point to the final goal!

Relax for a while......

©FASID

33

3. Evaluation Phase

3-1. Evaluation Study Design a) Outline

Objective:

To clarify what is necessary to be studied and how it is conducted, in order to achieve the evaluation purpose.

Procedures:

1. Reconfirm the evaluation purpose from Scope of Evaluation and Log-frame. 2. By defining the evaluation purpose, set evaluation criteria and evaluation questions. 3. Set study methods to answer the evaluation questions. 4. Compose the above information into a matrix named "Evaluation Grid", which should be shared with stakeholders.

Output:

Evaluation Grid

©FASID

34

b)

Output Sample: Evaluation Grid Example of Evaluation Grid

Evaluation Purpose

Evaluation Questions (Criteria)

Evaluation Questions (Level 1)

Evaluation Questions (Level 2)

Research Methods

Information Source

To conduct

1.

1-1.

1-1-1.

Analyse the past trends by

Statistical

comprehensive

Is the program

Is the Overall

Whether the rate of

using statistics on

Document

evaluation of

effective?

Program Objective

secondary education

secondary education in

published by

achieved?

enrolment in the capital

order to predict future

government.

city is increased up to

trends.

middle school education support

90%.

programs,

1-1-2.

Analyse the past trends by

Statistical

which are

Does ratio of graduation/

using statistics on

Document

currently

entrance increase up to

secondary education in

published by

implemented in

0.9 as planned?

order to predict future

government.

trends.

Results from

the capital of Amelia, to

sample survey.

modify these in order to make

1-2.

1-2-1.

them more

Is the achievement

To what extent J’s

Examine the association

Document on J’s

effective and

of overall

development assistance

between the enrolment rate

assistance in

efficient.

objectives

contributed to the

and J’s assistance by using

each school.

attributed to

increase of the education

regression model.

Country J’s

continuance rate?

assistance?

1-2-2.

Conduct questionnaire

Contract out the

Which programs are

among stakeholders

survey to a local

How much is its

significantly effective or

particularly on effective

consultant.

contribution?

ineffective?

programs, including

Conduct

questions on the reasons

interviews by the

why they think that is so.

evaluation team.

1-2-3.

Conduct qualitative

Program

To what degree was the

evaluation of the impacts of

evaluation report

contribution of

the Country M’s education

done by Country

assistance from other

project based on its

M.

donors or agencies?

evaluation report. Conduct questionnaire and interviews among stakeholders on other factors influencing the school enrolment.

Contract out the survey to a local consultant. Conduct interviews by the evaluation team.

©FASID

2.

2-1.

2-1-1.

Is the program

Is there any

Is there any redundant

efficient?

redundant or

program or project?

overlapping program?

Check with the Logic Tree

Logic Tree.

2-1-2.

Lay out an actual use of

Overall Inputs

Is there any program that

inputs to identify

Matrix.

has identical inputs?

redundancies.

35

2-2.

2-2-1.

Ask stakeholders to rank

Contract out the

Are there any inputs that

the order of utilization of

survey to a local

sufficiently utilized

have significantly low or

inputs and also ask for the

consultant.

and effectively

high level utilization?

transformed to

2-2-2.

reasons. Ask stakeholders to raise

Contract out the

outcomes?

How is the transformation

indicators to assess the

survey to a local

from inputs to outcomes?

contribution of inputs to

consultant.

outcomes, and analyse

Analyse the

them by running a

results of the

regression analysis.

survey by using

Are inputs

in-house staff.

©FASID

36

c) Explanatory Notes 3-1-1. Evaluation Purpose and Evaluation Criteria In order to make the evaluation more effective (i.e. to achieve the purpose) and more efficient (i.e. to consume the least cost), it is crucial to focus the evaluation purpose before designing the evaluation study.

LEAD requires the clarification of the evaluation purpose at the initiation phase (See 1-1

"Scope of Evaluation"), so the evaluation purpose is reconfirmed at this step. If necessary, the evaluation purpose can then be adjusted and defined more clearly. For evaluation of international cooperation projects, it has been recommended to consider the five evaluation criteria of efficiency, effectiveness, impact, relevance and sustainability. In practice, however, applying all five criteria simultaneously is not recommended when there is limited time and resources. There is also a risk that all of the five may be outside of the clients' interests, so that the evaluation would be useless in the end. With limited time and resources, the evaluator should consult with the clients or the users of the evaluation, and clarify their interests in the evaluation. For customisation, any basic improvements needed should be clarified at this stage, such as whether the client wants to improve the effectiveness or the efficiency of the program, to thereby alleviate the negative impacts, etc. Of course, there can be a desire to pursue the improvement of every aspect of the program, however such an ambitious (but ambiguous) comprehensive evaluation tends to be obligated with enormous amount of tasks, and with a tendency to end with unsatisfactory results. If there is a request for several evaluation criteria to be evaluated, then these should be prioritised by ranking them with the stakeholders. The designing of the Evaluation Grid can be commenced from the high ranked evaluation criteria.

3-1-2. Program Level Evaluation A project evaluation in general considers the whole effect of the project, while a program level evaluation, in which a program includes several projects, needs to have some evaluation added on the structure of the program and on the relationships among projects as sub-programs. Especially for LEAD, which composes the proxy-program containing projects as sub-programs to be evaluated, one of the most probable recommendations resulting from the LEAD evaluation is to re-construct the program by adjusting and coordinating existing projects. Thus, the question about which component (project) should be improved by adding, deleting and modifying is very crucial, so the evaluation on each component within the program structure is indispensable in any evaluation criteria selected.

©FASID

37

The Concept Map of Program Evaluation

Evaluation of Program

Overall Goal (Program Goal)

Sub-program

Sub-program

Sub-program

Sub-program

Goal 1

Goal 2

Goal 3

Goal ….n

Activities

Activities

Activities

Activities

Activities

Activities

Activities

Inputs

Inputs

Inputs

Inputs

Inputs

Inputs

Inputs

Evaluation of Program Structure

©FASID

38

Evaluation topics for each of the five evaluation criteria can be described as follows.

Five Evaluation Criteria in Program Level Evaluation Evaluation Criteria

Basic Approach

Advanced Approach

(As a whole)

(About the Structure)

Consider

Efficiency

the

Productivity

in

Consider adequacy in the use of the inputs

implementation. How efficient is

provided by different agencies. Are there

the manner in which the outputs

any

are produced by the inputs? Are

adequacy in the sequence of executed

any inputs wasted? What about

projects? Is there any efficiency-making

the

effort, e.g. to share the same input by

adequacy

considering

of

the

the

inputs,

outputs

and

redundant

inputs?

What

about

different agencies?

outcomes achieved?

Effectiveness

Consider the achievement level of

Consider which component projects are

the Overall Goal. Consider as to

effective or ineffective to the achievement

whether

of

Objectives

the within

Intermediate the

program

contribute the achievement of the

the

Overall

Goal.

Consider

the

synergistic effects among the component projects.

Overall Goal.

Impact

Consider positive and negative

Consider which component projects are

impacts (changes) accrued directly

the most or least influencing in causing the

or indirectly by the program. The

impacts, in a positive or negative manner.

impacts that are not expected from the planning are also included.

Relevance

Consider the relevance of the

Consider the relevance of component

Overall Goal to the needs of the

projects and activities to achieve the

beneficiaries,

Overall Goal. Consider the coherence

international

the

national

and

goals, socio-

economic trends, etc.

among the component projects. Consider the maturity of the proxy-program as a genuine program.

Sustainability

©FASID

Consider the sustainability of the

Consider

impact and benefits accrued by the

provide the most or least influence to the

program.

sustainability of the program.

which

component

projects

39

The above definitions are rough examples to provide some hints for creating evaluation questions. The evaluator should communicate with the clients and other stakeholders in order to clarify the evaluation criteria and items to be studied in accordance with the Scope of Evaluation.

3-1-3. Designing Evaluation Study The purpose of the evaluation study is to satisfy the needs of evaluation. Thus, the evaluation study should be designed by considering how to collect the necessary information in order to address the needs. The process of designing can be defined as follows.

①

②

③

What you should know to meet the needs？

What are the needs for evaluation (from the scope of evaluation)

④

What and how

The research

you should

method is both

examine to

effective and

answer the

efficient?

questions

It is recommended to undertake the design by using the following matrix, known as the “Evaluation Grid”, which contains the evaluation questions, methods and other necessary information. Evaluation Grid: Example Evaluation Purpose

Evaluation Questions (Criteria Level)

Evaluation Questions (Level 1…n)

Research Methods

Information Source

Needs for the

Issues and questions

If the issues and

Research methods

Resources (cost,

present

to be answered in

questions are not

to answer the

information) to

evaluation

order to meet the

clearly defined, specify

evaluation

meet the research

derived from

evaluation needs. As

these by breaking them

questions in the cell

methods

the scope of

the starting point,

down into smaller

to the left cell

evaluation

selected evaluation

questions

criteria is clarified at this level

Narrow question

Middle question Narrow question

Broad question Narrow question Middle question Narrow question

* Our Output Sample has Level 2 Questions.

For each evaluation criteria, several evaluation questions should be created from which the criteria can be defined in a concrete manner. In cases where the question is broad and ambiguous, it is

©FASID

40

recommended to decompose the question into a handful of sub-questions that are more easily answerable in practice, such as questions at level 1, level 2, level 3 and so on. The method to collect the information for answering each evaluation question is stated in the column to the right of each question. The number of these research methods for each question can be more than one. The other necessary information, such as the names of required documents can be added in the next column. As a result, the Evaluation Grid gains more rows (or layers) as you move towards the right. The Evaluation Grid can be used not only for designing the evaluation study, but also for placing the results of the evaluation study within it.

3-1-4. Setting the Methods of Data Collection Setting the methods of collecting data should need consideration not only of the accuracy and reliability of the data, but also of the time and resources that will be available in reality. There are several options for data collection methods as follows.

Method 1.

Description

Literature

Collection and analysis of

Survey

existing statistics, survey

Advantages Economical and efficient

Remarks Difficult to verify the reliability of materials

reports, work records, research data, etc. 2.

3.

Direct

Collection of data that can be

Provides accurate and

Facts only, no

Measurement

measured and classified based

reliable data; relatively

explanations

on actual surveys

low cost.

An interview or

Provides quantifiable

questionnaire-based survey,

and reliable data

Sample Survey

High cost

targeting a sample population of a sufficient size with regard to the parent population. 4.

5.

Case Studies

Observation

A detailed study of a small

Provides self-contained

number of cases.

detailed information

Undertaken using a checklist

Little need for

Results can vary widely

or survey sheet. May also

preparation. Particularly

depending on the

involve simple interviews.

suitable for a preliminary

observer.

Difficult to generalize

study. 6.

Key

Informant

Interviews

７Group Interviews

A technique often used in an

Low cost and easy to

Risk of bias due to the

evaluation. After selecting

implement. Provides

personal views of

persons well-versed in the

useful information and

interviewer and

matters concerned, detailed

sometimes draws

interviewee. Data cannot

interviews are conducted.

attention to unexpected

be quantified. Time

problems.

consuming

Interviews with representatives

Background information

There is a risk of creating

of the community or groups

can be obtained from

biased conclusions

(5-6 people each) in the form of

interviewees’ attitudes

through manipulation in

an open meeting.

and reactions.

the interviewee selection process.

©FASID

41

Although the selection of data collection methods is limited to available time and resources, it is recommended to use the following selection criteria in order to select the best methods.

・Validity ························· Do the evaluation questions cover what should be studied? ・Importance ···················· How important are the answers of the evaluation questions to the evaluation ? ・Reliability······················ How much are they reliable in their sources of information and methods? ・Accessibility ··················· Is the necessary information easily accessed? ・Cost ······························ Is the cost of conducting the method appropriate?

Selection Criteria of Research Methods Validity Importance

Selection of Methods

Cost

Reliability

Accessibility

In the process of designing the Evaluation Grid, participation of stakeholders including the clients is important. The evaluator should share the process of designing the Evaluation Grid with stakeholders from the draft version to the final version. This makes the evaluation more realistic and acceptable in its conclusions and recommendations at the end.

3-1-5. Data Collection for Factor Analysis The designing process of the Evaluation Grid should consider the issue of

a factor analysis on the

results of the evaluation study. Evaluation about effectiveness is concerned with the achievement of the Overall Goal, and a factor analysis needs information about all factors (positive and negative) influencing the Overall Goal’s achievement, such as the achievement of Intermediate Objectives, Activities, Inputs, and relative Important Assumptions. Such sub-topic information should be collected together with the main topic information for greater efficiency. Therefore, the evaluator should include the data collection for factor analysis at the

©FASID

42

beginning by examining the Log-frame or Logic Tree, having preliminary interviews with stakeholders, and predicting the results with experiences of similar programs. In practice, the following viewpoints are generally needed in collecting data for factor analysis. They are (1) the achievement level of each components of the Log-frame, (2) the conduct of management in Activities, and (3) cross-cutting issues.

Are overall goals achieved? YES

NO

Why are they achieved?

Why are they not achieved?

Factor Analysis What are reasons for success?

Degree of Achievement of the Log-frame

¾

What are reasons for failure?

Process of Activities (i.e. Stakeholder Relationship)

Cross-Cutting Issues

“Degree of Achievement of the Log-frame” is checked at each level of the Narrative Summary, such as Intermediate Objectives, Activities, Inputs and relative Important Assumptions.

¾

“The conduct of management in activities” concerns the way of conducting the planned activities, daily management style, and relationship among stakeholders in reality.

¾

“Cross-cutting issues” consists of aspects of policy, technology, environment, society, institution, and finance (or economy) that influence the program performance.

For example, the sample Evaluation Grid contains the evaluation question to check ,by conducting a questionnaire-based survey, as to whether there are other social factors or situations influencing the secondary school enrollment. This is one of the examples of collecting factor analysis data from the view of cross-cutting issues in “society” and “environment”.

©FASID

43

3. Evaluation Study Phase 3-2. Evaluation Study Implementation a) Outline

Objective:

To collect the necessary information by conducting the planned evaluation study.

Procedures:

1. To reconfirm the evaluation questions in the Evaluation Grid. 2. To conduct the planned evaluation study using the designed methods. 3. To describe and sort the results of the evaluation questions into the Evaluation Grid. 4. To inform the progress of the evaluation study.

Output:

Evaluation Grid with Results

©FASID

44

b) Output Sample: Evaluation Grid with Results Evaluation Grid with Results: Example Evaluation Purpose

Evaluation Questions (Criteria)

Evaluation Questions (Level 1)

Evaluation Questions (Level 2)

Research Methods

Results

1

1-1.

1-1-1.

Analyse the past

To conduct

Is the program

Is the Overall

Whether the rate of

trends by using

The continuance rate

comprehensive

effective?

Program

secondary education

statistics on

has been increasing

evaluation of

Objective

enrolment in the

secondary

year by year, but the rate

middle school

achieved?

capital city is

education in order

stays between 60% &

increased up to 90%.

to predict future

70% in the year XXXX.

trends.

The rate of students

education support programs,

from a poor background

which are

remains in the low level.

currently

1-1-2.

Analyse the past

implemented in

Does ratio of

trends by using

The ratio of graduation/

the capital of

graduation/

statistics on

entrance is diverse

Amelia, to

entrance

secondary

depending on the

modify these to

increase up to

education in order

school, but the average

make them

0.9 as

to predict future

ratio is approximately 8

more effective

planned?

trends.

to 10.

and efficient.

1-2.

1-2-1.

Is the

To what extent J’s

Examine the

Based on the results of

achievement of

development

association

regression analysis, J’s

overall

assistance

between the

overall contribution is

objectives

contributed to the

enrolment rate and

R^2=70%, which

attributed to

increase of the

J’s assistance by

indicates a positive

Country J’s

education

using regression

result, but the scale of

assistance?

continuance rate?

model.

the contribution is greatly different among programs.

How much is its contribution?

1-2-2.

Conduct

Most stakeholders point

Which programs are

questionnaire

out that program ○○ is

significantly

among

particularly effective but,

effective or

stakeholders

on the other hand,

ineffective?

particularly on

program xx is not quite

effective programs,

effective.

including questions on the reasons why they think this is so. 1-2-3.

Conduct qualitative

To what degree is the

evaluation of the

A primary goal of M’s

contribution of

impacts of the

project is to give training

assistance from other

Country M’s

on public administration,

donors or agencies?

education project

and therefore there is no

based on its

direct impact.

evaluation report.

©FASID

45

Conduct

Educational support

questionnaire and

activities for women run

interviews among

by a local NGO have as

stakeholders on

big an impact on

other factors

broadening educational

influencing the

opportunities for women

school enrolment.

as does J’s assistance.

2.

2-1.

2-1-1.

Is the program

Is there any

Is there any

Check with the

Activities of A and B are

efficient?

redundant or

redundant program

Logic Tree.

partially redundant.

overlapping

or project?

Is there any program

Lay out an actual

Although Programs A

that has identical

use of inputs to

and B have almost the

inputs?

identify

same items of inputs,

redundancies.

they have been

program? 2-1-2.

arranged independently and neither have been shared nor coordinated. 2-2-1.

Ask stakeholders

Are there any inputs

to rank the order of

Opinions from

sufficiently

that have a

utilization of inputs,

Stakeholders reveal that

utilized and

significantly low or

and also ask for

inputs XX are the best to

effectively

high level of

the reasons.

use, and for the reason

transformed to

utilization?

2-2. Are inputs

_____

outcomes? 2-2-2.

Ask stakeholders

How is the

to raise indicators

Input-Outcome Analysis

transformation from

to assess the

says inputs ○○&

inputs to outcomes?

contribution of

the highest

inputs to outcomes

transformation rate of all

and analyse them

inputs. The overall

by running a

transformation rate is

regression

XX %

analysis.

©FASID

46

has

c) Explanatory Notes 3-2-1. Implementation of the Evaluation Study The evaluation study is conducted in accordance with the Evaluation Grid. Sub-contractors can be procured to undertake the simpler tasks such as a questionnaire survey, so that the evaluator can concentrate on the whole picture of the evaluation. In using the sub-contractors

(e.g. surveyors or

private consultants), the scope of work of the contract should be linked coherently with the Evaluation Grid, which is shared with the sub-contractors, so that the sub-contractors should conduct only but the necessary tasks. When progressing the evaluation study, there may be a need to alter some components of the Evaluation Grid . In this case, the Scope of Evaluation should be reviewed before the identified adjustment is accepted. In other words, “the Scope of Evaluation”, “the Evaluation Grid” and “the Scope of Work with sub-contractors” should be integrated and managed consistently.

3-2-2. Question of Attribution One of the most challenging tasks of the evaluation process is the factor analysis to clarify the level of attribution of each factor to the results. The following diagram depicts the hypothetical model of attribution.

The literacy of students from Overall Goal

District A is improved

1 School facility is renovated.

2 Teacher’s capability is improved

3 Teaching materials are improved

Intermediate Objectives

In the above diagram, there are three factors at the Intermediate Objectives level to attribute the achievement of the Overall Goal. It is important to know which of these factors are the greater or least contributors to the Overall Goal. It is, however, a daunting task to pursue the answer in an objective manner. Currently, there are several techniques to do this type of analysis as follows.

©FASID

47

Techniques

Description

Comparative Analysis

Compare with the similar program which particular component

with a similar program

projects or activities are not included. From the comparison,

Remarks It is rare to find a similar program.

estimate the negative effects of the lost part of the program. In another word, benefits accrued from the program of our interest is equivalent to the negative effects derived from the estimation.

Comparative Analysis

Measure the degree of the change of the achievement level of the

in Time Series

Overall Goal between before and after the target project or activity started.

It is difficult to distinguish the sole effects from the others that have already existed.

Regression Analysis

Enumerate the achievement of each item in the program and apply the variables into the regression model to calculate the coefficients

Enumeration and modelling are difficult.

(a1-aN). (Overall Goal) = a1 x (IO1) + a2 x(IO2)+…+ aN x(ION)+ e

where “IO” means “Intermediate Objectives” This can be applied into panel data or time series data.

Qualitative Analysis

Qualitative judgment made by stakeholders whom are well-versed. May be quantified by making questionnaire surveys and analysed

Proper sampling is difficult. Result is still subjective.

with statistical techniques (Average and Variance). Experimental

From the beginning, apply the various sets of components in

Techniques

various areas and monitor the difference of the results.

Not possible to apply in ex-post evaluation. Basically denied from the ethical point of view.

*Details of these techniques can be referred to in books listed in the annex.

There are other techniques applicable for attribution analysis, but there are no standardized techniques in reality. From the other point of view, it is not recommended to use a highly complex technique that is understandable only to a limited number of specialists, because such a complex technique obstructs the understanding of the stakeholders, most of which are not experts on the technical issues. The reduction of understanding among the stakeholders will be more harmful than losing the technical robustness of evaluation, in the sense that such common stakeholders are the users of the evaluation in practice. If the user cannot understand the results, the recommendations won’t be accepted. Using highly complex techniques and modelling is also harmful in the sense that it

©FASID

48

would make the stakeholders, as well as the evaluator, lose the whole picture by focusing excessively on detail. Thus, when conducting the evaluation study, the evaluator should remember the needs and purpose of the evaluation. The evaluation study should be implemented in order to make good recommendations for improvement, and to give useful information for the users of the evaluation at the end.

3-2-3. Sorting the Results of the Evaluation Study Results of the evaluation study are put into the Evaluation Grid. The style chosen for the description of the results is part of the evaluation process to be decided. The results are placed to the right of the description of the research methods in the Evaluation Grid. If the information in the results column is too extensive, the results should be summarised with the detailed information being attached in an annex.

Evaluation Grid with Results Evaluation Purpose

Evaluation Questions

Evaluation Questions

(Criteria)

(Level 1...n)

Research Methods

Results

Notes

Needs for the

Issues and

If the issues and

Research

Summary of

Documents

evaluation

questions to be

questions are not

methods to give

Evaluation

related to the

derived from

answered in

clearly defined,

answers to the

Results

results and their

the Scope of

order to meet

specify these by

evaluation

Evaluation

the evaluation

breaking them

questions in the

needs.

down into small

cell to the left.

(Evaluation

questions.

serial numbers.

Criteria selected at this level)

The attached documents should be written in a detailed but understandable form by using charts and graphs. These documents do not just form a record. The documents in the evaluation should be always be written by bearing in mind that they will be read by important stakeholders.

©FASID

49

3. Evaluation Study Phase 3-3. Conclusions, Recommendations and Lessons Learned a) Outline

Objective::

To draw recommendations and lessons learned with overall conclusions.

Procedures:

1. Draw overall conclusions by reviewing the results of the evaluation study. 2. Draw recommendations. 3. Draw lessons learned. 4. Make presentation materials by summarizing the progress and results of the evaluation.

Output:

Conclusions, Recommendations and Lessons Learned

©FASID

50

b) Output Sample :”Conclusions, Recommendations and Lessons learned”

Summary of Evaluation Results for Secondary Education Assistance of Country J アメリア国中等教育支援プログラムの総合評価取りまとめ例 Title

Contents 評価結果取りまとめ

Summary and 総合評価

The degree of the achievement of the Overall Goal indicates that the 全体目標の達成度については堅実に中等教育の進学率は伸びてきて enrolment of the secondary school has been improved, and especially おり、さらに卒業率の改善も目覚しい。一方で、その改善は貧困層 that the graduation rate has been increased dramatically. There is, however, concern that the improvement seems to be enjoyed solely in に属する学生とっては限られたものである傾向があり、また有効性 the area populated mainly by middle income class households, and that からみても、貧困層や女性をターゲットにした事業が十分にその効 the improvement is limited in the “poor” area, where the population of 果を発揮していないことが懸念される。特に貧困層をターゲットと the residents have the lowest level of income. In the gender perspective, the improvement of the state of the female students is not した一部の事業の有効性が低かったことは、貧困が一部の事業で対 significant compared with the male students. 応するには大きすぎる問題であることと、事業単位で実施されてお The tendency for benefits not to be received by poor or female students should be of great concern, in the sense that Country J’s り、J 国の他の事業や NGO など既存の活動との総合的なアプローチ assistance is one of the programs for the poverty alleviation policy. がとられていないことに起因するものと考えられる。 There has been no significant improvement observed for the poorest 効率性の観点からも、 同じような事業や活動をしている場合も多く、 group of students. There are some reasons for this. Firstly, poverty itself is the complex and difficult problem that the secondary school また他の活動の経験や教訓などが十分に共有されていないなどの問 program can not solve on its own. Secondly, the projects have been 題が指摘されうる。投入財についてはおおむね効率的に成果に転換 conducted independently in practice and there is no effective coordination system among them. Thirdly, there has not been any されていると判断されるが、より事業を統合化し、共有するべきと cooperation with other active supporters, such as local NGOs and ころは共有するなどすれば、さらなる効率化が図られる可能性があ private schools. The last two matters are concerned from the point of view of る。 efficiency. Several inputs and activities are overlapped but neither (J 国に対して) shared nor coordinated. The lessons learned from a project did not get ・ back 貧困層支援の視点から、 J 国の各プロジェクトの組み合わせを見 fed to other projects, thus the same type of mistakes have been repeated. All inputs are efficiently utilized to the full, but a coordination 直し、ロジック・モデル及びログフレームを改善することを提 of projects will increase the efficiency of the whole structure of Country 言する。 J’s assistance. ・ 重複している活動については統合し、連携を強める。

Conclusions

提

言：

Recommendations

教

訓：

Lessons Learned

©FASID

(For J) ・ Country 今後はそのログ・フレームを元に、各プロジェクトが進捗状況 ・ The coordination of projects should be fundamentally reviewed and や情報を共有し、連携を保ちながら実施する。 reconstructed from the view of assisting the poor, by improving the ・ Logic 上記プログラムを運営管理するための担当者を任命する。 Tree and the Log-frame. ・ The overlapped or very similar activities should be combined and (アメリア国に対して) coordinated consistently. ・ The 以下の情報を、J 国のロジック・モデル作成の際には、それを提 ・ multiple projects should be managed as a program by using the improved Log-frame. 供する。 ・ A full-time program manager for managing the program should be - 中等教育に係る政府の中期戦略 assigned, while the project-level management should be conducted by project leaders, andNGO notの活動 by the headquarters office. - the 中等教育にかかる現地 (For Amelia government) - 中等教育にかかる他のドナ-の活動 In the improvement process, the government of Amelia should 中等教育に関する計画を検討する際には、貧困層への十分な配慮が provide the information as follows to Country J. - The mid-termなどのコミュニティーの協力は不可欠である。 strategy for secondary education. 必要であり、NGO - The detailed information of the local NGOs concerned. - The detailed information of the main donors concerned. In planning the assistance of secondary education in Amelia, it is indispensable to consider the situation of poor people, and to pursue the support from the community such as local NGOs.

51

c) Explanatory Notes 3-3-1. Drawing Conclusions Conclusions are not the summary of the results from the evaluation. They consist of the following points.

・

How are the results relating to the needs and purpose of the evaluation expressed in the Scope of Evaluation?

・

In the case of setting several evaluation criteria, which of the criteria are the results considered to be very successful or unsuccessful?

・

Is there any underlying common factor which boosts or hampers the results?

・

In the cases of where problems have been found, is it necessary to implement any improvement measures?

The conclusions are a basis from drawing the recommendations, addressing any problems pointed out in the conclusions.

3-3-2. Drawing Recommendations Recommendations are drawn from the conclusions in order to improve the situation. In practice, drawing recommendations can be promoted by using a participatory approach involving the relevant stakeholders, including the program implementer who will be given the recommendations. Through open discussions for giving ideas about possible recommendations, the final version of these recommendations can be drafted and their feasibility improved. The following remarks should be reiterated.

・

Recommendations should be drawn from the conclusions. Recommendations without any link with conclusions are not accepted.

・

Clarify with whom the recommendations are made.

・

Avoid using ambiguous expressions, and make them as concrete as possible.

・

The number of the recommendations should be a handful, and less than seven.

・

In order to realize the recommendations, the recommendations can address the need for structural reform of an organization or a detailed action plan, if necessary.

©FASID

52

3-3-3. Drawing Lessons Learned Lessons learned are lessons that should be considered when planning and implementing similar programs. Like drawing the recommendations, a participatory approach is encouraged to be applied in drafting the lessons, utilizing the various participatory methods such as brainstorming. The finalizing of lessons learned should also be made with consideration to feasibility. The following are critical points when drawing lessons learned.

・

Clarify “for whom” and “which kind of program” that the lessons learned contribute to.

・

Avoid using ambiguous expressions, and make them as concrete as possible.

・

The number should be limited to no more than three.

・

If the lessons learned can not be applied without a reform of the organisational structure, then such a point should be included as a long-term recommendation.

3-3-4. Preparing Presentation Materials Preparing presentation materials is one of the most important tasks in evaluation, since an evaluation that is not understood by the stakeholders is likely to be judged as a bad evaluation, even though its methods and contents are correct. The Evaluation Study Phase is closed only after the evaluation results are understood by the stakeholders, including the program implementers. Thus, the documents of the evaluation should be carefully prepared for the purpose of presentation. The presentation materials should consist, at least, of the following items.

・

Evaluation Report

¾

Chapter 1

Program to be evaluated

¾

Chapter 2

Evaluation Study Design

¾

Chapter 3

Results of Evaluation

¾

Chapter 4

Conclusions, recommendations and lessons learned.

・

Executive Summary of Report

¾

Program to be evaluated

¾

Conclusions, recommendations and lessons learned.

・

Slide show data with computer software (e.g. MS Power Point)

＊

In the cases where the client has a particular preferred format of reporting, then that should be

utilised.

©FASID

53

4. Improvement Phase

4-1. Feedback a) Outline

Objective::

To acknowledge the results of the evaluation with the stakeholders

Procedures:

1. Present the progress and results of the evaluation 2. Obtain comments from stakeholders 3. Discuss with stakeholders about improvement 4. Clarify the Commitment of Improvement with stakeholders

Output:

Commitment of Improvement

©FASID

54

b) Output Sample: Commitment of Improvement Title: Commitment of Improvement for Country J’s assistance for the improvement of secondary education in Amelia. Items

Contents

Recommendation

LEAD evaluation, conducted in the year XXXX, reported the following recommendations for Country J’s assistance for the improvement of secondary education in the Republic of Amelia: 1. The coordination of projects should be fundamentally reviewed and reconstructed from the view of assisting the poor, by improving the Logic Tree and the Log-frame. 2. The overlapped or very similar activities should be combined and coordinated consistently. 3. The multiple projects should be managed as a program by using the improved Log-frame. 4. A full-time program manager for managing the program should be assigned, while the project-level management should be conducted by the project leaders, not by the headquarters office.

Commitment of Improvement

In response to recommendations, the country of J will implement the following actions. 1. Hold a LEAD workshop to re-build a LEAD Logic Tree in reviewing the combination of several projects, except for Project A. Also hold a discussion to construct the LEAD Log-frame. 2. Take necessary measures to cut off redundant activities based on the improved Logic Tree. 3. It is difficult to administer Log-frame management in J’s assistance where project-based management is done. Log-frame management should be implemented in a few years time, but is referred to in the comprehensive review. 4. Appoint a commissioner to promote project harmonisation between the country of J and the Republic of Amelia, and initiate research that helps to implement it in a few years time.

Responsible Personnel/ Agency Others

For actions 1 to 3, Department of XX, J’s Agency for Technical Assistance For 4, appoint a responsible person for coordination by XX, XXXX. Next action: Run a preparatory committee for the workshop stated in the first action point in XX, XXXX.

Confirmed and signed by:

©FASID

Mr. X Ms. Y

Mr. Z (J’s Agency for Technical Assistance)

55

c) Explanatory Notes 4-1-1. Commitment of Improvement Unfortunately, there are a lot of cases that good recommendations from correct evaluations are not implemented in reality. One of the main causes of this failure is the lack of follow-up action to ensure the implementation of recommendations, while there are other causes such as the giving of useless recommendations that are ambiguous, unfeasible and irrelevant to the needs of the program implementers. Recommendations should be considered as a request for improvement that needs a reply from the concerned party.

The concerned party (the program implementer in most cases) that

receives the recommendations should clarify their commitment to the request for improvement by issuing the Commitment of Improvement. The Commitment of Improvement expresses the basic strategy to ensure the implementation of the recommendations, and acknowledge the respective roles of the stakeholders. It is not necessary to determine a detailed plan for implementation. Such a plan will be constructed in the next step. The basic contents of the Commitment of Improvement are as follows (although they can be modified if necessary).

Title

Contents

Recommendations from the last

State the recommendations and clarify them here if some ambiguity is

evaluation

found

Basic Strategy of Improvement

Clarify which of the recommendations are accepted and complied with. Also clarify the measures taken concerning the recommendations that are not accepted. Decide whether the Log-frame should be changed or not.

Implementers

Clarify who is responsible for leading the improvement.

Schedule

State when and how the first action is commenced, such as the initial meeting.

Main Stakeholders

Describe other stakeholders such as institutional and financial supporters, and their roles.

Monitoring

©FASID

How is the progress of the improvement monitored?

56

Commitment of Improvement is the policy level document that describes a rough sketch of the basic strategies and roles of the stakeholders. Individual stakeholders consider their own detailed implementation plans.

4-1-2. A Participatory Approach in Making the Commitment of Improvement It is recommended to apply a participatory approach in making the Commitment of Improvement, by holding a meeting of stakeholders after the presentation of evaluation results. In the meeting, decisions on several topics such as whether all or part of the recommendations are accepted, and who will take the leading role of implementation, are discussed and determined. Such a meeting will enable the following actions to be implemented more realistically. The chairperson of the meeting should be the representative of the concerned party for whom the recommendations are intended. In practice, the meeting may need a facilitator or a consultant from the evaluation team, in order to support a smooth procedure at the meeting. The representatives of the concerned party produce the draft version of the Commitment of Improvement.

However, it should be shared and agreed by consensus of all the main stakeholders

before final authorization is given.

©FASID

57

4. Improvement Phase 4-2. Improvement Planning a) Outline

Objective:

To improve the Log-frame in line with the Commitment of Improvement

Procedures:

1. Reconfirm the Commitment of Improvement 2. Improve the Logic Tree through a participatory planning workshop 3. Form the Log-frame from the improved Logic Tree 4. Hold a meeting to discuss the feasibility of the improved Log-frame 5. Authorize the stakeholders

improved

Log-frame

to

be

shared

with

Output:

Improved Logic Tree Improved Log-frame

©FASID

58

b) Output Sample: Improved Log-frame Amelia: J’s assistance for the improvement of secondary education: Improved Log-frame Target Area: Capital of Amelia Narrative Summary (of Program/Policy) Super Goal： ・ ・

Poverty in the capital of Amelia is reduced

Month/Date/Year Objectively Verifiable Indicators The population below the poverty line is less than 10% in the capital of Amelia by the year of XXXX

Means of Verification ・

・

Overall Goal：

・

The rate of poor population receiving secondary education is increased more than 80% in the capital by the year XXXX. The ratio of graduation/entrance of secondary students in Special Assistance Districts is more than 1 to 10 by the year XXXX.

Major Important Assumptions

National statistics derived from the Ministry of Social and Health UNDP Statistics Year Book National statistics on education derived from the Ministry of Education. Social Statistics

・

1-1.More than 90% of schools meet the national school facility standards level B in the special district.

1-1 Results of audit of the national standards of school facilities

・

The number of school facilities with sufficient educational function is increased.

2-1 An improved school curriculum is endorsed by the educational committee and by the third party by the year of XXXX.

2-1 Minutes of the Boards of Education

2

School curriculum is improved.

2-2 More than 80% of schools in the city introduce an improved curriculum by the year of XXXX.

3

Teachers’ skills and knowledge is improved.

・

The secondary education is more accessible for the poor in the capital of Amelia.

Intermediate Objectives： In Special Assistance Districts of the capital of Amelia-

・

・

・

・

・ 1

4

5

Students’ morale toward schooling (learning?) is increased. Accessibility of girls’ schooling is improved.

3-1.More than 70% of all teachers in the district have a license with which they can teach the program by the year of XXXX. 4-1.The rate of students from the special school district who think they enjoy schooling increases to more than 90%.

Government policies on secondary education are kept unchanged. Other poverty reduction programs are implemented. Trained teachers continue working for the school School buildings constructed by the project are adequately maintained

2-2 Statistics on education (the Ministry of Education) 3-1 Statistics on education (the Ministry of Education) 4-1 Results of Questionnaire and Survey 5-1 Results of Questionnaire and Survey

5-1.The rate of women who think girls have limited educational opportunities decreases to less than 10% by the year of XXXX.

Activities： 1.

1-1. 1-2. 1-3. 1-4. 1-5. 1-6. 1-7.

Provide school facility in a special district Select a school for the program Construction of a school building Equip the facility with a bathroom Equip with a water tank Training on how to manage and maintain the facility Procure teaching materials Audit the national facility standards

・

Inputs：

Project Name ・ Secondary school reconstruction project ・ Secondary school teacher training project ・Secondary school curriculum development project ・Secondary school scholarship program

・

Social condition surrounding students do not worsen. M’s projects to improve secondary school education are implemented.

Total Amounts: ooooooooo Yen Major Inputs: ○○、

、××

Pre-conditions：

Stakeholders: Amelia: the Ministry of Education, Ministry of Social Insurance, Ministry of Justice Local Public Agency: Department of Secondary Education

(The rest is omitted)

Remarks: After conducting the participatory workshops and social surveys, the program is improved to thereby be focusing more on the poor.“ Special Assistance Districts” are areas where the average income of households is less than 30% of the average of the total population of the capital.

©FASID

59

Output Sample 2: Improved Logic Tree created for forming the Log-frame

Poverty reduction in the capital city of Amelia is promoted. Enrollment rate for secondary education from the poor families in capital city of Amelia increases. The number of good school buildings increases

Activities

Activities

©FASID

Teaching curriculum improves

・Omitted・

Activities

Activities

Student's morale for schooling improves

Teaching methods improves

Activities

Activities

Activities

Activities

Female access to education increases

Activities

Activities

・Omitted・

60

c) Explanatory Notes 4-2-1. Cases Where an Improvement of the Log-frame is Not Necessary In cases where the Commitment of Improvement does not include the restructuring of the Log-frame in its basic strategy, those procedures in this stage can be skipped. In such a case, the existing Log-frame is accepted as the base plan and the action plan is formed for the necessary improvements.

4-2-2. Workshop for Improving the Log-frame In cases where an improvement of the Log-frame is needed, it is recommended to hold a participatory workshop with the stakeholders, rather than planning this at a desk or in a meeting attended by a limited number of planners.

A participatory workshop has several advantages as follows. ・

It nurtures and encourages the ownership of the stakeholders for the Log-frame and the action plan, which will increase their feasibility.

・

It improves the effectiveness of the plan by considering and adding various ideas from multiple views of stakeholders.

・

It promotes team building among the stakeholders through such teamwork.

There are, however, some disadvantages with the participatory workshop, such as “the planning is biased towards a few powerful participants”, “analysis tends to be short-sighted”, “the structure of the plan is inconsistent”, etc. To avoid these weaknesses of the participatory workshop, it should be carefully planned and managed by an experienced workshop facilitator. In the participatory workshop for considering improvements, the participants are the stakeholders indicated within the Commitment of Improvement. Discussions and analysis in the workshop can be progressed smoothly by using paper cards (half the size of A4 paper) and larger pieces of paper in order to visualize the discussions and analysis.

The goal of the workshop is to improve the Log-frame as the base plan.

Steps in the Participatory Workshop for Log-frame Improvement ①

Review of the existing Logic Tree - To review the Logic Tree critically from the Overall Goal, and identify the problems and shortcomings

②

Improvement of the Logic Tree - To refine the structure of the Logic Tree to attain the Overall Goal effectively.

③

Formation of the Log-frame improved.

©FASID

61

The above sequence of procedures is a basic example, with the program of the workshop being customized according to needs and constraints. When planning a participatory workshop of LEAD, it is recommended to read the guidebook about the Participatory Planning method of PCM, since the PCM is the foundation of the LEAD method. It is also a good idea to consult experienced moderators of the PCM workshop first of all. The evaluation study must have produced documents containing useful information. The results and other information from the last evaluation should be fully utilized within the planning process, especially when selecting the alternatives.

4-2-3. Ex-ante Evaluation of the Improved Log-frame After drafting the first version, the improved Log-frame is assessed and adjusted with the following evaluation criteria.

・

・

Feasibility of the Plan ¾

Is the program logic (or theory) well structured?

¾

What is the reality of the setting of each component (especially the inputs)?

Five Evaluation Criteria ¾

Is it “relevant” to the national policy and the needs of the beneficiaries?

¾

Is it effective to achieve the Overall Goal?

¾

Is it an efficient way for using the least inputs to produce the best outputs?

¾

Are there any positive or negative impacts to be concerned?

¾

Is it sustainable in its effects and good impacts?

Through the above ex-ante evaluation, the program expressed in the Log-frame is adjusted and improved before commencement if necessary. The ex-ante evaluation is recommended to be a part of the participatory planning workshop. For increasing the credibility of the plan, some of the ex-ante evaluation should be conducted by the third party.

©FASID

62

4. Improvement Phase 4-3. Improvement Implementation a) Outline

Objective:

To implement the improved Log-frame with a monitoring system, which enables the continuous improvement of the program.

Procedures:

1. Implement the action plan that complies with the improved Log-frame. 2. Conduct monitoring on the progress of the plan. 3. Implement a periodical evaluation study. 4. Iterate the improvement phase with the periodical evaluation study.

Output:

Continuous Improvement within the Improvement Cycle

©FASID

63

b) Output Sample: Improvement Cycle

Start from here!! Planning of

Commitment of

Activities

Improvement

Monitoring

Planning of

Planning

Improved Log-frame

(Plan) Implementation of Activities

Commitment of Improvement

Evaluation

Implementation

(See)

（Do）

Recommendation

Regular Evaluation

©FASID

64

c) Explanatory Notes 4-3-1. Progress Monitoring Progress monitoring is one of the most important activities after planning. Objectives of the progress monitoring are to confirm the progress towards the Overall Goal, and to conduct a proper adjustment if necessary. Thus, monitoring takes care not only of the Overall Goal, but also other components of the Log-frame such as the Intermediate Objectives, Activities, Inputs and Important Assumptions. The components to be monitored can be selected by referring to the Log-frame, in which the components are structured in the “Means-Ends” relationship.

Narrative

Objectively

Means of

Major

Summary (of

Verifiable

Verificati

Important

Program/Policy)

Indicators

on

Assumptions

Overall

”Means-Ends”

Goals

(Program

relationship

Goals)： Intermediate Objectives （

Sub-program

Goal） Activities:

Inputs： Pre-condition s

The most important item to be monitored is the achievement of the Overall Goal. It is, however, very rare that effects on the Overall Goal can be measured during the early stages of the program. Therefore, in the early period of the implementation, monitoring is concerned only with achievement of Inputs and Activities. As the program is progressed, the effects on Intermediate Objectives can be measured. Then finally the effects on the Overall Goal are reached, which is the final stage of the program.

In analysing the causes of delay in progress, the “means-ends” relationship in the Log-frame can be very useful. When finding that the effect on one of the Intermediate Objectives is not confirmed as planned, the achievement of the activities is firstly examined as the means for the Intermediate Objective. Next, the realisation of the relevant Important Assumptions to the Intermediate Objectives can be referred to. If finding that some of the activities were not implemented, the cause can be analysed by examining the achievement of the planned components in the Inputs section or the realisation of the relevant Important Assumptions.

©FASID

65

Intermediate Objectives are not achieved

Are activities successfully

Are major important assumptions

implemented?

realized?

Are inputs successfully

Do other activities achieve their

Are major important

projected?

goals?

assumptions realized?

If these problems are found, it requires counter-measures such as adjustment, avoidance, mitigation, and so on. The counter-measures are different in each situation, but it is important to plan for the counter-measures within the basic organisation.

4-3-2.

Monitoring System

Monitoring should be conducted systematically within a monitoring system, which must consist of “data collection”, “aggregation”, “decision” and “communication” as its components. The personnel and organizations responsible for this should be clarified for each component. It helps the understanding if the monitoring system is described in a diagram or a table as follows. Example of a Monitoring System Narrative

Objectively

Means of

Data

Aggregation

Decision

Method of

summary

Verifiable

Verifi-

Collection

・ personnel

・ personnel

counter-

Indicators

cation

・ personnel

・ schedule

・ schedule

measures

・ schedule

・ method

・ method

・ Leader of

・ Steering

・ method (Intermediat

By the year

Monitoring

e Objective:

XXXX, more

report of

women’s

literacy

Committe

counter-

1)

than 80% of

the

groups

program

e

measures can

Literacy rate

adult

program

of women in

women in

village A is

village A can

increased.

read basic texts.

・ Chiefs of

・ Every 3

・ Every 3

・ Every 3

months

months

months

・ Sample survey

・ Aggregate data

・ Discuss and vote

Necessary

be conducted by the implementers, if they are at the activities level.

©FASID

66

The above table is just an example. The monitoring system should be customized with the needs of the stakeholders and based on utility.

4-3-3. A Periodic Evaluation for Further Improvement A periodic evaluation is necessary in order to improve fundamentally the structure of the program as the program is being progressed. The periodic evaluation starts by creating a Scope of Evaluation, and with the procedure then just following the steps from the evaluation phase of LEAD, as described previously. Conducting the LEAD procedure the second time will be much easier than the first, since the Log-frame has already been constructed and the stakeholders will be more experienced.

©FASID

67

5. Epilogue:

Everlasting Cycle of Improvement

The concept of PCM (Project Cycle Management) is to manage the entire cycle of a project, consisting of planning, implementation and evaluation, consistently with a prepared plan named PDM (Project Design Matrix), which is the Log-frame. In contrast, LEAD (Log-frame Evaluation Application Design) starts from the evaluation by combining related projects and activities into a proxy-program into a Log-frame, which is then evaluated as a program. LEAD continues to the following improvement phase, in which the recommendations from the evaluation lead to the improvement of the existing program by a re-planning of the Log-frame. The improved Log-frame will be used in the management of the program cycle.

LEAD results in management via a program cycle of continuous improvement, known as

“KAIZEN” in Japanese. The basic concept of the cycle of continuous improvement is the same as that of the PCM, such as a cycle of “plan”- “Do”- “See”.

LEAD places greater emphasis on improvement, so that a PDSA

(Plan-Do-See-Action) cycle applied in Japanese TQC (Total Quality Control) is more suitable to LEAD. The message of the PDSA cycle is that the evaluation should never be conducted only just for the purpose of evaluation, and that the evaluation must be strongly linked with further actions to improve the situation. The PDSA cycle never stops, with improvement being continuous. The objective of the LEAD evaluation is not to give recommendations that are only tentatively worthy. Rather, it is to install a cycle of continuous improvement by the use of a Log-frame. LEAD leads you to establish such an integrated management system with a Log-frame.

Program

Program

Program

Program

A

B

C

D

Integrating as Program Evaluation

Regular

Improved

Evaluation

Plan

Implementation of Plan Cycle of Improvement

©FASID

68

Reference Books (not completed, but useful) ・ FASID (2002), PCM Project Cycle Management FASID

Participatory Planning,

・ FASID (2000), PCM Project Cycle Management Monitoring and Evaluation, FASID ・ Rossi, Peter H, Forward E. Freeman, and Lipsey, Mark W.(1999), Evaluation: A Systematic Approach, 6th ed., SAGE Publications ・ Weiss, Carol H.(1998), Evaluation: methods for studying programs and policies, 2nd ed.., Prentice Hall ・ Wholey Joseph S., Harry, and Kathryn E. Newcomer, eds(1994), Handbook of Practical Program Evaluation, Jossey-Bass ・ Patton, Michael Quinn. (1997). Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage

©FASID

69

Member List Members of -

Study Group for Evaluation on Public Policy and Programs

Ryo SASAKI Keiko NISHINO Kiyoshi YAMAYA Masaki SAITO Yujiro HAYAMI Masafumi YAGI Yoko ISHIDA Naonobu MINATO

Members of PULUG (Production Unit for LEAD Users Guide) -

Takahiro MIYOSHI Yasushi HAYASHI Mari FURUYA Mitsuru KAYAMA

All comments and inquiries must be received by Naonobu Minato. The contact address is [email protected] Foundation for Advanced Studies on International Development (FASID) Chiyoda Kaikan Building (4th and 5th fl. ) 1-6-17, Kudan-Minami, Chiyoda-ku, Tokyo 102-0074, Japan

©FASID

70