Participatory Monitoring Detailed Outline

  • November 2019
  • PDF

This document was uploaded by user and they confirmed that they have the permission to share it. If you are author or own the copyright of this book, please report to us by using this DMCA report form. Report DMCA


Overview

Download & View Participatory Monitoring Detailed Outline as PDF for free.

More details

  • Words: 6,536
  • Pages: 28
Participatory Monitoring and Evaluation Detailed Outline TABLE OF CONTENTS

Section I:

Introduction and Overview • Welcome • Shifts in Development Thinking • Learning Objectives • Methodology • Target Audience • Expected Outcomes • Lesson Plan • Exercise

Section II:

Concepts and Principles of Participatory Monitoring &Evaluation • Definitions • Indicators, Outputs, Outcomes And Impact • Quantitative, Qualitative and Participatory Methods • Principles of PM&E • Role of Gender • Value-Added of PM&E • Exercise

Section III:

The PM&E Process Cycle

Section IV:

PM&E Case Studies • Micro or Project Level • Meso or Local Governance Level • Macro or Policy Level

Section V:

Exercise: Group Action Planning

Section VI:

Additional PM&E Resources

Appendix A:

Monitoring Mechanisms References

1

SECTION I:

INTRODUCTION

Welcome Most of us would agree that in theory, evaluation is a good thing. But putting this ideal into practice is often seen as too complicated, conflict-ridden, and/or costly. This Module is designed to introduce diverse development stakeholders to Participatory Monitoring and Evaluation in different contexts – what it is and how it adds value to development work at both the project and the policy level. Participatory Monitoring and Evaluation (PM&E) is a tool for bringing diverse stakeholders together, giving them an opportunity to negotiate regarding what success should look like and what indicators should be used. This process of negotiating the definition of success is a first step in institutionalizing participation and voice in development. Whose information is used to assess the success of a policy or project is important if we aim to make projects and policies more grounded in the realities of poor and disenfranchised people. PM&E ensures that all stakeholders have a voice in what indicators are used and how the results are analyzed. Multiple sources of information guarantee the greatest accountability and triangulation of results, while allowing for the monitoring of monitors. There are many potential positive spillovers from PM&E -- for example, experiential learning through working in diverse groups builds trust and the ability to cooperate across sectors/stakeholders for future development endeavors. Shifts in Development Thinking PM&E encompasses some of the major conceptual shifts in development work today. In accordance, this Module is based on the following assumptions: 1. Participation and ownership of all stakeholders is essential for sustainable development. 2. Monitoring and evaluation are critical for holding development actors (all members of society) accountable, measuring our successes and learning from our mistakes. This means going beyond measuring inputs and outputs. 3. Expertise is not the monopoly of technocrats; people at all levels hold important relevant information. 4. Working across groups in an atmosphere of trust (with some healthy skepticism) is important. Experiential learning prepares people to work cooperatively across groups and organizations; people often need spaces and opportunities to build healthy social capital across institutions. Learning Objectives of this Module By the end of this module, participants should be able to: Articulate the importance of monitoring and evaluation – who measures; what goals; and how should it be done; Highlight the value-added of participatory monitoring and evaluation versus traditional approaches; Describe the process cycle for designing a participatory monitoring and evaluation program applicable to various projects, policies and processes; Appreciate the wide variance in evaluation approaches; each approach needs to be tailored to specific context and capacity; Recall model practices and case studies on PM&E in different settings and refer to additional on-line resources; 2

Collaborate with other groups of stakeholders on PM&E activities, based on practice during the module; Call upon a community of practice to support one another post-training. Methodology In order to meet the above objectives, this Module is delivered in a participatory, interactive way, and is intended to be adapted to your local needs. Follow-up activities may involve Distance Learning (computer-based/internet training and video-conferencing) where appropriate. The minimum time required for delivery of this module is three hours, however several face-to-face and/or Distance Learning activities can be added to deepen understanding and skills. Case studies will also be used to help participants apply their learning during the course itself. Target Audience The primary audience for this Module is community-based groups. Other target audiences include representatives from NGOs, local and national government, private sector, media, think tanks and donors organizations. Ideally a diverse group of organizations will be represented during the training and have the opportunity tot work together on some of the practical applications during and after this module. Trainers for any of the above mentioned constituencies will also be targeted. Expected Outcomes By the end of this Module, participants will hopefully view PM&E as a doable, collaborative and worthwhile investment for designing and delivering programs and policies that meet the needs of a wide range of stakeholders, especially poor people. Participants will be familiar with the PM&E Process Wheel and will have practical experience working through some of its stages. Participants will also know where to find additional resources and technical assistance via the internet and through the community established during the training. Note: this module will offer sufficient background and experiential learning for stakeholders to participate in on-going or future PM&E activities. However, it will not provide sufficient technical skill-building to enable all participants to design and implement their own monitoring and evaluation. Additional learning will probably be necessary. Lesson Plan • Section II presents a Conceptual Framework including Definitions and Principles of PM&E. • Section III introduces the PM&E Process Cycle which can be modified and applied in a variety of different contexts. • Section IV provides short Case Studies of PM&E at the micro, meso and macro levels with Learning Activities which allow you to practice different stages of the Process Cycle. • Section V offers space for Action Planning regarding potential PM&E relevant to your own context. • Section VI refers you to additional resources (some of them on-line) and opportunities to stay engaged in a PM&E Community of Practice in order to be involved in follow-up learning activities and to request and/or provide technical assistance to and from your fellow participants.

3

Before You Begin… Before you begin, we would like you to reflect a little on your own situation by answering the following questions.

1. What project or policy in your local community or region would you like to see monitored and evaluated using participatory techniques? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________

2. What are two expectations you have for this course/module? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________

3. How will we judge success at the end of this module? How will we know if this course has been successful and if it has met your expectations? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________

=====================================================================

SECTION II: CONCEPTS AND PRINCIPLES OF PARTICIPATORY MONITORING & EVALUATION ===================================================================== Definitions Monitoring examines whether the project has been implemented as planned, made good progress, and faced any problems. It also considers how to solve such problems. Monitoring helps all stakeholders to recognize problems and obstacles, and to be able to solve them immediately. It thus plays an important role to ensure that activities are performed on schedule and within the allocated budget, that all concerned are satisfied, and that the project outputs and outcomes correspond to the objectives. It enables a more responsive, accountable and transparent system of operating.

4

Evaluation systematically judges the value of changes (planned and unplanned) resulting from project outputs and outcomes. Evaluation also reveals the state of changes, the persons such changes affect, and the extent of the benefits from such changes. Evaluation helps to target project outcomes, and the extent and features of project benefits. This makes sure that we measure not just inputs and outputs but what significant changes have taken place. Monitoring and evaluation yield the greatest benefits when they are performed in a participatory manner. In this process, all stakeholders discuss and plan the project together from the outset, jointly setting the objectives, targets, and work process. The evaluation should take into consideration the impact upon individuals and the community in terms of the development of new ideas and quality of life, resource mobilization, income distribution, selfreliance, and environmental and natural resource conservation. Participation effects monitoring and evaluation in the following ways: • The purpose of the evaluation • The criteria and indictors identified • The design and implementation of the evaluation • Who participates in the monitoring and analysis • How the information is used. PM&E is a systematic management tool designed to reveal the degree of effectiveness and efficiency in the achievement of objectives according to the perspectives of various stakeholders. The mission of the monitors is not to find fault, but to give advice and assistance so that the project can fulfill its objectives efficiently and effectively. ===================================================================== Citizen participation in monitoring the activities of local administrative organizations is an important method for creating good local governance. Participation ensures four things: • • • •

Responsiveness: objectives are met according to the priorities of all stakeholders; Transparency: clarity and openness regarding activities and relationships; Accountability: actors are held responsible for their contributions by other constituents; Equity: inputs and benefits reach the various stakeholders fairly.

Monitoring and evaluation offer not only tools for effective and efficient project implementation, but also a learning process for all concerned. Notably, all interested parties and civil society can play a role themselves and take part in making decisions and planning policies for development and problem solving. Evaluation can be conducted at several levels. For instance, the evaluation of a project designed to change unfavorable citizen behavior can be done at three levels, as follows: 2) Primary evaluation or phase 1 is to assess the output or immediate outcome. It can be done immediately after the training, by measuring and comparing the trainees’

5

knowledge before and after the training. 3) Secondary evaluation or phase 2 is to assess project effectiveness. Output is compared with objectives by following the trainees’ behavior to assess whether and how they change their behavior after a certain period. 4) Tertiary evaluation or phase 3 is to assess the outcome and impact, by assessing how such behavioral changes affect society.

Indicators, Outputs, Outcomes And Impact Four key terms tell a monitoring and evaluation group the effects of a project: indicator, output, outcome, and impact. Indicator: a variable to measure the progress of activities or changes resulting from the project activity, which assesses whether a change is in the desired direction, and whether the objective will be achieved. A good indicator reflects both quantitative and qualitative changes. Quantitative indicators can be displayed as a ‘single figure’ (such as the amount of produce in a given year), or a ‘cumulative figure’ (such as the produce over several years combined together.) This figure can be in the form of percentages and proportions. Qualitative indicators measure either a behavior or an indirect representation. These indicators often require open-ended responses. For instance, qualitative indicators may: • Measure behavior in food consumption from weight and height gained • Measure the health situation by the availability of clean drinking water and toilets in a village • Measure poverty by the condition of houses and construction materials Output: the product or service produced in order to achieve a target. For instance, to achieve a target to enable 75% of women to understand laws and the Constitution, the outputs include training documents, materials, trainers, workshops, and so on. Outcome: a result after the use of outputs. For instance: • The availability of clean drinking water in the village results in an outcome of a lower incidence of diarrhea. • School construction results in an outcome of higher enrollment. • Provision of loans without collateral results in an outcome of more capital for women to invest. Impact: the final result of the outcome. The impact might be seen by the end of the project or several years afterwards. It may change the living condition of project participants. For instance, • Lower diarrhea results in a lower infant mortality rate, and healthier children. • Higher enrollment in a school might have the impact of increased literacy. • Women’s having more capital to invest results in this impact: higher income and better

6

nutrition for children. ===================================================================== Exercise: Based on the local project or policy you proposed to monitor and evaluate, identify one of each of the following: Indicator: _____________________________________________________________________ Output: ______________________________________________________________________ Outcome: _____________________________________________________________________ Impact: ______________________________________________________________________

==================================================================== Monitoring and Reporting

Principles or Characteristics of Participatory Evaluation

The Role of Gender

The Value-Added of PM&E

SECTION III: QUANTITATIVE, QUALITATIVE AND PARTICIPATORY METHODS

1



‘Quantitative’ methods often refer to random sampling for survey research, structured individual interviews for data collection and subsequent statistical analyses which allow the results to be considered representative, comparable and ‘generalizable’ to a wider population.



‘Qualitative’ methods often refer to a spectrum of data collection and analysis techniques where sampling is not necessarily random and more in-depth, unstructured, open-ended efforts are incorporated. This allows for in–depth analysis of social, political and economic issues. 1

Krishna and Shrader. 2000

7

Given the imprecise divide between quantitative and qualitative approaches, it may be more useful to distinguish between contextual versus noncontextual methods of data collection and analysis (Hentschel 1999). However, neither of these approaches are guaranteed to be participatory. And both methods can be carried out in participatory ways. Participation requires that various stakeholders have a voice and some leverage in deciding how a policy, project or evaluation is designed, implemented, monitored, and assessed.

SECTION IV: PARTICIPATORY MONITORING & EVALUATION PROCESS CYCLE Participatory Monitoring and Evaluation Process Cycle 2 a. Conduct a readiness assessment; b. Decide who participates, establish principles for working together; c. Establish goals; d. Develop indicators; e. Gather info (quantitative and qualitative); f. Construct baseline; g. Set realistic targets; h. Gather info again (quantitative and qualitative); i. Analyze results including all stakeholders; j. Share info/results with all relevant parties; k. Take action – use the findings!; l. Start over (making any needed modifications in process…)

SECTION V: THREE CASE STUDIES OF PARTICIPATORY MONITORING AND EVALUATION Case Study 1: Haiti3 This case study is divided into three parts: 1. Introduction and the case of the CCIC in Haiti 2. A few myths about PM&E 3. Activities for learning more about PM&E 1. Introduction and the Case of the CCIC in Haiti Evaluations classically exist to ensure accountability, so that funds are spent as they are intended and so that we learn from our mistakes and build on successes. The two underlying assumptions of most evaluations are that the people judging a project are a neutral and independent third party; and that an evaluation without these neutral, independent evaluations has no claims to

2

References: Gaventa IDS, Kusek and Rist, Narayan, and PRSP Sourcebook): The information in this case study is drawn from Françoise Coupal and Marie Simoneau, “Participatory Evaluation: A Case Study of CCIC Humanitarian Fund Projects in Haiti” (http://www.mosaic-netintl.ca/pehaiti15.html).

3

8

“objective” effectiveness. This approach is mainly quantitative because it chiefly measures inputs and outputs, and it seeks quasi-scientific evidence for judging project success. Many observers have called for a new approach to evaluation, one that is qualitative and participatory. The advantage of participatory evaluation can be summarized by this diagram,4 which illustrates its added benefits: Accountability to Stakeholders, Managers, and Donors

Build capacity of stakeholders to reflect, analyze, and take action.

Various functions of Participatory Evaluation Development of lessons learned for corrective action by stakeholders

Development of lessons learned which help program staff improve Improve rapport between government and civil society.

4

The diagram is adapted from (and slightly changed) Coupal and Simoneau, 3. Adapted from Who Are the Question-Makers? A Participatory Evaluation Handbook (New York: OESP, UNDP, 1997).

9

Recently, the Canadian Council for International Cooperation (CCIC), an aid organization, took the principles of participatory M&E and tested them, with notable success, in a somewhat volatile and unstable setting: Haiti, in the wake of a political coup. Haiti is the poorest country in the Western hemisphere. In 1990, Haiti held its first-ever democratic elections, and elected Father Jean-Bertrand Aristide as President. After seven months there was a military coup, ousting President Aristide and stirring a wave of violence and unrest. The United States invaded Haiti in 1994 and President Aristide was restored to power. As Haiti began to rebuild its fragile civil society, NGOs and grassroots organizations began to implement development programs. The CCIC, a coalition of volunteer groups, funded thirty-six Haitian development projects, ranging from agriculture, to nutrition, to credit, to communications. As part of its broad program, the CCIC made a point of using participatory monitoring and evaluation. Its intent was to go beyond the simple accountability function. Accountability only documents inputs and outputs instead of impact. Instead, the CCIC aimed to build capacity among Haitian and Canadian NGOs by training local participants in participatory methods. The CCIC trained approximately thirty personnel from NGOs in Haiti in participatory methods of monitoring and evaluation. Some of these personnel were reluctant to participatory methods at first. After conducting a four-day workshop and an evaluation process, the participants concluded that the success of any development project depended on several factors: • • • • • • • •

It must respond to real priorities in the community Training should create and build capacity of organizations and groups Group dynamics should enhance feelings of belonging and of honesty Community participation from the onset is important Good governance and leadership of the project is key Good project structure and follow-up is essential Strategies must be flexible Sufficient resources must be available

The program also found that project aims can be limited or blocked by the following factors: • • • • • • 2.

The centralization of power, or inadequate sharing of information on a project Poor management and lack of transparency Poor leadership and paternalistic attitudes Egotism A lack of community participation and communication between project managers and project beneficiaries Poor monitoring and follow-up A few myths about participatory evaluation and monitoring

10

The CCIC found that its NGO personnel participants brought with them four reservations about PM&E, which their experience in the workshop significantly changed: 1. Participatory evaluations can only be done with projects designed in a participatory manner. 2. Participation takes time and is costly. 3. Project participants cannot “objectively” evaluate projects. 4. Evaluations require “experts.” The first of these, that participatory evaluation only works in a project devised with participation as its basic principle, failed to hold water. Though many of the CCIC projects were designed traditionally, to respond to donor priorities, the inclusion of a participatory M&E approach “did not disrupt the status quo.”5 In fact participatory approaches to M&E allowed NGO personnel to appreciate its effectiveness in all aspects of project design, and they became better skilled in its methods. Time and cost objections were also reversed when the participatory M&E was revealed to (1) cost the same as traditional evaluation; (2) take the same amount of time; and (3) purchase, for its time and money investment, a greater impact than a traditional evaluation. Facilitators declared after the participatory M&E training that “it really changed the way that [they] saw development.” Finally, the concerns about objectivity and expertise were also found to be unsupported; participatory M&E facilitators found that a core of committed and skilled practitioners can train participants to evaluate projects fairly and effectively. Field Insight: Lesotho Community people know well the relative wealth and poverty of other community members. In Lesotho, a way was found to put community knowledge to use without resorting to socioeconomic surveys. Based on information from earlier studies, a set of thirty small picture cards was drawn, signifying indicators of local wealth and poverty, such as maize porridge, rice, cattle, chickens, and a nice house. A few blank cards were provided so that any missing indicators could be added.

Village women sorted the pictures into three categories of items owned by the rich, average, and poor people. They then picked three indicators for each category of wealth. Feeling no need for secrecy, the women took only a few minutes to rate themselves and neighboring families in the appropriate categories. In total agreement, they then recategorized the cards cased on the gender of the head of household, thus identifying wealth differences with gender. Source: Deepa Narayan, Participatory Evaluation, (World Bank Working Paper, No. 207), 64.

5

Coupal and Simoneau, 15.

11

3.

Activities.

Activity 1. What project do you want to monitor and evaluate? What knowledge or expertise of this project do you bring to the table? What knowledge and expertise would you like to see involved in monitoring and evaluation, and who else might you engage in your M&E process to add this?

Activity 2. What are three words that come to mind when you think of evaluation? List them here.

Here is a list of evaluation characteristics. Mark “T” next to the ones you consider typical of traditional evaluation, and “P” next to the ones you consider participatory. _____Conducted by experts _____Empowers local people to take action _____Focuses on scientific objectivity _____Open results _____Frequent small evaluations _____Conducted by community members _____Results have limited access _____Indicators set by inputs and outputs _____Procedures simple and adapted to local culture _____Conducted to determine accountability _____Takes place at the end of a project _____Procedures are uniform and complex Source: Narayan, 12.

12

Case Study 2: Porto Alegre, Brazil6 This lesson is divided into two parts: 1. Introduction and the case of Porto Alegre 2. Activities to become acquainted with participatory monitoring 1. Introduction and the case of Porto Alegre Setting policy on a broad scale, such as deciding the budget of a region, is a complicated and lengthy process; participatory practices and approaches prevent a local administration from designing a policy, at great effort, which may ultimately fail because it does not reflect or respond to local priorities.

Citizen participation in monitoring and evaluating the activities of local administrative organizations helps to ensure four things: Transparency: the clarity and accessibility of local administration Accountability: tasks are done and money is spent as planned Responsiveness: objectives are met Equity: benefits reach the locality fairly The example of Porto Alegre suggests that participatory monitoring not only relieved local communities from some of the severe effects of the macroeconomic crisis experienced during Brazil’s process of decentralization in 1988, but also that it improved local conditions. Porto Alegre implemented participatory budgeting in 1988, during the first challenging year of the new decentralized administration in Brazil. One of the best examples in terms of participatory budgeting, Porto Alegre’s system is an instrument that combines direct democracy with the empowerment and promotion of parliamentary representation.7 Now over ten years old, participatory budgeting consists of a series of open meetings held March through June by the city, the community, and by associations and groups such as trade unions. The purpose of the small-scale meetings is to elect representatives for regional meetings, and to set the budgetary agenda to be promoted by the municipality at the regional meeting. After local meetings are completed in all the regions, representatives meet as a Council to set the budget. Porto Alegre reports the following successes since the inception of participatory budgeting: 6

The information in this case study is drawn from Zander Navarro, “Participation, Democratizing Practices, and the Formation of a Modern Polity: The Case of ‘Participatory Budgeting’ in Porto Alegre, Brazil (198901998)” (Development 41: 3 [Sept ‘98]: 68-71). 7 Participatory Budget, Municipal City Hall of Porto Alegre (Municipal Department of Culture, Coordination of Social Communications, 1995).

13



Considerable reduction in widespread corrupt behavior and administrative malpractice. Open meetings mean open books. By holding open meetings and making the budget every citizen’s business, the processes of allocation and distribution are less susceptible to bribery, graft, or other “backstage” arrangements between civil servants and private interests. Reduced corruption also increases public confidence in local government, which encourages even greater levels of civic engagement and participation.



Considerable reduction in clientelism (or patronage). Confronting a better-informed and more demanding citizenry, well versed in the new “rules of the game,” city councilors and would-be candidates notably improved in unfair practices of awarding patronage.



More equitable, “progressive” distribution of taxes and resources. Property tax, the main source of revenue in the region, was reassigned on a progressive scale. In effect, wealthier neighborhoods are now taxed at a higher rate than poorer neighborhoods. Correspondingly, poor neighborhoods are allocated a greater share of resources, in keeping with their greater neediness.

The participatory budget practice in Porto Alegre offers a way to redistribute resources to the poor, to allocate funds more fairly overall, and to reclaim political relations in a formerly corrupt system. The process, in brief, is this: • • • • • •

The region is divided into 16 areas for a more representative system The Participatory Budget Council includes two representatives from each area Municipal meetings, intermediate meetings, and thematic meetings, organized by the community, are held to discuss and decide public works decisions and budget priorities, and other topics raised by community leadership Local delegates attend the Participatory Budget Council to discuss and decide the priorities of the region, i.e. where the municipality should invest The Council delegates attend the Forum of Delegates for the region, to discuss and decide public works, to inform the community about their actions, and to receive new requests The Council establishes the next year’s budget and it is approved by elected city councillors

Citizens in the region were able to identify and prioritize their needs; co-manage municipal resources; control and evaluate the mayor’s commitments; and create opportunities for participation, accountability, and transparency.8

8

Participatory Budget, Porto Alegre.

14

2. Activities Activity 1. What is the quality of public services in your area? Give each one a letter grade from A to F. ___Roads: Paved, no potholes; traffic lines recently painted; trees plentiful and in good condition ___Buildings: No abandoned buildings; building and other permits are issued without difficulty ___Lighting: Working street lights ___Sanitation: Streets cleared of litter; garbage collected regularly; recycling pick-up available ___Motor vehicles: No abandoned cars; parking regulations enforced; traffic lights work ___Other: Public telephones, mailboxes, newspaper machines available and in good working condition ___Safety: police and fire departments come when they’re called; stray dogs collected regularly; crime not severe ___Resources: food and other stores are within reasonable distance ___Amenities: libraries, cinemas, museums, or parks within reasonable distance

Which public service received the lowest grade?_______________________ Which public service received the highest grade?______________________ What specific things, in your opinion, caused the worst public service to perform poorly? What caused the best public service to perform well?

Using the M&E Learning Wheel, how would you describe the “baseline” of one of these two public services?

15

Activity 2. Do you think a participatory budget process could work in your community? Why or why not?

Consult the M&E Learning Wheel. Which steps of the Wheel would present the greatest challenge for doing participatory budgeting? Which steps would be the most useful?

16

Field Insight: Nigeria In Nigeria, a workshop was held to evaluate the budgetary process for a large externally-financed rural water and sanitation project. In attendance were the project coordinator, technicians, and local government management and extension staff. In the informal atmosphere, the participants were each asked to list who made decisions about resources for different project activities and how those decisions were reached. The subsequent discussion revealed that while the project was trying to support greater autonomy on the part of local governments in the planning and implementation of rural water and sanitation systems, in effect the main decisions regarding disbursement of funds were made by the project coordinator. This was done to ensure accountability and to avoid mismanagement of funds, but it also had the unintended effect of reducing the sense of ownership and control over the project by the local government administration. Similarly, while the project strove to encourage community participation and self-determination, the schedule of project activities was also beyond the control of community committees. Although this situation was a disincentive for more committed involvement in the project by both government and communities, the need to account for externally provided public funds and the pressure placed by donors to achieve construction targets in a timely fashion made it difficult for project managers to contemplate an alternative approach. Source: Narayan, 57.

17

Case Study 3: Sri Lanka9 This lesson is divided into two parts: 1. Introduction and the case of Sri Lanka 2. Participatory planning and evaluation activities 1.

Introduction and the case of Sri Lanka

Rural development includes projects such as equipment repair and replacement, the expansion of credit services to farmers, and the introduction of subsidies for certain crops. It can also include investment in local health care, education, water supply, transportation, and electrification. To make rural development sustainable, a key component of project design is participation by beneficiaries. Those on the receiving end of rural development see the greatest benefit when a project is planned around their contributions and involvement and they are participants in the whole process. Such participation develops a sense of ownership among stakeholders, who take a greater interest in the continuation and maintenance of the assets and processes that the project seeks to improve. The example of Sri Lanka offers a lesson in how participatory monitoring and evaluation--which is typically added onto development projects at their conclusion--could have significantly transformed project results and impacts if it had been integrated into the project from the start. This case study is about a project that had limited success, according to its traditionally-set goals, indicators, and methods. Importantly, during participatory evaluation, project managers learned that participatory methods would have improved the and outcome of the entire project. The disappointment of the rural development aims, alongside the successes of participatory M&E, offer strong evidence of the potential this approach to project design, especially in setting indicators. In Sri Lanka in the 1970s, the World Bank funded two major rural development projects (the Kurunegala Rural Development Project, or KRDP; and the Second Rural Development Project, or SRDP). At that time, rural development projects were generally not designed with participatory approaches in mind. The two Sri Lankan projects enjoyed mixed successes, and the evaluation of these projects suggested that the difficulty arose from the use of a “blueprint,” or top-down, model for project design. The project did not necessarily offer useful tools, nor did beneficiaries necessarily want what was offered. Local beneficiaries (farmers, community organizations, households, government workers) were consulted only during the evaluation process, when the defects of the projects became clear. (The rates of return on agricultural products were optimistically estimated to be 24 to 34 percent, but were actually only 10 percent.)

9

The information in this case study is drawn from John English, “Sri Lanka: Kurunegala Rural Development Project and Second Rural Development Project” (http://wbln0018.worldbank.org/oed/inter.nsf/htmlmedia/publications.html).

18

The Sri Lankan experience provided valuable lessons about project design, which have since guided rural development in that country. Here are some of the findings of the Sri Lankan KRDP and SRDP: •

Project planners did not consider how to sustain benefits after projects closed, and so the assets created (irrigation systems, roads) were not regularly maintained. Beneficiaries, who were left out of the planning process, did not take on a sense of ownership of these assets.



User groups, which are crucial to operation and maintenance in rural development, were not part of the project. Farmers saw user groups promoted by the government as outside impositions, and did not participate. User groups succeed when they are initiated by the users themselves.



Management practices in irrigation, again recommended “from outside,” were not followed by farmers, who perceived a limited benefit in doing so.



When farmers were involved in all stages of irrigation work, structures were properly maintained, and conflicts with contractors were minimal. When they were not, quality control suffered.



The project had one objective, and unintentionally provided another. The latter turned out to be far more useful to the community. Although the project sought to diversify crops in the district and increase agricultural production, farmers reported that the greatest benefit of the project was the construction of roads, replacing footpaths. Travel, job opportunities, and access to markets, medical facilities, and schools significantly benefited the community, and raised the average household income some 30 percent—a far more significant impact than the income increases from improved farming output (10 percent).

The Sri Lankan project evaluators drew two principles from these data: •

Participation by stakeholders at all stages, especially monitoring and evaluation. It is essential that M&E be conducted using truly significant variables, rather than those imposed from outside. Variation between the intended outcome and actual outcome (the modest increase in agricultural output, versus the significant impact of building roads) demonstrates the importance of relying on local values for M&E.



Flexibility of design. Differing conditions in a variety of rural areas call for a processoriented, learn-by-doing approach, following one- to two-year rolling plans. In this approach, participatory monitoring is a management tool, to keep the project in line, and prevent it from wasting resources on assets and procedures that are not useful to the community.

19

Field Insight: Indonesia Lack of information, confidence, trust, and clarity about who owns facilities can prevent corrective action from being taken. In a rural water project in Indonesia, a pump remained broken for six months because the government officials in the provincial capital assumed that the well had run dry. When a participatory evaluation team came to the village, Pak Kake, an elderly man whose love of tinkering was well known, was asked why the well was dry. He said, “I have measured with a rope and the water level has fallen to 52 meters, whereas the cylinder hangs at 45 meters. We have not repaired the pump because we are waiting for the chief administrator to give us permission.” When this information was conveyed to the senior government officials in the provincial capital, a technical team visited the site, verified Pak Kake’s observations, clarified ownership rights, and worked with the water group to fix the pump. Within hours, the water was flowing again. Source: Narayan, 62.

2. Activity: A tool for participatory planning

The Participatory Monitoring and Evaluation Learning Wheel, described earlier in this Module, includes these important steps: Establish goals;

Develop indicators; and Take action. When a group establishes shared goals for M&E, it knows what it is measuring. When a group develops shared indicators for M&E, it knows what success looks like.

When a

group takes thoughtful and realistic action as a result of M&E, it addresses problems more effectively. This section offers an activity for learning and enhancing the common thread of these three steps in the PM&E Wheel: determining and setting priorities. Given that all activities have their own costs including money, time, and labor, the implementation of one project or goal means that there are fewer resources left for others. Finite resources mean that it is necessary to

20

prioritize your goals and your indicators. Important factors in setting priorities include: • the urgency and severity of the problem • the possibility the problem could be solved with existing resources • the number and types of stakeholders who will be either positively or negatively affected by the project or activity • the community’s participation • the budget (amount and source) • the sustainability of the development which would result from the project/activity The aim of this section is to provide trainees with basic knowledge and skills in prioritizing, with emphasis on the participation of the community and stakeholders. The following table can be used to compare the merits of various goals of the group, and to prioritize them: Goals

Beneficiari

Urgency

Resource

Impact of

Ranking

(fill in

es

of goal

s required

not

of this

what your

(positive

pursuing

goal

group is

and

this goal

prioritizing

negative)

)

21

(The table can be adapted to take action, by inserting “actions” into the first column.)

22

A table for developing indicators might look like this: Indicators

Who can

How much Resource

Is it

Ranking

and can't

participati

quantitive

of this

use this

on does

or

indicator

indicator?

indicator

qualitative

call for?

?

s required

23

24

Activity 7.4: Prioritization of a project/activity Take a copy of today’s newspaper. Make a list of some of the projects in your community that are reported there. In a small group, choose two projects from that list. Use the chart above to evaluate those two projects. What did each small group discover about its chosen projects? Report your findings to the class.

25

SECTION VI: EXERCISE -- ACTION PLANNING Here you have some time to work through the stages of the Participatory Monitoring and Evaluation Process Cycle in the context of a project or policy which affects your community. Jot down some ideas for steps you could take to complete each phase of the process cycle. Consider ideas you have drawn from case studies and adapt them to your own needs/context. • • • • • • • • • • • •

Conduct a readiness assessment Decide who participates; establish principles for working together Establish goals Develop indicators Gather info (quantitative and qualitative) Construct baseline Set realistic targets Gather info again (quantitative and qualitative) Analyze results including all stakeholders Share info/results with all relevant parties Take action – use the findings! Start over (making any needed modifications in process…)

SECTION VII: ADDITIONAL RESOURCES • •

On-Line Other

26

APPENDIX A: MONITORING MECHANISMS • • • •

Report Cards (Ukraine, Philippines) E-Gov (India, Brazil) Household Surveys (Guatemala, Tanzania) Citizen Dialogues/For a (Brazil)

27



REFERENCES:

1. Michael Bamberger 2. Soniya Carvalho, Indicators for monitoring poverty reduction World Bank Discussion Paper 254, Washington D.C. 1994. 3. Guidance note on indicators for measuring and assessing primary stakeholder participation. ODA/DfID. July 1995 http://www.dfid.gov.uk/public/what/advisory/group7/pdf/sddstak2.pdf 4. David Fetterman, Stanford University. Empowerment Evaluation: Collaboration, Action Research, and a Case Example. The Action Evaluation Research Institute. Go to: www.aepro.org/inprint/conference/fetterman.html 5. John Gaventa, IDS. 6. Jesko Hentschel 7. Sam Kaner et al. 1996. Facilitator’s Guide to Participatory Decision-Making. New Society Publishers. 8. Kusek and Rist 2001. Course for Establishing M & E Systems in Government. World Bank. 9. Mosaic Intenrational Training in PM&E Ottawa, Canada 10. Deepa Narayan. 19993. Participatory Evaluation: Tools for Managing Change in Water and Sanitation. World Bank 11. Deepa Narayan, 1995. The Contribution of People’s Participation: Evidence from 121 Rural Water Supply Projects, World Bank. 12. Lyra Srinavasan et al. 1994. Community Participation Strategies and Tools: A Trainers’ Manual for the Rural Water Supply and Sanitation Sector in Pakistan. UNDP and Govt of Pakistan. 13. Lyra Srinavasan. 1990. Tools for Community Participation: A Manual for Training Trainers in Participatory Techniques. PROWWESS/UNDP Technical Series. 14. PRSP Sourcebook Chapter on Monitoring and Evaluation. 2001. http://www.worldbank.org/poverty/strategies/chapters/monitoring/moneval.htm 15. World Bank Participatory Monitoring and Evaluation Website: http://www.worldbank.org/participation/partme.htm 16. Evaluation of Programmes Promoting Participatory Development and Good Governance: Synthesis Report. DAC http://www.oecd.org/dac/htm/pubs/p-pdggev.htm 17. Evaluation Paradigms: four generations of evaluation (short review of Guba and Lincoln's Fourth Generation Evaluation charting the evolution of evaluation thinking) http://www2.deakin.edu.au/dcad/FTL/evaluation/paradigms/paradigms.htm 18. Guidance note on Increasing the Participation of the Poor in the Assessment of the Impact of Development Interventions. DfID/Actionaid. http://www.mande.co.uk/guidance.htm 19. Participatory Evaluation in programmes involving governance decentralization. UNDP. 20. Participatory Monitoring, Evaluation and Reporting: Online Manual, PACT 21. Larry Salmen

28

Related Documents