Rapid Appraisal Methods To-assess Impact

1

“ Rapid Appraisal Methodologies for Assessing Impact” Notes edited by the consultant and the FAO Evaluation Service

A. General The discussion focussed on Rapid Appraisal Methodologies (RAM) to be used (1) at field level and (2) during a project evaluation mission. The basic feature distinguishing RAMs from more "impressionistic" methodologies is the use of "triangulation" in order to cross-validate all findings. Triangulation means gathering data from (at least) 2 different sources - preferably using 2 different techniques (e.g. key informant interviews, focus groups, observation) – for each variable or issue being studied. The following steps of a rapid appraisal were outlined. They are to be used in approximately this chronological order or, if time and resources allow, concurrently: 1. Literature Review = from technical journals; case studies, demographic data and general reference documents and maps; and locally available project reports. 2. Quantitative data gathering = from statistical documents (central bank, statistical office) available locally and at national levels; project reports; analysis of existing data. 3. Interviews with key informants from both an outsider’s and insider's angle of vision: • •

outsiders may include other donors, competing NGOs, other institutions; insiders can be national and local authorities overseeing the project; project management and front line field and project staff.

During this phase, the objective is to identify underlying issues and develop a first topic list to be used for field investigations. 4. Interviews with focus groups and key informants at regional, local and village/field levels. At this point, Ms. Blumberg emphasized the need for: •

small (5 being the ideal number), homogenous focus groups. Homogeneity within the group being interviewed means avoiding groups composed of people with conflicting interests and/or resources within the community (e.g. large landowners and landless families and, in most places, men and women should be interviewed separately);

•

the use of control groups1 selected from non-project’s clients but with salient features similar to the project's clientele (focus groups);

1

Control groups can help ensure that the results being verified are truly the results of project activities and not due to ongoing events or other externalities. They can also serve as the basis for comparison to measure effects, impact, and changes in living/working conditions brought about by the project (see also para. 4 below). Notes based on the Seminar given in FAO th By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999

2 •

the use of outsider key informants such as staff at health centres, credit/lending institutions and/or schools who are knowledgeable about the community and possess both qualitative information about the local situation and, ideally, lists of clients that can be used for eventual random sampling (see mini-surveys below).

Homogeneity can be achieved by identifying different categories of focus groups distributed according to: agro-ecological areas; gender; age groups; social, religious and racial-ethnic variations within the community. Social, religious and racial-ethnic variations tend to be linked and usually can be correlated with class differences. In the absence of marked social divisions involving caste, race-ethnicity and/or religion, class differences could be initially identified at three levels: the landless/semi-landless; those that possess a moderate level of resources (land, livestock, shop, etc); and the elite - rich and/or village authorities (religious, administrative, spiritual, traditional, etc). 5. Field Observation = used to verify what is actually being done by the participants themselves (e.g. the level/quality of use of land conservation practices, use being made of outputs); tangible measures of standard of living (such as type of housing, household possessions); and, what has been achieved versus what is being reported. 6. Feedback and verification of mission's findings back and throughout the pyramid of stakeholders starting at village level. At this first level, findings and issues that have emerged are shared at a village (large) group meeting to seek confirmation, assent and ideas for potential use and/or reforms. 7. Feedback, verification and discussion of mission's findings at institutional level. Discussions here focused on the use of multiple stakeholder brainstorming sessions and inter-institutional workshops to: • • • •

evaluate the validity, relevance and potential use of results with insiders and outsiders; and, assist in future work-planning and detect administrative, or other, obstacles to overcome; identify additional success criteria; and discern measures to ensure the sustainability and/or "institutionalization" of results.

8. Finally, the possible use of mini-surveys, at the end of the investigative process, was noted. These would be composed of a few well-defined/focused questions, and should preferably involve random sampling and control groups. Mini-surveys would be used as a way to: • • •

establish parametric data about the people in the project, especially those with complex social settings where it is difficult to identify the right weight to give to the various social components and focus group distribution; answer questions that remain open; quantify findings and/or provide quantitative information in support of these findings. Notes based on the Seminar given in FAO th By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999

3 N.B. Mini-surveys should not be used for general information gathering, such as identifying broad socio-economic features of the project clients. Instead, mini-surveys should be focused on specific issues and targeted on pertinent groups.

B. Rapid Appraisal Methodologies for Assessing Results (for M&E at Village level) 1. At this level, results need to measured according to success criteria that come, at least partially, from within the target groups. Therefore, very early in the project cycle, village focus groups need to be selected to develop indicators of success and the consequent M&E system. These focus groups should help suggest the M&E data to be collected to followup progress; and to help select new data to be included in the system as activities develop. For this purpose, a system of semi-annual or annual focus group meetings can be established to collect information/data for indicator measurement (results and impact) and to detect problems. N.B. Participants of the focus groups may change at each data gathering exercise, or be maintained as panel members. The choice is to be made on a case-by-case basis either to ensure continuity of the information gathered; or to guarantee the validity of the findings by enlarging the number of people covered by the reporting system. However, there are generally more drawbacks to the panel method in these rural settings. 2. Control groups need to be utilized to serve as the basis for comparison from which to measure changes brought about by the project. Control groups can be derived from focus groups in a “snowball sample” by asking one or more focus group participants to suggest persons similar to themselves but who are not involved in the project. This helps ensure similarity in wealth, education and other characteristics with the focus groups. They can also be chosen at random from adjacent villages not covered by the project if sampling lists are available. In either case, they can provide comparative information on indicators such as: • • •

the spread (use being made) of results into non-project areas; unanticipated negative effects of the project not found in control groups; the incidence of externalities in project results, e.g. a general economic downturn or boom, or drought, etc. in the whole area, affecting both project and non-project households.

3. Impact indicators to be gathered at ultimate target group/village level can estimate three main impact areas: • •

what has happened to the business/individual plot area? what has happened to the participant/stakeholder including increases/decreases in empowerment and wellbeing?

Notes based on the Seminar given in FAO th By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999

4 •

what has happened within the households including increases/decreases in children’s (male and female) nutrition, education and health; changes in the division of labour, time constraints, etc.? N.B. Men tend to spread increased income under their control over a wide range of activities, whereas women with increased income under their control tend to spend it in a much more focussed manner on: a. improved family diet b. improved education for children c. improved healthcare. 4. The outsider helping to set up the results-oriented M&E system should be accompanied to the field by at least two project staff members who will be in charge of maintaining the system and, if at all possible, their boss, in order to convert him/her into a stakeholder. During the focus group phase, several intelligent, dynamic participants should be recruited and trained to help facilitate subsequent waves of focus groups, as well as to join the staff of M&E people in orally presenting the results of the latest focus groups to project management. 5. Data collection should be done regularly (e.g. by semester). If no M&E staff person is present at the focus group, and the village facilitator is not totally literate, a designated M&E village representative, who can be a child or youth of the village, a school teacher, or other interested and literate volunteer, can take notes on the focus group discussion. N.B. Villagers should progressively become stakeholders in the M&E system. •

Interested and potential candidates for the data collection exercises should be identified and trained (while on-the-job) at each field visit.

•

Village reporters should work with project staff to convert findings and data gathered into the more formal institutional M&E system.

•

Data gathering should encompass some form of remuneration and recognition, e.g. transport costs; honours; certificates, etc.

C. MULTIPLE STAKEHOLDER ANALYSIS The steps and criteria for identifying multiple stakeholders are somewhat similar to the selection of focus groups. However, stakeholders start at the top of the pyramid at national/donor level and proceed down to the local/village levels as follows:

Notes based on the Seminar given in FAO th By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999

5

TABLE 1: Category

Insider Stakeholders

(Control group) Outsiders

1. National/donor level – line ministry/donor – admin. Authorities

1. Other donors/NGOs

2. Regional/local level

2. Staff from NGOs, private sector, other/similar projects

3. Project management & front line staff – extension workers and local health/school workers

3. Local front line staff – extension workers and local health workers, school teachers

For each agro-ecological zone:

From nearby and similar areas:

4. Community leaders & large landowners – traditional, religious 5. Middle stratum -men - average households

3. Middle/lower stratum men

6. Lower stratum -men - poor/landless families

4. Middle/lower stratum women

7. Middle stratum -women - average households 8. Lower stratum women - poor/landless families 9. Youth (male and female) – from within the project 10. Other minorities/ethnic classes (covered by the project) – if not represented in the above classifications

5. Other minorities/ethnic classes (covered by the project) – if not represented in the above classifications

N.B. This minimum of 15 categories of focus groups containing a wide range of stakeholders needs to be covered. Additional stakeholders can be identified as the fieldwork progresses, in consultation with management and project staff and/or the various "outsider" categories. It is important that the information given by project-related focus groups be crosschecked through group meetings with a control group (or groups). Additionally, if time and resources permit, attention should be paid to on-the-job training of a followup M&E team (at least two staff members, a member of project management, and selected stakeholders at village level). Finally, early in the project cycle, the minisurvey questionnaire to be carried out if and when authorities at country level so warrant it.

D. PROGRAMMING RAM INTO EVALUATION MISSIONS Notes based on the Seminar given in FAO th By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999

6

Nearly all of the requirements for the use of RAM can be planned and written into the TORs and mission field visits, in close consultation with project and national authorities. 1. At national level, a large briefing meeting (to take place, if possible, on day 2) can bring together all of the authorities and donor representatives engaged in project activities and reduce the number of purely "formal" visits. However, it remains necessary to complement this briefing meeting with a few key informant interviews to further delineate the important issues. 2. From the above, a list of topics for further investigation can be drawn. This list, together with the main findings and issues emanating from the literature review (see section A.1) will form the basic list of questions to serve as a guideline for subsequent key informant interviews and focus group meetings (see below). These questions can be modified based on the results of early interviews. 3. After travelling to the field, a similar regional/local initial meeting can be arranged to bring together a large number of stakeholders. It can include government and NGO staff, private sector representatives and local authorities. N.B. Discussions at these initial meetings focus on the objectives and scope of the project and its relative comparative advantages and relevance to the country/sector/region. 4. Once in the project's headquarters, an initial "brainstorming" session with all of the project staff can be organized. Its purpose is to review together the weaknesses and strengths/benefits and achievements of the project. Another objective is to recruit people who will eventually become the stakeholders in the evaluation process i.e. who will have a continuing commitment to an ongoing M&E system. 5. At village level, selected focus group and key informant interviews need to be planned beforehand to encompass the various categories and classes of stakeholders. This is of course to be done on a case-by-case basis according to the nature and geographical coverage of project interventions. • However, in most cases at least the 10 main categories of "local stakeholders" (as well as 5 control groups), per agro-ecological zone, should be programmed into the field visits. • Focus group interviews can be carried out by one or more of the mission members in an iterative, interactive, question and answer mode with all of the focus group members (see section A.4). Ideally, all focus groups in the first round of data collection will also be used to familiarize the project staff who will manage the ongoing M&E system, as well as the selected villagers, with the techniques involved in running successful focus groups.

Notes based on the Seminar given in FAO th By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999

7 • Specifically, the mission should seek to provide on-the-job training to at least two staff and two stakeholders at village level in order to leave behind a follow-up team for continuous M&E. Feed-back 6. Feedback throughout the various layers of the "pyramid" needs also to be programmed through a series of group meetings to be used for restitution, cross-checking, validation and eventually negotiation of the mission's main findings. 7. At village level a town/local meeting can be programmed with members and non-members of project activities. Discussions should lead to an understanding of weaknesses and strengths present in the project, as well as to the planning of consequent and agreed upon follow-up actions. 8. At national/regional level, an inter-institutional evaluation workshop can be planned with representatives of insider and outsider organizations, as well as project personnel ranging from field staff to top management. • At both levels, discussions should lead to a better understanding of evaluation requirements and a commitment to maintaining an M&E process as feedback to both the community and the implementing institutions. • In both cases, project results are discussed to identify: a. The specific success criteria by which each of the stakeholder groups judge the success of the project; b. The use being made (or to be made) of project results i.e. potential effects; and c. The potential, opportunities and limitations to ensure continuity, sustainability and/or institutionalization. • In both cases, PRA visualisation techniques may be introduced, presented and facilitated by the team members, in order not to taint the self-evaluation process and initiate a process by which stakeholders have an interest in the continuous review of project results.

Notes based on the Seminar given in FAO th By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999

8 Annex 1

Impact Evaluation PRA Sequence • •

Review of Secondary Data Desk review In-country sources and grey literature

• • • •

Key Informant Interviews National level (insider/outsider) Regional (government, private sector, civil society) Front-line community workers (extension, etc.) Inter-institutional Seminar *

•

Zoning and Purposive Sampling of Communities Agro-ecological, ethnic, production systems

• • •

Key Informant Interviews at Community Level Leaders (traditional, modern) Upper stratum (rich households, leaders) Outsiders (schoolteacher, health post)

• • • • • •

Focus Group Interviews with Project Beneficiaries Middle stratum (average households) Males Lower stratum (poor households) Males Middle stratum (average households) Females Lower stratum (poor households) Females Youth groups (Male and Female) Representatives of Other (ethnic minorities, etc.)

• • • • •

Focus Group Interviews with Control Group ** (non-participants in project; non-beneficiaries) Middle stratum (average households) Male Middle stratum (average households) Female Lower stratum (poor households) Males Lower stratum (poor households) Females Other (ethnic minorities, youth, etc.)

• • •

Supplementary Cross-checks Field Observation Content analysis of newspaper articles PRA exercises within a Community Setting Notes based on the Seminar given in FAO th By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999

9

Household Interviews (few)

• • •

Feedback to Community at Village Meeting For Cross-checking and validation Restitution of Findings Negotiation of possible solutions

• • •

Feedback to Project Staff To Front-line workers (extensionists, etc.) Heads of line departments Project Management

• • • •

Quantitative Survey ("last Step") Parametric data To cover unanswered questions or To fill gaps in data To Substantiate Controversial findings

• • • •

Feedback to National Level National project team Minister Donors (FAOR, UNDP, etc.) Inter-institutional Seminar *

* PD - The Inter-institutional Seminar can be held ex-ante for impact evaluation to review: • validity and pertinence of project goals and timing • level of dissemination & institutionalisation of project results • the probable effects of externalities and the nature of these on project results • success criteria to be considered in addition to foreseen criteria. The Inter-institutional Seminar can be held ex-post to the field visits to review: • validity and pertinence of project goals to national policies and within the national context • validity and pertinence of project activities to ongoing efforts in partner institutions • level of dissemination & potential for institutionalisation of project results • success criteria to be considered in addition to foreseen criteria

Notes based on the Seminar given in FAO th By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999

10

** Focus Group Modality = no more than 5 group members, through a semi-structured interview with open and closed questions derived from the former 4 steps.

Facilitator / Interviewer

Seasonal Calendar Village Diagram/Transect Value Matrix Focus Group male/female, adults/youth, participants/non-participants, etc.

Notes based on the Seminar given in FAO th By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999

11 Annex 2

How People May Participate in Development Programmes and Projects2 1. PASSIVE PARTICIPATION People participate by being told what is going to happen or has already happened. It is a unilateral announcement by an administration or project management without any listening to people’s responses. The information being shared belongs only to external professionals. 2. PARTICIPATION IN INFORMATION GIVING People participate by answering questions posed by “extractive” researchers using questionnaire surveys or similar approaches. People do not have the opportunity to influence proceedings, as the findings are neither shared or checked for accuracy. 3. PARTICIPATION BY CONSULTATION People participate by being consulted and external agents listen to their views. These external agents define both problems and solutions, and may modify these in the light of people’s responses. Such a consultative process does not concede any share in decision making and professionals are under no obligation to take on board people’s views. 4. PARTICIPATION FOR MATERIAL INCENTIVES People participate by providing resources, for example labour, in return for food, cash or other material incentives. Much on-farm research falls in this category, as farmers provide the fields but are not involved in experimentation or the process of learning. It is very common to see this called participation, yet people have no stake in prolonging activities when the incentives end. 5. FUNCTIONAL PARTICIPATION People participate by forming groups to meet predetermined objectives related to the project, which can involve the development or promotion of externally initiated social organization. Such involvement does not tend to be at early stages of project cycles or planning but rather after major decisions have been made. These institutions tend to be dependent on external initiators and facilitators, but may become self-dependent. 6. INTERACTIVE PARTICIPATION People participate in joint analysis, which leads to action plans and the formation of new local institutions or the strengthening of existing ones. It tends to involve interdisciplinary methodologies that seek multiple perspectives and make use of systematic and structured learning processes. These groups take control over local decisions and so people have a stake in maintaining structures or practices. 7. SELF-MOBILISATION People participate by taking initiatives independent of external institutions to change systems. They develop contacts with external institutions for resources and technical advice they need, but retain control over how resources are used. Such self-initiated mobilisation and collective action may or may not challenge existing inequitable distributions of wealth and power.

2

Excerpt from ”Regenerating Agriculture - Policies and Practices for Sustainability and Self-Reliance”, Jules N. Pretty, Earthscan Publications Ldt., London, 1995. Notes based on the Seminar given in FAO th By: Professor Rae L. Blumberg, FAO, Rome. May 18-19 1999