Dfid Project Cycle Management Impact Assess

IMPACT ASSESSMENT & PROJECT MANAGEMENT CYCLE (PMC)

IMPORTANT NOTE: Most of this section will be familiar to DFID staff as it is a paraphrase of information contained in Office Instructions, with added commentary on the practical application of those instructions in the context of Enterprise Development interventions. The purpose of summarising the main points here is to enable NGO's, other partner organisations and consultants working with DFID to understand DFID's progammme management procedures, and to place Impact Assessment in the context of these procedures. Further information can be obtained from the Office Instructions which are now reproduced on the DFID website, and on CD (address given on website)

CONTENTS: Summary 1 Introduction 2 Project Header Sheet 3 Project Concept Note 4 Stakeholder Analysis 5 Logical Framework 6 Project Submission: Project Memorandum 7 Monitoring requirements 8 Reporting requirements 9 Review procedures 10 Sample Logical Framework: TEEM (Traidcraft Exchange), Malawi SUMMARY: Most of this section will be familiar to DFID staff as it is a paraphrase of information contained in Office Instructions, with added commentary on the practical application of those instructions in the context of Enterprise Development interventions. The purpose of summarising the main points here is to enable NGO's, other partner organisations and consultants working with DFID to understand DFID's progammme management procedures, and to place Impact Assessment in the context of these procedures. The paper goes through the PMC in chronological order, looking at how impact assessment should be built into each stage of the cycle, beginning with appraisal and completion of the Project Header Sheet, through development of the logical framework, project memorandum, to Project Completion Report. This text has been prepared by Tertia Gavin and Caroline Pinder

1

INTRODUCTION:

Enterprise development is about a series of different interventions, involving a wide range of possible partners, over varying timescales, but all with the common underlying goal of contributing to the elimination of poverty. Interventions may range from macro to micro level, with different implications for enterprise development impact assessment at each level. Working at different levels may involve DFID in partnerships with different types of organisations, ranging from multilaterals to other bilateral donors, international NGOs, banks, professional organisations and local NGOs. Whatever the nature of the partnership, there are some common underlying principles for success. The key factor is to develop a good working relationship with the partner organisations, based on trust and good communication. Different methods will be required depending on the nature of the partner, for example, some partners may feel comfortable using logframes as a tool for exploring and agreeing on important programme issues. Other partners, may find this formalised approach difficult to work with, and may prefer methods commonly used within their own organisations. Good communication, involving a listening role by DFID, is essential from the beginning to ensure that partners do not feel that an outside agenda is being imposed on them. Channels of communication will again vary depending on the nature of the partner and the type of intervention. It is vital that ways are found to ensure honest and open communication that will allow an atmosphere of lesson learning and problem solving. Frequently during interventions, unexpected outcomes arise (both positive and negative) and it is important that partners feel comfortable raising these issues with DFID as and when they become evident, so that their implications for the programme can be considered in time for review.

Table 1: Levels of Intervention

Level

Key areas for intervention

Goals

Implications for Impact Assessment

Macro Government / administration

Economic policy reform; trade policy; regulatory environment, eg: CDF, PRSPs1

General legal and fiscal framework of conditions oriented to enterprise and competition

DFID usually only one player among several multi-laterals and bilaterals. Has an influencing role, so IA more difficult to control

Organisational and Institutional development, eg strengthening of representative agencies as part of support for civil society

Targeted policy implementation, eg job creation in labour intensive sectors, income safety nets and re-training programmes

DFID sometimes one of several donors; other times sole or major donor. Need to take partners' agendas into consideration and to develop coherent IA framework with partners

Starting and expanding individual SMEs; financial and non-financial support services

Job and selfemployment creation; income generation

DFID often sole or main donor, and able to set agenda for IA. Eg: micro finance, BDS institutions

Meso Business associations; trade unions; trade/industrial sectors;quangos national & international NGOs

Micro SMEs; local NGOs/service providers

1

CDF = Comprehensive Development Framework; PRSP = Poverty Reduction Strategy Paper. For more information refer to the World Bank website.

FIGURE 1. PROGRAMME CYCLE: AREAS WHERE IMPACT ASSESSMENT NEEDS CONSIDERATION IDENTIFICATION

Look at EDIAIS for lessons from other programmes *PROJECT HEADER SHEET (PIMS, POM, PAM) • Decision on markings give indication of which TSP checklists to consider(eg gender, poverty etc) *CONCEPT NOTE • Indicate consideration of level and methodology of IA

DESIGN

*STAKEHOLDER ANALYSIS • Which stakeholders will be involved in proposed IA? Are these the appropriate stakeholders? What methodologies are best for securing their involvement? *LOGFRAME • Agree purpose of IA and indicators • Agree methodologies and mix to fit • Agree on baseline data required, & how this will be collected • Agree frequency/timing of IA

APPRAISAL

*PROJECT MEMORANDUM (& Technical Annexes) • Ensure IA reflected throughout document including budget • Collate multi-disciplinary tech annexes (in particular, economic, social and environmental); negotiate and agree IA strategy for project with multi-disciplinary Advisers/team members

APPROVAL

*MONITORING ANNEX • Detail proposals as agreed at earlier stages, ie purpose, methods, frequency etc

IMPLEMENTATION AND MONITORING

*MONITORING . DFID staff monitoring visits at least once a year . Scoring for PRISM *REPORTING • internally written reports by project staff required at least once a year for projects over £250 000 reporting against logframe indicators, activities and budget • End-of-project report will include IA findings

REVIEW / IMPACT ASSESSMENT

*REVIEWING • Mid-term review should include a mid-term IA • Output-to-purpose review held 6mths before end of project; should include full-term IA • If extension likely review IA strategy & include in new PM(Project Memorandum)

The outputs, or what happens on the ground, is clearly the most important part of programme cycle management. To ensure that the desired outputs occur, thorough preparation is essential, and this will be reflected in the project documentation. The following notes are intended as a guide for project preparation and subsequent monitoring and reviewing. It is not intended to suggest that documentation and paperwork are the most important aspects of programme management! 2

PROJECT HEADER SHEET

This requires the Enterprise Development Adviser to classify ('score' or 'mark') the proposed intervention in three ways2: 2.1

Policy Information Marker System (PIMS)

PIMS is an indicator of initial intent but scope exists within the system to change markers during the course of project implementation. The markers provide a policy relevant framework against which project performance can be judged. However, the main value of PIMs in in providing a measure of the extent to which projects are being targeted to pursue key policy areas, and it is therefore most relevant at the pipeline and commitment stages of projects. Most enterprise development interventions will fall within the PIMS framework for "Policies and actions which promote sustainable livelihoods". 2.2

Policy Objective Marker (POM)

There are however, three categories of PIMS objectives of which "Policies and actions which promote sustainable livelihoods", as mentioned above at 2.1, is likely to be the one into which will fall most enterprise development interventions. The other two objectives are "Better education, health and opportunities for poor people" and "Protection and better management of the natural and physical environment". Classification under just one of these headings is not easy. Sustainable poverty elimination requires a dynamic balance between the promotion of each of these three objectives, and projects often contribute to more than one objective. Each of the three broad objectives is therefore broken down further, into POMs. Within the broad category "Policies and actions which promote sustainable livelihoods" the objectives (POMs) most likely to apply to enterprise development programmes are: Sound social and economic policies Direct assistance to the private sector Access of poor people to land, resources and markets 2

This section is drawn from "PIMS" produced by DFID Statistics Dept, March 1998

Business partnership Training and skills development 2.3

Poverty Aim Marker (PAM)

The PAM system comprises three categories for the way in which poverty elimination is assessed: • •

•

enabling actions which under-pin pro-poor economic growth and the broader policies and context for poverty reduction and elimination, and will lead to social, environmental and/or economic benefits for poor people inclusive actions (eg sectoral development programmes) which aim to benefit broad-based population groups, including poor people, but also address issues of equity and barriers to participation or access of poor people actions focused predominantly on the rights, interests and needs of poor people

In many cases projects will fall into more than one PAM category, but Advisers are required to indicate the predominant means by which individual projects target poverty elimination. 3

PROJECT CONCEPT NOTE

The requirements for the Project Concept Note (PCN) are set out in Office Instructions. Many of the important points relating to consideration of impact assessment in the PCN are discussed below at Section 5, the Logical Framework, which must be submitted with the PCN. Critically at this early stage, some consideration is required of baseline data against which subsequent impact can be measured. Frequently, this process is neglected which has significant negative repercussions for future impact assessments, as is evident in several of the case studies (eg MDA in South Africa, AGENT in Zimbabwe – forthcoming link). There is an increasing trend with 'process' type projects to have a series of phases, which often start with a feasibility study (for example, Training for Enterprise and Exports in Malawi (TEEM), a Traidcraft project). This often provides an opportunity to collect information about baseline conditions, and make an early exploration of indicators with future stakeholders.

4

STAKEHOLDER ANALYSIS

A separate Technical Note describes how to carry out a Stakeholder Analysis (SHA). To briefly summarise, an SHA should be carried out at the design stage of every project and programme with a view to: • • • •

Identifying key stakeholders, their interests and potential degree of influence on the project Identifying areas of potential conflicting interests, with a view to minimising these and/or looking at ways for ensuring everyone has something to 'win' from working together Drawing out key risks and assumptions which will need to be taken into account in the project design Identifying possible areas of Impact Assessment: likely indicators based on stakeholder's perceptions and expectations of impact

5

LOGICAL FRAMEWORK

Box 1: From: “CARE INTERNATIONAL SOUTH WEST AFRICA REGIONAL MANAGEMENT UNIT (SWARMU) :Developing Design of Monitoring & Evaluation (DM&E) Capacities “A major weakness is the ability of project staff to utilise their logframe for designing a coherent and integrated, overall information system, where a manageable and limited number of feasible information activities are planned, which together will ensure that effective effect and impact level monitoring will occur. It is typical for projects to end up collecting too much rather than too little information. Frequently though, much of this information is not relevant to monitoring the results and impacts for which the project is accountable, and that which is, is not collected sufficiently reliably or regularly. By restricting the number, but improving the quality and reliability of their major information gathering activities, projects will much improve their information systems.” The logical framework is a key component of the approval process for DFID funding. It can also be a helpful tool in working through issues between stakeholders - whatever the nature of intervention – from a micro-level project to a less tangible, macro-level intervention aimed at, for example, influencing international trade relations or government economic policy. Although the logframe can sometimes be used to shape discussions, some partners may find the logframe approach difficult to work with, and other methods will need to be used to explore the issues with these stakeholders. Development of the Project Framework should involve primary beneficiaries (ie the poor people who will be the ultimate users of the services provided by the intervention) as well the intermediary partner organisations responsible for delivering those services. The Project Cycle Management (PCM) approach emphasises the importance of participation of stakeholders in a "meeting of minds" to agree the project purpose. At the start those stakeholders who should be involved in the development of a framework must be identified, while recognising that, in turn, these stakeholders may identify others who should contribute to the process. The PCM approach can be used most easily between donor and recipient (partner) institutions but the broad elements of the approach should also be used with all other interested parties. (See the technical notes on Stakeholder Analysis and Participatory Methods for more information about how to involve stakeholders in a participatory way) In designing the Project Framework the principal stakeholders need to be involved in: • • •

agreeing objectives, outputs and activities determining and weighting risks (assumptions) identifying indicators for measuring progress.

Different stakeholders may well select different indicators. Minimally there should be some DFID-driven indicators, some from the main implementing

agency and some from the primary stakeholders: in other words the range of indicators should reflect the various critical expectations of all the key stakeholders. The experience of several NGOs has shown that participatory processes using people's own indicators can generate quantifiable and timebound indicators which relate to outputs relatively easily. Box 2: From: Central Region Infrastructure Maintenance Programme (CRIMP Malawi). A baseline study was carried out early in 2000. Using CARE's Household Livelihood Security (HLS) framework, the study aimed to accurately identify the percentage of households in different wealth/livelihood categories, and to develop an information system that is responsive to the needs of CARE, donors, community and partners, and can be used by communities to monitor change over time. The study was conducted in 36 villages across the five Traditional Authorities over a period of three weeks by a team of 24 researchers. The researchers were oriented in the research protocol over a one-week period proceeded by pre-testing of the tools in the community before the start of data collection. A total of 709 households were interviewed over the five TAs. In addition social maps across the TAs incorporated basic data on 2,032 households. As a result of the pre-test, a standard set of indicators was developed for the differentiation of households into livelihood categories. These are: • • • • •

Land Food Security Assets Income Sources Education/skills/capabilities

See case study for illustration of how these indicators are used. Although indicators of purpose and goal are more difficult to develop in a participatory fashion, it is possible; and in fact debates with other stakeholders on this issue have often started to show up differences in objectives not only between beneficiary stakeholders but between them and the implementing agency, leading to a changed project purpose. Clearly it is better for these differences to be aired at the start so that accommodation can be made for the full range of partners' views, rather than allow them to be a potential source of conflict at a later stage. The extra time spent in developing key indicators of achievement with stakeholders is an important investment in the success of the project, leading to a sense of greater shared commitment among the various stakeholders. Local intermediary organisations should be identified who would be responsible to the implementing agency for much of the monitoring, with DFID providing support and technical assistance as necessary.

The project framework should be seen by all parties, as a working document that can be adapted as necessary during the life of the intervention. Many partner organisations do not recognise the flexibility of the logframe, and regard it more as a restrictive device against which their performance will be judged. It is important that a clear understanding is reached that the logframe is an active part of management and can be changed as appropriate throughout the intervention to reflect changing circumstances. Even with the most carefully designed intervention there is always the possibility of unexpected outcomes both positive and negative. These need to be identified and recognised throughout an intervention, and unexpected positive outcomes reinforced, and methods sought to reduce negative impacts. Table 2: Summary of keypoints for consideration of Impact Assessment in the Logical Framework Goal and purpose: • Are these shared and understood by all stakeholders?

OVI

MoV

Assumptions

•

Are target beneficiaries mutually agreed upon? What is purpose of IA? What are indicators? Have these been selected in consultation with stakeholders? Will they provide necessary information for purpose of IA? Are they consistent with other partners requirements?

•

Who is responsible for collecting information? Is there a clear timetable Has necessary baseline data been collected? Has monitoring and IA been budgeted for? How will info feed back into project?

•

Do these include scope for negative and positive unexpected outcomes

Have indicators been agreed with stakeholders? Do they include a mix of DFID, partnerorganisation and stakeholder indicators?

•

Have local intermediaries been identified to carry out IA? Has a timetable for IA been agreed? Will this provide the necessary information? How will info feed back into project?

•

What are the risks if any, which might impact on the achievement of outputs? Does achievement of impacts/outputs depend on any particular institutional or other factors?

Have sufficient funds been allowed for

•

• • •

•

•

Outputs • What are the expected impacts?

• •

• • • •

• • •

Activities • Do these include necessary inputs for IA?

•

Is collection of baseline data an identified

•

•

Do inputs and activities depend on any particular

•

•

activity? Has implementation of MIS been included? What arrangements are made for ongoing collection & review of data

baseline study, setting up MIS, ongoing collection and review of data?

factors?

6

PROJECT SUBMISSION

The required format for project submissions is set out in Office Instructions Volume II D6 (forthcoming link to DFID site with OIs). This format is required for all types of DFID funded intervention (other than those being jointly funded through Civil Society Department). Consideration of IA is required in a number of different sections of the Project Submission (also often referred to as Project Memorandum). It should not be thought of as merely a subsection of the document under the heading monitoring; rather it should be integrated throughout. Here is a checklist of IA requirements within the text of the submission: 6.1

Summary and recommendation • • •

6.2

Background • •

6.3

What will be the outputs (ie expected impacts)? Who will benefit? Has a comprehensive stakeholder analysis been done? Project budget: Does the budget provide scope for effective IA activities?

What has been done before by DFID and by others? Check EDIAIS database for relevant IAs within DFID/elsewhere What else needs to be done prior to implementation? eg - What baseline data needs to be collected to assist with meaningful IA during project? - Who is responsible for baseline data collection? Is there a workplan setting out tasks, personnel and deadlines?

Project Approach •

What are the expected benefits? What impact is the project hoping to achieve? • Who are the key stakeholders and how has participation been handled? • What are the options, if any, for review and change during implementation? - Are there appropriate indicators? - Are there realistic means of measuring these indicators? - Is there a clear plan setting out who is responsible, how information should be collected, who should compile and analyse it, and how often it should be reported? - Where possible are these built into organisations' MIS?

- Is training required for partner staff to ensure effective use of MIS? - Who is responsible for collecting information and reporting on them? - Are there plans for regular stakeholder meetings for lesson learning and forward planning? 6.4 • • • •

6.5

Evaluation What lessons of best practice are being applied? Consult EDIAIS database for relevant projects How will the indicators for Goal, Purpose and Output levels provide the basis on which to judge success? Do indicators reflect careful consideration of mix of IA methodologies needed? (refer to Core Text for more information) Will indicators assist internal learning throughout the life of the project as well as being a means of judging project success at it's completion? Implementation i) Management arrangements • What will be the role of key stakeholders? • How will their participation be handled? • How will they be involved in IA? ii) Timing • Is collection of baseline data allowed for before project start • How often during the project will external IA exercises be required? iii) • • • • •

Inputs Are additional inputs associated with IA specified? Is facilitation required for regular stakeholder meetings? Is training required for MIS systems? Are regular monitoring visits set out? Have funds been budgeted for training, facilitation and other needs identified? Has sufficient time been allowed for staff to do monitoring? Or will there be a special appointee to carry out M&E work?

iv) Monitoring • Who gathers what information and when, for chosen indicators, at each level of the logical framework?

• • •

What management arrangements will ensure that action is taken on the results of the monitoring? Who reports to whom and when? What arrangements will be made for Output to Purpose Reviews?

v) Technical Annexes • MIS may need to be covered in some technical annexes (eg micro-finance projects usually have a technical annex detailing the types of ratios and other financial information that should be build into MIS from the start) Box 3: Central Region Infrastructure Maintenance Programme (CRIMP), Malawi Integration of Monitoring Work into Staff Time The project has a Monitoring, Research, Evaluation and Documentation Coordinator who is responsible for the collection of data, analysis and reporting on a quarterly basis. Sometimes the project hires part time research assistants to help with the collection of livelihood monitoring data. This is funded from the Monitoring and Evaluation budget. The Monitoring, Research, Evaluation and Documentation Co-ordinator has the primary responsibility of ensuring that appropriate information is collected systematically and used to inform programme activities. Staff implementing the project activities contribute to the monitoring process through reports from which qualitative information is drawn. The qualitative information from the reports, to a large extent helps to explain the quantitative information collected during the scheduled (quarterly) monitoring exercises. The qualitative information is stored separately to the quantitative, but in a way that is accessible to any user – on the server hard drive, and filed in hard copy form. The findings from the monitoring exercises continue to guide and inform the implementation processes adopted by the project. The information collected is also be used to inform the design of the second phase of the project.

6.6

Monitoring Annex This should set out: • • • •

Purpose of IA Mix of IA methodologies, and how they will be integrated What the partner-organisation is expected to produce as part of internal MIS Capacity building required to ensure effective MIS/monitoring

• • • Box 4:

Necessary external IA inputs, (who? how often?) Management processes to ensure IA findings feedback promptly into project Dissemination of findings beyond project EXAMPLE OF DETAILED MONITORING ANNEX

Annex F of the TEEM (Training for Enterprise and Export Malawi) Project Memorandum lays out the broad areas of impact which will be investigated and disaggregated through the life of the project. It creates an innovative set of research questions in pursuing the impact at the following levels: • impact of business development services offered by the Business Service Centre • impact of Trading Company on SME production • impact of the project as a whole on poverty. Each of these levels require careful methodologies not only to ensure impact is correctly measured and attributed, but also that valuable lessons are fed back into the project to enable adaptation and innovation. The methodologies should also ensure that valuable information is integrated from TEEM’s internal MIS, and that these are correctly designed to provide complementary data.. The implementation of these studies should be externally tendered and managed by DFID in conjunction with TX and the local TEEM institutions.

7

MONITORING

This is undertaken by DFID staff (including field managers) together with project management and focuses on progress to achieving outputs and purpose. It is often viewed by stakeholders as a policing exercise and can cause suspicion and distrust. With sensitive handling, however, monitoring should be an opportunity to assess project progress against various levels, and more importantly give the opportunity of face to face contact with key stakeholders which can be an important opportunity for lesson learning. Monitoring visits can help assist a good relationship between DFID staff and local stakeholders. With the improvement of channels such as e-mail, ongoing communication should be encouraged to help identify and deal with problems and queries as they arise during project implementation. This can help to foster a sense of partnership between colleagues, as opposed to an “us-andthem” mentality. This is more likely to be the case if implenting agency staff have been involved from the start so that they do not feel that they are undergoing job performance assessments. The Project Memorandum must include the plans for monitoring. This will usually include visits by DFID staff to project sites and regular meetings with project management, as well as arrangements for monitoring the project's progress through perusal of the reports provided by project staff. It is preferable that monitoring visits are undertaken with the recipient's project management staff. Also, as far as possible, the same DFID staff or consultants should undertake monitoring for each project, to ensure continuity and to allow the building of relationships between project personnel and DFID. Monitoring visits are not only time consuming for DFID staff, but also require a substantial investment of time by project management and often a variety of stakeholders. It is therefore particularly important that when planning monitoring visits, while having to fit them into the schedules of DFID staff, consideration is also shown to the project staff to allow them the necessary time to plan for the visit. It also underlines the importance of ensuring that the monitoring visit is seen as a useful exercise for all participants, rather than as an onerous chore imposed by donors. This can be facilitated by taking time in advance to find out whether there are any particular issues that stakeholders wish to raise and by ensuring that sufficient time is allowed for a thorough debriefing and discussion of issues before leaving the project. It is also important to ensure that monitoring reports are issued swiftly with clear indications of what actions are required by whom. Stakeholders, including project staff can feel let down after monitoring visits, if expectations are raised and no action subsequently taken.

Conventional periodic review missions often involve a large DFID interdisciplinary team and representatives from a range of recipient institutions joined, in co-financed projects, by representatives from other donors. Meetings often concentrate on senior bureaucrats in urban centres, which means that the viewpoints of beneficiaries, representatives of local-level institutions and of field workers are not taken into account. Because such review missions tend to be large, hierarchical and formal, even field visits fail to establish any real communication with junior officials and primary stakeholders. Strategies to overcome these problems include: • designing review missions as a series of shorter, more varied, visits, rather than an all-inclusive single visit: one component of such a review would be a preliminary participatory field visit, using participatory appraisal techniques; • avoiding large and, for some stakeholders, intimidating meetings where lower level staff or women beneficiaries may be afraid to speak out. Smaller, more informal discussions, perhaps on the lines of a focus group, may help to elucidate more "truthful" opinions.

8

REPORTING

As set out in Office Instructions, these are usually completed by non-DFID staff (eg contractors, consultants, project staff) and compare progress of project implementation with what was planned at the project output and activity level. The report should also cover any unexpected outcomes that have arisen as a result of the intervention, as well as progress towards the planned outcomes. The project management must have a work plan agreed with DFID and the recipient which is: • •

developed from the work breakdown activity chart in the Project Memorandum; regularly reported against and updated.

The work plan content should: • •

relate to the Logical Framework include activities needed to gather data against which to assess progress in relation to agreed targets/indicators in the logical framework, and reporting frequency.

The format will require reporting against the logical framework and the work plans to review progress against activities and outputs (and purpose where appropriate). Report formats should be thought through at project design stage, when considering the purpose of IA, the information required and who is responsible for gathering it. (Office Instructions Vol II G1 Annex 1 provides a recommended format). If projects are co-funded with other donors a common format and timetable of reporting should be agreed if at all possible, to minimise the administration for the partner organisation. Preparing multiple reports on the same project for different donors with different timing and format requirements can contribute to reporting being seen as more of a burden and less of a helpful learning process. Reports should : • • • • •

give full reasons if progress is not proceeding as planned identify any unexpected outcomes (positive or negative), check that the assumptions in the logical framework remain valid, highlight recommended action points by DFID, recipient or others identify responsibility for follow up with a target date for action.

Reports should be sent to the DFID project officer, who is responsible for arranging circulation within DFID to all those who should see the report, and for ensuring relevant follow up action is taken.

9

REVIEWING

Most projects' monitoring plans will include arrangements for an Output to Purpose Review (OPR) and an MTR (Midterm review.) The aim of this is to take a strategic overview, particularly assessing progress towards achieving the project Purpose and Goal. OPRs should be undertaken jointly with the recipient, and should involve key project stakeholders. The purpose is to take a strategic overview, particularly assessing progress towards achievement of the project's objectives (Purpose and Goal) and checking on whether the assumptions remain valid, particularly in the light of experience of any unexpected and /or negative outcomes. In the past, midterm reviews may have, in part, covered this ground but they sometimes lacked a clear statement of the project's impact and overall achievements against objectives. OPRs may also identify important lessons for DFID's future project management systems, both for the project under review and for new projects. The review will also provide an initial assessment of progress against the PIMS markers (see back to section 2 on project header sheet). Meaningful reviews, involving primary stakeholders, require long term planning and may include a number of stages for information gathering and assessment. The main DFID project team should participate in the final stage of the OPR so that the widest interdisciplinary view and assessment can be established. The same principles and caveats apply in planning and implementing review missions as applied above with monitoring visits. Where external consultants are appointed to undertake the OPR, it is particularly important that they understand the DFID approach, and that they have the capacity to undertake follow on work which may arise from the OPR. Guidance on commissioning IA work is given in the Core Text. Frequently, OPRs may recommend some extension to the intervention to ensure that the project purpose is achieved. Successful OPRs will require the Review Team to establish a good rapport with the participants, and this should be built on in subsequent work by use of the same external personnel. Frequent changes of consultant for different stages of work, not only takes up time as each person has to learn about the organisation from scratch, but also undermines the sense of partnership and continuity with the recipient organisation.

Box 5: Example of Staff Participation in Impact Assessment: MDA Impact Assessment, South Africa Feb 2000 - extract from Case Study (For full report click here) As important as the findings themselves was the involvement of staff at all the Centres, with a view to developing their capacity and understanding of clients’ needs in order to inform future strategic planning across the Centres and organisation. The assessment was therefore a learning experience in terms of MDA’s institutional capacity as well as its programme outputs. This meant the research was perhaps not as refined or objective as it might have been if it had been carried out by external, professional or academic evaluators; on the other hand, the learning gained by the trainers is likely to extend their empathy towards clients, and enable them to offer more appropriate advice and solutions to their business problems in the future. Research methodology: The role of the Consultant for this assignment was to facilitate the design and implementation of a monitoring and evaluation system which would enable an impact assessment to be carried out. As far as was practical in the time available, a process approach was used, beginning with a workshop for the training staff based at MDA’s Head Office in Johannesburg, to identify indicators and design the structure of the system. At the workshop, staff also designed the questionnaire which would be the main assessment tool, and put in place the logistics for carrying out the survey. Head Office training staff then became team leaders of the trainers based at each of the local Centres, taking on responsibility for co-ordinating the field research to be undertaken at the Centre allocated to them. This included making arrangements for interviewing clients, supervising the trainers, compiling and analysing the results, and writing up the report to be presented to a workshop a few days following completion of the field research. The Consultant then summarised the contributions from the teams, and compiled the final report. A mix of different methods was used to collect information: • Desk research • Survey • Focus groups • Case studies • Validation Lack of baseline data The principal problem with this first impact assessment was the absence of any baseline information against which to compare the organisation’s and clients’ progress at the end of the project. The report recommended the adoption of an MIS which will enable easy collection and analysis of such information for future reports. Future impact assessment timings were also recommended, and the suggested timings are scheduled to coincide with other management reporting requirements such as reporting to the board. This is intended to streamline all information processing and to minimise extra work.

Dissemination of findings and Follow on work The outcomes of the IA were disseminated to all staff through workshops where all MDA Units and District Centres (DCs) were represented. The representatives from the DCs reported back in their areas. It was a logical progression to follow the IA with an organisational review. The same consultant was contracted for this. This exercise was effectively an internal IA look at the impact of MDA units on the DC’s. The organizational review showed anomalies in the structure of MDA and that some services could better be delivered through re-positioning them within the structure of MDA. Following the IA and organisational review a national strategic planning workshop took place in April 2000 where the findings of the IA were discussed in the context of institutional development. At the end of MDA’s process of strategic planning with all staff, centre managers and centre staff representatives, certain key themes and strategic priorities emerged. These were presented to stakeholders at MDA’s AGM. This is a good illustration of how IA findings were used as a dynamic tool for influencing and changing the organisation from within, rather than being a paper exercise where the recommendations remain unacted upon in a dusty report.

10: Sample Logical Framework: TEEM (Training for Enterprise and Export in Malawi), Traidcraft Exchange, Malawi (for more information please refer to the Case Study notes and annexes) NARRATIVE GOAL The alleviation of poverty, through equitable trade, in Malawi

PURPOSE To improve the livelihoods of poor people in Malawi through the expansion of market-led local businesses and exports

OBJECTIVELY VERIFIABLE INDICATORS (OVIs)

MEANS OF VERIFICATION (MOVs)

RISKS AND ASSUMPTIONS

Reduction by 10,000 of people living under one US dollar per day in Malawi

1. Baseline study 2. External impact assessment 3. Income, Consumption and Expenditure (ICE) survey 4. Project documents & records 5. Trading Company transactions

That monitoring and evaluation systems can measure change in income levels for participants

1

1. External impact

That equitable trade with and within Malawi is possible, given current level of poverty

2

3 OUTPUTS 1. Business Service Centre sustainably delivering market-led business counselling to SMEs.

2. Sustainable Trading Company operating and exporting in Malawi, with growing annual turnover. 3. Products developed in conjunction with the business counselling service, being sold through the Trading Company, and linkages made between larger and smaller producers

2,000 wage earning positions created or sustained by EOP in the manufacturing and smallholder sectors Change in Employment Practices (wage levels, use of child labour & security) in the majority participating businesses* 20% increase in participants’ cash sales

assessment data & project client records 2. External impact assessment data & project client records 3. Trading Company transactions and household ICE survey

1.Business Service Centre cost recovery 40% local costs by end of project based upon a clear fee structure 2.Business Service Centre client base 360 over project life 3.6 Business Service Centre business counsellors in place and working to the satisfaction of their client base

Business Service Centre quarterly and annual reports to board

Business Service Centre client base are willing to grow from informal to formal sector

1 Trading Company break-even by Year 2 of project (Annex E) 2 Sales follow path outlined in Annexes P & A, or as indicated in the business plan during Phase 1. 1. 80% of the Business Service Centre clients state an improvement in business turnover by the mid term review (MTR), & EOP 2. 80% of clients have received product related training and counselling by MTR & EOP.

Trading Company quarterly and annual reports to board

That a suitable client base exists for the Business Service Centre & the Trading Company Political, economic and social stability is maintained

MTR & EOP Reports Project Monitoring Records Training records SME Records and ICE survey Impact survey

NARRATIVE

OBJECTIVELY VERIFIABLE INDICATORS (OVIs)

MEANS OF VERIFICATION (MOVs)

RISKS AND ASSUMPTIONS

1 & 2. Project and training Records 3. Minutes of annual consultations 4. Produced recommendations by MTR & EOP

All students successfully completed training course

3. 50% of businesses supply 4. A contribution to the development of a BDS industry for SMEs in Malawi.

5. Fair trade / ethical criteria and standards operating in all intermediary organisations within the project

larger businesses by EOP 1. 2 business counsellors completed training and employed elsewhere as training / business counsellors 2. 6 business counsellors completed training successfully and employed within the Business Service Centre 3. Project active in disseminating lessons with other sector players and MCCI annually 4. Project drafts recommendations by MTR & EOP on changes to regulatory environment and BDS in Malawi 1. Ethical criteria targeting poverty (based on CDC / IFAT guidelines)*, and which meet the needs of stakeholders, operating in 80% of businesses by EOP.

1. Ethical standards report Inception Review MTR & EOP Reports Survey of participants ICE

2. Gender strategy established and operating effectively by end of inception period, with measured impact by EoP.

2. Gender Strategy Report Inception Review MTR & EOP Reports Survey of participants ICE 3. Child Labour Strategy Report Inception Review MTR & EOP Reports Survey of participants ICE 4. HIV/AIDS Strategy Paper Inception Review Survey of participants ICE survey 1. Inception Review M&E system manuals

3. Children Labour strategy operational by end of inception period, with national consensus with stakeholder interests by EoP 4. HIV/AIDS policy adopted in 80% of participating businesses, and is beneficial to employees by end PY2. 6. Monitoring, evaluation and research system which feeds back lessons / insights to improve project performance

1. Internal M&E system established by inception review

2. External Poverty impact monitoring system established by end of PY1 3. Action Research programme established by end of inception

2. IA Inception Report Completed External Baseline Survey IA progress and final reports 3. Action Research Plan Strategy Papers

NARRATIVE

ACTIVITIES Milestones achieved during the inception period of 18 months

Business Service Centre set-up and operational by Month 6 (Annex E)

OBJECTIVELY VERIFIABLE INDICATORS (OVIs)

MEANS OF VERIFICATION (MOVs)

period and co-ordinated using Project Steering Committee (PSC)

PSC Papers

PROJECT BUDGET of: £2,657,645 (Annex G for breakdown) See attached action plan for detailed activities (Annex E) Inception Review in Month18 DFID M&E visits Progress, as a minimum, to meet objectives set out in Annex E (Figure 5 ).

TX Visit reports Quarterly TX reports to DFID & PSC Quarterly financial reports to boards Annual, MTR & EOP evaluations TX Visit reports Quarterly TX reports to DFID & PSC Quarterly financial reports to boards Annual, MTR & EOP evaluations

Annual dissemination workshops operational by Month 12, and thereafter annually.

Trading Company set up and operational by Month 9 (Annex E)

Progress, as a minimum, to meet objectives set out in Annex E (Figure 6).

Contractual link between the Business Service Centre and Trading Company established by Months 9 - 12 (Annex E)

MoU or legal contract by month 12 between two parts of TEEM.

TX works together with the Business Service Centre to deliver a market-led business counselling service.

Products design based on client surveys and fee collection systems established which maximise sustainability by end of inception phase.

Impact Assessment tendered and contracted by DFID, and agreed with TX Action Research programme deigned and initiated

Tenders accepted by Month 5, and initial baseline work started by Month 7.

Client Surveys Fee collection Manuals Quarterly TX reports to DFID & PSC Quarterly financial reports to boards Annual, MTR & EOP evaluations Tender documents Inception and baseline reports

Activities laid out in Annex G (Table 8) and Annex E planned and time-tabled by Month 18.

Action Research Strategy PSC minutes Tender documents

TX Visit reports Quarterly TX reports to DFID & PSC Quarterly financial reports to boards Annual, MTR & EOP evaluations MoU Quarterly TX reports to DFID & PSC Quarterly financial reports to boards Annual, MTR & EOP evaluations

RISKS AND ASSUMPTIONS

It is possible to create a Trading Company in Malawi within the given time frame 1. That the project can attract and retain staff of a suitable calibre 2. That the Business Service Centre clients are willing to contribute to the cost of inputs That an ethical Trading Company can be set up in Malawi That a workable relationship develops between the Business Service Centre & the Trading Company That the GoM is willing & able to create an SME friendly environment

That contractors are available to carry out the envisaged work That action research is appropriate to the needs of the project, and consultants can be engaged.