Internal Monitoring Handbook Eng

The European Union's Technical Assistance Programme for Ukraine

HANDBOOK on Internal Monitoring of Twinning Operations in Ukraine Working Document

Project is funded by the European Union

Kyiv, October 2009

Project is implemented by Human Dynamics Consortium

HANDBOOK on Internal Monitoring of Twinning Operations in Ukraine Working Document

Kyiv, October 2009

Prepared by a Team of Experts: Ms V. Valeri, Mr R. Reichert, Ms T. Gaidarenko and Mr V. Kuchynsky; Team Leader: Dr Janos Zakonyi, 'Implementation of Twinning Operations in Ukraine', EC/Tacis project, October 2009. This working document has been produced with the assistance of the European Union. The contents of it are the sole responsibility of the ITO project and can in no way be taken to reflect the views of the European Union.

FOREWORD It is far not enough any more just to be active and implement a task. It is not enough just 'to do something', even if it will be considered well done. A task is always a part of a wider context, and deliberately or not, is implemented always with an aim. Sometimes the aim is too ambitious and requires many intermediate steps to achieve. Hence, many tasks should be implemented, and – if the general or external conditions are favourable – they should produce those intermediary 'outputs' that will lead finally to achieving the aim of the exercise. Sometimes it is easy to establish whether the aim was achieved, but sometimes a specific methodology is necessary to see a clear and detailed picture of the implemented activities, the produced outputs and results, the achieved effects and purpose, and the achieved impacts or broader objective. This is where 'Monitoring and Evaluation' can help. The role of 'Monitoring' – yes, with a capital 'M' in this foreword – has become paramount in the modern world and the involved 'structured thinking' is inevitable not only in technical assistance but also in every aspect of both private and public life, e.g. planning and achieving smaller or bigger life goals, assessing public policies, programmes, initiatives and others. Sometimes the 'Monitoring' task is separate from the implementers and is being done as an 'external' task by independent monitors. However, 'Monitoring' must be a part of every implementation process, and it should be by its character 'internal' too, or indeed, 'internal' first of all. Every organisation having some aims to achieve – must fulfil 'Internal Monitoring' to provide timely and adequate information to decision makers about the work of the organisation and its performance. The key feature of 'Internal Monitoring' is not about controlling, checking and criticising but rather being a comprehensive tool for securing high quality performance and serving as a tool for verification of the progress and providing early warning as and when necessary. The 'Implementation of Twinning Operations in Ukraine' EC/Tacis project staff has learned a lot about Twinning and the related preparation and implementation methodologies during the more than three years of the project operations. We learned – among other lessons – that no matter how excellent the design of a project can (and should) be, no matter how committed the partners are (and should be), no matter how well a project is prepared and implemented (and should be) – without clear 'Monitoring' rules and practices the production of Mandatory Results and ultimately the achievement of the Project Purpose and Objective of a (Twinning) project or programme is hardly possible, AND/OR establishing, verifying and presenting the Results and Objectives in a convincing way become extremely difficult. The findings of the unique 'Rapid Assessment of the First Five Twinning Projects Implemented in Ukraine', conducted by the ITO project this year clearly demonstrate a need for strengthening the role of 'Internal Monitoring' by the implementing partners, and particularly by the Ukrainian side, who bears the ultimate responsibility for further implementation and sustainability of the achieved Results. The Assessment strongly emphasises the need for developing the internal monitoring culture and capacity of Ukrainian Twinning ministries and agencies further as part of the overall effort to disseminate European internal control standards and practice within the Ukrainian administration. We have not only felt the necessity of developing internal monitoring: we have also acted upon the issue as far as it was made possible by the resources of our project. We have organised seminars, workshops and training sessions on monitoring and promoted its development with every available to us ways and means. The present working document is one of the modest results of these efforts. I hope that many colleagues will read and utilise it in their everyday work. Please send your feedback to [email protected], and we will make them available to others. The latest version of the document may be found at www.twinning.com.ua too. As Team Leader of the project (and monitoring & evaluation specialist too) I want to thank the team of experts – Mr R. Reichert, Ms V. Valeri, Ms T. Gaidarenko and Mr V. Kuchynsky who contributed directly or indirectly to the preparation of this essential document for Twinning in Ukraine. Dr Janos Zakonyi

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

1

Table of Contents FOREWORD 1.

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4

2. 2.1 2.2 2.2.1 2.3 2.3.1 2.4

TWINNING MONITORING FRAMEWORK IN UKRAINE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 The Monitoring Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Internal Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 The Twinning Information Management System TWINS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 External Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Results Oriented Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Twinning Review Missions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

3. 3.1 3.1.1 3.2 3.3 3.4 3.5 3.6 3.7

ROLES AND RESPONSIBILITIES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 Scope of PAO Coordination Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 The Scope of Monitoring for PAO Project Officers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 Monitoring by Beneficiary Administrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 TPCG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 Cabinet of Ministers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Project Steering Committee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Allocation of Monitoring Activities by Intervention Level and Criteria . . . . . . . . . . . . . . . . . . . . . . . . . .14 The Twinning Monitoring Matrix for Twinning Projects and Programme in Ukraine . . . . . . . . . . . . . .15

4. 4.1. 4.2. 4.3. 4.3.1 4.3.2 4.4. 4.4.1 4.4.2 4.5.

TOOLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 Key Monitoring Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 Tool A: Checklist for Monitoring of Twinning projects by Intervention Level . . . . . . . . . . . . . . . . . . . . .17 Tool B: Checklist for Interim Quarterly Report Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 General Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 Cheklist for Rewiew of Interim Quarterly Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 Tool C: Checklist for Regular Project Monitoring VisitsOn Site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 General Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 Cheklist for OntheSpot Cheks Within Twinning Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23 Tool D: How Compile a Monitoring Report for TPCG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24

5. GOOD PRACTICE/EXAMPLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27 ANNEXES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 Annex 1: Key Factors for Successful Project Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 Annex 2: Criteria for Quality Assessment of Twinning Projects Design and Implementation . . . . . . . . . . . . . . .28 Annex 3: Description of the Five Evaluation Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31 Annex 4: Using the LFA During the Project Cycle. Strength and Common Difficulties . . . . . . . . . . . . . . . . . . .35 Strengths and Common Difficulties with the Application of the LFA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 Annex 5: Glossary of Key Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36 Annex 6: Web References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38

2

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

Acronyms and Abbreviations AO

Administrative Office as defined by the Common Twinning Manual

BA

Beneficiary Administration

BC

Beneficiary Country

CMU

Cabinet of Ministers of Ukraine

CTM

Common Twinning Manual

DIS

Decentralised Implementation System

EC

European Commission

ECD

Delegation of the European Commission to Ukraine

ENPI

European Neighbourhood and Partnership Instrument

EU

European Union

FP

Focal Point established within line Ministries

LFA

Log frame Analysis

LFM

Logframe Matrix

IE

Interim Evaluation

IQR

Interim Quarterly Report

MDCS

Main Department of Civil Service of Ukraine

MR

Monitoring Reports

MS

Member State

NCU

Ministry of Economy of Ukraine

OVI

Objectively Verifiable Indicator

PAO

(Twinning) Programme Administration Office

PCM

Project Cycle Management

PM

Project Manager

PSC

Project Steering Committee

ROM

ResultOriented Monitoring

RTA

Resident Twinning Adviser

TA

Technical Assistance

TAIEX

Technical Assistance Information Exchange Unit

ToR

Terms of Reference

TPCG

Twinning Programme Coordination Group

TWG

Twinning

WP

Work Plan

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

3

1. INTRODUCTION The Twinning instrument was introduced in Ukraine in 2006 as a new form of assistance under the European Neighbourhood and Partnership Instrument (ENPI), in order to allow Community assistance to partner countries to be explicitly policydriven. Ukraine, together with the European Commission as well as more than one hundred countries and donor organisations, has signed the 2005 Paris Declaration on Aid Effectiveness1 which is a major international agreement aimed at raising the quality of aid and its impact on development. The Declaration appeals to better manage for results and mutual accountability which supposes to promote greatly the aid's ownership and effectiveness. The partner countries are encouraged to establish resultoriented reporting and assessment frameworks. Twinning as a major participatory instrument to induce the reform process by concretely responding to national strategies is in full correspondence to the Paris Declaration's objectives. The number of Twinning projects is rapidly increasing in Ukraine which demonstrates that this new instrument corresponds to the demand of the Ukrainian Beneficiary Administrations to pursue reforms in specific areas of harmonisation as laid down in the official documents of EUUkraine cooperation. More and more twinning projects have started the implementation phase which makes systematic Project Monitoring a task of utmost importance. As the Common Twinning Manual 20092 iterates (Chapter 6.3. Monitoring): 'The sole reason and justification for Twinning is the achievement of the mandatory results. In the case of ambitious, largescale and lengthy projects, there is a real danger that this purpose may become obscured as time goes by, and that the rest of the exercise will achieve only piecemeal and limited advances in a few areas. Close monitoring of projects will therefore be needed in order to identify and rectify any problems that may arise'. As a key qualityassurance method, monitoring has to be continuous, systematic and linked to decisionmaking. The responsibility of coordination is intrinsically linked to monitoring, without direct process involvement no basis for judgment and decisionmaking is available. The monitoring system must correspond to the institutional management setup of the Twinning programme and adapted when major changes occur, such as decentralisation of responsibilities or highly increased portfolio and many different sectors involved. This Handbook on Internal Monitoring of Twinning Operations in Ukraine provides useful and comprehensive information for all stakeholders involved in the preparation, implementation and monitoring of Twinning projects in Ukraine. All Ukrainian institutions involved in twinning operations are advised to contribute to monitoring in fulfilling their responsibilities to increase efficiency, effectiveness and impact of Twinning in Ukraine. This Handbook does not substitute but rather complements the European Commission's Common Twinning Manual, in particular by specifying the monitoring framework for Twinning Operations in Ukraine and providing tools for internal monitoring. It should be used in conjunction with the Methodological Recommendations on Using the Twinning Instrument for BAs issued by the MDCS, the Internal Manual of Procedures for the Operations of the Programme Administration Office, the Practical Guidelines of Twinning Operations in Ukraine, the Twins Quick Start Manual3 as well as other normative or methodological documents related to Twinning. The Handbook provides tools to project managers to systematically follow projects and to maintain an updated and accurate knowledge on each project within their portfolio. Resident Twinning Advisers and their Counterparts, the Project Leaders, the Focal Points and particularly the PAO Project Officers bear specific responsibilities contributing to an integrated internal monitoring system which, if all levels actively contribute according to their obligations, consists of a balanced and effective project quality control system. Templates and checklists can be and need to be updated as new information needs arise and implementation structures eventually change. They further provide insights into the underlying methodologies, and concepts invite to collect good practice examples. Users are invited to use good practice material as well as the PCM materials whenever appropriate. This is a resource handbook to deal with upcoming questions when actions take place. It can be expanded into particular areas of project design and planning. http://www.oecd.org/dataoecd/11/41/34428351.pdf As twinning projects are financed under the ENPI framework, when consulting the Common Twinning Manual only references applicable to ENPI should be taken into consideration, thus not considering references to IPA (before or under EDIS) which are not applicable to Ukraine. http://ec.europa.eu/enlargement/howdoesitwork/financialassistance/institution_building/twinning_en.htm 3 http://twinning.com.ua 1 2

4

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

Moreover, Twinning managers in Ukraine can count on the dedicated TWINS4 web facility which allows them to input and produce updated information for monitoring purposes, such us monthly reports and tables on Project and Programme status, analytical information on individual projects, Twinning statistics, etc.

2. TWINNING MONITORING FRAMEWORK IN UKRAINE 2.1 The Monitoring Function As stated above, the importance of monitoring of twinning projects in order to ensure quality and achievement of mandatory results is stipulated in the Common Twinning Manual as well as other guidelines and reference materials developed locally. In the Manual, role and responsibilities of different stakeholders are described in detail including the role of Focal Points in the line ministries. In compliance with the Common Twinning Manual, the monitoring function for twinning operations in Ukraine is shared between the Programme Administration Office (PAO), the project partners (MS and BA) and the EC Delegation (ECD) which manages the project's contract and which, apart from regular monitoring of project implementation, has contracted out external monitoring. The Twinning Programme Coordination Group (TPCG) coordinates planning and implementation of the Twinning Programme with the general process of EU external assistance in Ukraine and thus, together with the ECD, is the prime addressee of monitoring reports. The Programme Administration Office, which prepares and contributes to the meetings of the TPCG, ensures that continuous monitoring of each Twinning project takes place, that IQRs are correct and informative, problems are identified and solved or submitted to instances for corrective actions, that PSC meetings serve the purposes and that the TPCG receives condensed information on the status of project implementation highlighting major achievements and issues. The Monitoring Matrix for Twinning Projects and Programme in Ukraine5 demonstrates that the stakeholders need to be involved at different stages of programming and implementation and that monitoring overall supports project cycle management and helps to achieve quality twinning projects and a welldeveloped twinning programme to achieve necessary institutional reforms. Focal Points are inter alia responsible for organisation of Monthly Monitoring Meetings and in the future for compilation of monitoring reports within their sector (e.g. energy, public finance), in case several TA and Twinning and TAIEX projects are ongoing or planned in the same sector. The acquaintance and capacity of the different stakeholders with the monitoring methodology and methods greatly varies, which is one of the major reasons, why this handbook has been commissioned6. Monitoring as a key quality assurance instrument remains literature unless standard knowledge is common among the participants and a continuous willingness to improve the skills through communication among peers, interest in training sessions and the practical use of tools are embedded. Most obviously, sustainable monitoring systems are best achieved through continuous practice. Twinning follows a strict partnership approach which requests from MS and BC an equivalent effort in planning and implementation of the project. Monitoring responsibilities are shared among all involved parties, with different scope and intensity. As a joint venture, care and emphasis has to be given to make best use of the monitoring instruments. The time consuming planning effort, the availability of a detailed work plan and the procedural framework as laid down in the Common Twinning Manual and other documents make structured monitoring possible if the parties engage sufficiently in the exercise. The key to monitoring is engagement, commitment to results and appreciation of transparency and effectiveness. It is not a description of rigid controlling measures but rather a review process as necessary to develop a joint success.

TWINS: Twinning Information Management System. Ref. to Section 3.7 below. 6 It is of particular interest that the stakeholders share same approaches (and whenever possible working tools): in June 2007 EuropeAid released a useful guiding document 'Strengthening project internal monitoring – How to enhance the role of EC task manager' in Tools and Method Series Reference Document 3 which may be downloaded from: http://ec.europa.eu/europeaid/multimedia/publications/publications/manualstools/t108_en.htm 4 5

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

5

Monitoring can be internal and external and should focus on the entire programme and project cycle. As regards projects under implementation, internal monitoring is the responsibility of those involved in project implementation and is performed on an ongoing basis. External monitoring is performed at regular intervals by the entity which finances the operation being monitored, as well as independent monitors contracted by such entity or any other entitled entities. Monitoring is however to cover also the preimplementation phase, that is identification, preparation, contracting and set up of BA arrangements for efficient and effective start of twinning implementation7. The following table provides an overview of monitoring responsibilities for twinning operations in Ukraine at project and programme level: Twinning Stakeholder BA/MS Partner(s) PSC Focal Points within line Ministries PAO TPCG Secretariat of the Cabinet of Ministers ECD and EC Monitoring Agents

Responsibility Project Level Project Level Project and Sector Level Overall coordination at project, sector, and programme level Programme level Programme level Project, sector and programme level

Whilst the Twinning Beneficiary Administrations and their Member States hold the primary internal monitoring responsibility under Project Steering Committee guidance, the Focal Points, PAO, TPCG, NCU and Cabinet of Ministers perform a monitoring role that includes coordination functions.

2.2 Internal Monitoring Monitoring is defined as the systematic and continuous collecting, analysing and using of information for the purpose of management and decisionmaking. The purpose of monitoring is to achieve efficient and effective performance of an operation. Monitoring systems should therefore provide information to the right people at the right time to help them make informed decisions. Monitoring must highlight the strengths and weaknesses in project implementation, enabling managers to deal with problems, finding solutions and adapt to changing circumstances in order to improve project performance.

Effective monitoring provides an 'early warning system', which allows for timely and appropriate intervention if a project is not adhering to the plan: moreover, it provides information on project progress, gathered through a number of resources, e.g. the Financing Agreement or Proposal; a sector programme or country agreement, the Log Frame Matrix, Work Plan, reporting documentation and most importantly, by familiarising with the project in the field. The status of a project can be only assessed following interviews with all parties involved in the project, including the beneficiaries. It is difficult to properly monitor project progress without field visits. Monitoring systems with quantitative and qualitative performance indicators focussing on results have been established over the last years to get improved feedback from the field. Monitoring should focus on collecting and analysing information on: • Physical progress (input provision, activities undertaken, benchmarks8 delivered and results achieved, eventual deviations from the work plan) and the quality of process (i.e. stakeholder participation and local capacity building); • Financial progress (budget versus expenditure): in the case of Twinning, this is assessed by the ECD; According to the Order of the Main Department of the Civil Service of Ukraine No 191, dated July 11, 2008 on 'Approving the Procedure for Monitoring the Preparation of Twinning Projects in Ukraine' registered with the Ministry of Justice of Ukraine on September 29, 2008 under No 907/15598, with consent of the head of a central executive authority – beneficiary of a Twinning project, the Head of the Main Department of the Civil Service of Ukraine or his deputy, as appropriate within their respective remits, can decide to examine the readiness of such authority for the implementation of the Twinning project and its compliance with the requirements for preparation of Twinning projects set forth in normative legal acts. 8 The terms benchmark is used in the Twinning Common Manual to define the 'Objectively Verifiable Indicators' in the Logframe Matrix, see Box 1 on 'Benchmarks and Benchmarking Defined'. 7

6

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

• Preliminary response by target groups to project activities (i.e. use of services or facilities and changes in knowledge, attitudes or practices); • Reasons for any unexpected or adverse response by target groups, and what remedial action can be taken. Monitoring cannot be described as being effective simply because the required information is collected. The information collected must be communicated – in the right form, to the right persons, at the right time. Only then timely and appropriate management decisions can be made to address eventual problems and ensure that, in case of problems, the project is brought 'back on track'. Monitoring is an essential part of the Project Cycle Management process and a vital management tool. It is therefore: • An information gathering exercise based on knowledge of project documentation, current status and general project environment. • Interviews with all actors involved to arrive at a structured opinion on progress. • A facilitator for good project management. • A transparent exercise, whereby all parties are aware of project progress and difficulties (if any). • A speedy and effective way of providing brief and informative reports. • A service provided to all stakeholders to keep them informed regarding project progress. • An overview of project implementation at a given point in time, which is carried out against a clear set of objective criteria. Monitoring in essence is a continuous review process of collecting evidencebased information, assessing, communicating and reporting. The aim is to identify opportunities to raise project efficiency, which depends, in the context of twinning, on timely and equal provision of resources from the two partners, and what support can be provided to reach the planned results and objectives, i.e. the project effectiveness. Internal monitoring is therefore the foundation of any monitoring system. The Common Twinning Manual instructs on the use of 'benchmarks' to measure the achievement of results at each level of a twinning project over time (as presented under Box 1 below). Benchmarks are objectively verifiable indicators and their proper definition provides the basis for designing an appropriate monitoring system.

Box 1: Benchmarks and Benchmarking Defined Common Twinning Manual 2009, Annex A Benchmarks: How will the achievement of the results at each level of the project (i.e. from Overall Objective down to Activities) be measured? Make sure that the benchmarks always define the following: 1. Quantity 2. Quality 3. Target Group 4. Time 5. Place http://ec.europa.eu/enlargement/pdf/financial_assistance/institution_building/final_version_contract_ annexes_201107.pdf Objectively Verifiable Indicators (OVIs) show whether or not objectives have been achieved at the three highest levels of the logframe. OVIs provide the basis for designing an appropriate monitoring system. European Commission, Project Cycle Management Guidelines http://ec.europa.eu/europeaid/multimedia/publications/documents/tools/europeaid_adm_pcm_guidelines_ 2004_en.pdf Indicators and Intervention Logic Extract from: Directorate General External Relations. Directorate General Development. EuropeAid Co operation Office. Joint Evaluation Unit. Evaluation Methods for the European Union's External Assistance. Methodological Bases for Evaluation. Volume 1.

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

7

Category

Definition

Input Indicators

Financial, human, material, organisational or regulatory resources mobilised during the implementation of the intervention

Activity Indicators

Implementation and management processes

Output Indicators

Goods and services that are delivered under the responsibility of the managers of the intervention. To put it simply, it can be said that outputs are what is bought with the public money

Result Indicators

Immediate effects of the intervention for its direct addressees. An effect is immediate if the operator notices it easily while he is in contact with the addressees. Because they are easily acknowledged by the operators, result indicators can be monitored systematically

Impact Indicators

Far reaching and indirect consequences of the intervention

http://ec.europa.eu/europeaid/evaluation/methodology/examples/guide1_en.pdf Benchmarking: the process used in management in which organizations evaluate various aspects of their processes in relation to the best practice, usually within their own sector http://en.wikipedia.org/wiki/Benchmark Benchmark: Reference point or standard against which progress or achievements can be assessed. A benchmark refers to the performance that has been achieved in the recent past by other comparable organizations, or what can be reasonably inferred to have been achieved in similar circumstances. OECD / DAC GLOSSARY, Glossary of key terms in evaluation and resultsbased management http://www.oecd.org/findDocument/0,2350,en_2649_34435_1_119678_1_1_1,00.html Benchmark: Qualitative and quantitative standard for comparison of the performance of an intervention. Such a standard will often be the best in the same domain of intervention or in a related domain. European Commission, Regional Policy Inforegio, The resource for the evaluation of socioeconomic development http://ec.europa.eu/regional_policy/sources/docgener/evaluation/evalsed/glossary/glossary_b_en.htm# Beneficiary Benchmark: A measured 'best in class' achievement. The performance level which is recognised as the standard of excellence for a specific business process. A systematic and continuous measurement process; a process of continuously comparing and measuring an organization's business processes against business leaders anywhere in the world to gain information which will help the organization take action to improve its performance. Benchmarking Glossary http://creativeideas.org.uk/glossary.htm

2.2.1 The Twinning Information Management System TWINS The Twinning Information Management System TWINS is operational since March 2009. Information onto the system may and can be loaded by the PAO, the ECD and Twinning Managers (particularly RTAs and coRTAs). This webaccessible and userfriendly facility allows comprehensive and updated reporting, its main output being the monthly tables on the 'Status of the Twinning Pipeline and Work Plan9', which provide information on individual projects organised on the basis of projects status, as per the following phases: • Completed Projects • Implementation Phase • Contracting Phase • Calls for Proposal Phase • Twinning Fiche Phase • Terms of Reference Phase • Identification Phase

9

8

See footnote 12 for a definition of 'Twinning Pipeline and Work Plan' in the Ukrainian context. Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

For each project, the status table that can be generated provides the projects key references (i.e. Beneficiary Administration, TWG code, Title, SectorSubsector, Project Manager at ECD, Project Manager at PAO) together with a brief description of the project status. The 'Twins Quick Start Manual' is the TWINS User Manual available to assist new users in familiarising with the TWINS operation.

2.3 External Monitoring External monitoring is performed by those not involved in project implementation and does not substitute dayto day internal monitoring by the Beneficiary Administrations and their EU Partners. External Monitoring adds value to other information already available, and is therefore complementary to internal monitoring information. Beginning with 2000, the European Commission adopted the 'Results Oriented Monitoring' methodology as applicable to all EU spending. The ROM methodology can be described as a rapid assessment technique generally to be applied for all ECfunded projects in all countries. It consists of a systematic resultsoriented assessment of the progress achieved within a project through the following five criteria: Relevance and quality of project design

The extent to which an intervention's objectives are consistent with needs, problems and issues to be addressed, i.e. beneficiaries 'requirements, country needs, global priorities and partners' and EC's policies. The extent to which effects corresponds/corresponded with the needs problems and issues to be addressed.

Efficiency

The extent to which outputs and/or the desired effects are achieved with the lowest possible use of resources/inputs (funds, expertise, time, administrative work, etc.) at a reasonable cost.

Effectiveness

The extent to which the development intervention's objectives were achieved, or are expected to be achieved, taking into account their relative importance and how far that is/was directly due to the project.

Impact potential

Positive and negative, primary and secondary longterm effects produced by a development intervention, directly or indirectly, intended or unintended.

Sustainability prospects

The probability of continued longterm benefits. The extent to which positive effects are likely to last after an intervention has terminated. The continuation of benefits from a development intervention after major development assistance has been completed. The ability of effects to last in the middle or long term10.

Annex 3 expands on the above five criteria.

2.3.1 Results Oriented Monitoring The ROM system was launched in 2000; in 2001 this project monitoring system was operational on a large scale. The ROM system was conceived to give the EC an overview of the operations undertaken and of the progress of the projects, through Monitoring Reports. In 2006, 1616 ROM monitoring reports were produced, covering 1402 ongoing projects in 148 countries. The monitoring reports are prepared by external consultants deployed to the field generally once a year. For each monitored project, the principal outputs of ROM are: Project Synopsis: Prepared by the ROM monitors during their missions, this one page document presents the project, its origin and its intervention logic in a very concise way. 10

The ability of effects to last in the middle or long term. Effects are sustainable if they last after the funding granted by the intervention has ceased. They are not sustainable if an activity is unable to generate its own resources, or if it accompanied by negative effects, particularly in the environment, and if that leads to blockages or rejection.

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

9

Monitoring Report: The monitoring report is the main ROM product. It consists of a summary of maximum two pages assessing the project through 5 criteria: Relevance and quality of design of the project, Efficiency of the implementation up to the date of the monitoring, Effectiveness up to the date of the monitoring, Impact potential, prospects of Sustainability. It also contains recommendations of the monitors to the Commission and to the Agency/Administration implementing the project. The following will ultimately benefit from ROM: • MS and BC partners who will have a valuable management tool. • The target group and final beneficiaries who should have the benefit of a better project. • The representatives of the Commission, both the Delegation and HQ staff who can judge if the project is achieving the intended mandatory results. • The representatives of the National Authority (National Coordinator), signatory to a financing agreement or agreement with similar status, who can judge if the project is achieving the results. The Monitoring Report alone will not solve identified problems, but it will indicate the key actions required and who should implement them. It is the responsibility of the parties identified to take action.

2.4 Twinning Review Missions The Common Twinning Manual 2009 has introduced a provision, which was not included under the CTM previous versions, regarding 'Twinning Review Missions'. According to section 9.2.2 of the 2009 CTM, 'Each Twinning project can be followed, 6 to 12 months after its finalisation, by a Twinning Review Mission. This mission aims at reporting whether sustainable impacts or spin offs have been observed after Twinning project finalisation. The objectives of the Twinning Review Mission are: • Assessment of the sustainability of the Twinning project's (mandatory) results and the long term impact on the project's objectives; • Extract lessons learned from the project, positive and negative; • Dissemination of good practice. The Twinning Review Missions are organised and financed via the TAIEX instrument, involving the following steps: 1. Between 6 and 12 months after the closure of the Twinning project Commission Headquarters initiates the organisation of the mission; 2. Participants: The mission will be led by a public sector expert from a different Member State than the Lead or Junior Member State partners of the project concerned, who has not participated in the Twinning project in question (to be identified by the Commission in the database of finalised Twinning projects, preferably a former RTA in a similar project): the Twinning Review Expert (TRE). This TRE will write the review report. He will be assisted and accompanied by a core group, consisting of: a. The former MS PL and RTA; b. The former BC PL and RTA counterpart; c. The Sector Manager from the EC Delegation who followed the project or the sector involved; 3. The standard format for the terms of reference of the mission and the standard questionnaire are to be completed; 4. The mission would in principle not be longer than a week: 35 working days; 5. In the beneficiary country, the TRE will convene a meeting with the core group (see under point 2) and discuss the Twinning project in question following the standardised evaluation format. He will then conduct further visits to stakeholders in the beneficiary country, to which the members of the core group can accompany him; 6. At the end of the mission the TRE will share his preliminary outcomes with the core group and listen to their comments; 7. Within two weeks following the mission the TRE will finalise the final report, in which the comments from the core group are integrated. 8. The final report will be included in a database of Twinning assessments accessible for the Twinning network: BC partners, MS Twinning and the Commission'. Step 8 is of great interest to all Twinning stakeholders who shall have the possibility to access a 'database of Twinning assessments'. 10

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

3. ROLES AND RESPONSIBILITIES 3.1 Scope of PAO Coordination Role The coordination mandate of the PAO as per Ukrainian legal basis and as defined in EC / Ukrainian documents necessitates an active role on continuous monitoring. The linkage role of the PAO implies that other levels depend on a systemoriented approach which follows a common terminology. The proximity of the PAO to the twinning implementation and the mandate to ensure full commitment as well as effective operations in compliance with national priorities provide a unique opportunity. The reporting responsibilities of the PAO to the Twinning Programme CoOrdination Group include providing material for the TPCG for reviewing the outcome of any monitoring, audit and evaluation activities carried out by any entitled entities and eventually brought to the PAO attention, in order to advise on the necessary followup. Both external monitors as well as ECD Sector Managers rely on the PAO capacity to ensure that implementation arrangements and the implementation of the agreed work plan are operative. Any greater difficulty eventually observed by external instances would raise questions on the performance of PAO monitoring. The communication between Beneficiary Administration, MS partners, project team and the ECD should be structured and objectiveoriented. Project managers designated to sectors have a wide responsibility of supporting the beneficiary organisation and to facilitate the proper implementation of all twinning operations. PAO has in addition to the Project Level a responsibility to address the Programme Level of Twinning which is to identify general issues and lessons learned through the programming and implementation process of twinning projects, and formulate topics for discussion at the TPCG meetings. The monitoring of the programme level is mainly performed through meeting events with Beneficiary Administrations, Focal Points, Resident Twinning Advisers and other concerned and relevant subjects.

The current scope of the role of the PAO role shall be increased in the forthcoming years through a decentralisation process which might introduce DIS arrangements in Ukraine. The (future) Administrative Office (AO) as defined by the Common Twinning Manual will be a body within the Administration of the Beneficiary country, designated to retain the overall procedural, financial and contractual management of the Twinning projects. Monitoring is therein the base level of management and the capacity, methods and instruments needs to be a constant concern. Highlight. With an increasing portfolio and scope of responsibilities the PAO / AO will have to strengthen internal monitoring capacities as well as actively support BAs / Focal Points to improve their internal monitoring capacity, in order to ensure a coordinated approach of all twinning projects within one ministry and ensure coordination with other EC instruments as well as with other donor and local support. An integrated monitoring system, which takes into account inputs at various levels, is therefore a requirement.

3.1.1 The Scope of Monitoring for PAO Project Officers Quarterly Monitoring Plan 1. Specify what major events are scheduled for each project in each quarter. 2. Select monitor interventions and estimate the associated workloads. 3. Distinguish between externally fixed dates and time windows. 4. Prepare a quarterly monitoring plan (at individual and team level). 5. Submit the plan to your hierarchy and get approval. 6. Integrate regular monitoring into individual and team work schedules. 7. Update the quarterly monitoring plan on a monthly basis. Monitoring Tasks • Study procedures carefully (mainly the Twinning Manual). • Maintain accurate information by systematically studying documents and organising files logically. • Communicate with different stakeholders, in particular RTAs, counterpart RTAs, ECD Sector Managers.

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

11

• Review and assess project work outputs, particularly the reports (IQR and Final Report) and the project website, if any. • Attend major project events, e.g. Project Steering Committee, working group meetings, seminars, mid term conference. • Carry out site visits (including during training seminars). • Draft monitoring reports. • Consult superiors on project status. • Participate in coordinating meetings. • Formulate recommendations. Monitoring Workloads As a rough and general rule 46 working days per month per project are an absolute minimum time allocation for monitoring each of the projects in a Project Manager portfolio under the assumption that no major problems were identified which would absorb additional time. Estimates are to take into account routine activities, such as: reviewing Reports; preparing brief assessment reports; preparing for and participating in an event such as a working group meeting, Project Steering Committee meeting or site visit with talks to different stakeholders.

3.2 Monitoring by Beneficiary Administrations 'Additional monitoring will take place in the framework of regular operational coordination meetings between the BC, the MS and the AO in coordination with the Commission Delegation where applicable. For this purpose monthly meetings between the Commission Delegation / AO, BC Ministries concerned with Twinning projects and RTAs should be held'11. Highlight. Such monthly meetings have unfortunately not become common practice in Ukraine. The Practical Guidelines on Twinning Procedures in Ukraine suggests the Focal Point within the BC line ministry organises Monthly Monitoring Meetings. This is fully in line with the Common Twinning Manual and is to be encouraged, though it needs more awareness raising/capacity building to be provided to the Focal Points. Monthly Monitoring Meetings follow the subsidiarity principle that problems should be solved at the appropriate level and as early as detected. Focal Points have the adequate sectoral and institutional knowledge and can greatly help project implementation by early detecting hindrance for implementation and suggesting acceptable solutions. Therefore the Focal Point continuously follows the project and attends meetings to collect and share information.

Regular Monthly Monitoring Meetings prevent technical problems to raise and absorb the full capacity of the following PSC meeting, thus enabling the PSC to better consider and focus on project achievements.

3.3 TPCG12 The Twinning Programme Coordination Group has been established to review and coordinate the development of twinning projects in the frame of the overall programming and implementation of EU assistance in Ukraine. The TPCG's role is to: a. Review the twinning projects pipeline and work plan13, as well as the project portfolio; b. Consider and provide recommendations on the detailed modalities for twinning arrangements as well as any other specific modalities related to the Twinning Programme implementation; c. Review the outcome of the monitoring, evaluation and audit activities, and advise on the necessary followup. Common Twinning Manual, Section 6.3. Described in detail in the Internal Manual of Procedures for the Operations of the Programme Administration Office 13 In Ukraine 'pipeline' is meant to be the list of all twinning initiatives and projects at all stages of identification; 'work plan' refers to the Twinning projects approved by the TPCG which are therefore in the preparation or implementation phase. Projects contracted form the project 'portfolio'. 11 12

12

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

The TPCG is composed of members from the following bodies: • The Main Department of the Civil Service of Ukraine • The Delegation of the European Commission to Ukraine • Ministry of Economy of Ukraine • Ministry of Foreign Affairs • Coordination Bureau for European and EuroAtlantic Integration under the Cabinet of Minister's Secretariat • Other public bodies. The TPCG meets at least once every three months and more frequently if needed, at the request of at least one of its members. Highlight. The TPCG is not supposed to address or solve problems of individual projects. However general or severe problems of projects will be brought to the attention of the TPCG, to be discussed and possibly brought to the attention of the relevant instances for corrective action. Mentioning these problems is in addition helpful to identify the need for measures to be taken at the programme level.

3.4 Cabinet of Ministers In August 2008 a new coordination bureau on EU integration was established. The Regulation on Ukraine – EU Relations Monitoring Department of the Coordination Bureau for European and EuroAtlantic Integration within the Secretariat of the Cabinet of Ministers of Ukraine was released in December 2008. Twinning and TAIEX as tools of the EU are mentioned in the document and therefore it is expected that this body will play an important role in the programming and coordination of Twinning projects implementation. At this stage, it is evident that such a body is extremely important in boosting quality coordination of all assistance of EU to Ukraine, including the Twinning tool. According to the Cabinet of Ministers of Ukraine Resolution No. 868 dated 1st October 2008 the Ukrainian Part of the Committee for Cooperation between the European Union and Ukraine preapproves all new twinning ideas and concepts before TPCG meetings. This is done to ensure that the new TWG projects are in line with Government's priorities.

3.5 Project Steering Committee The Twinning Contract Annex A 1 (Description of the Action) Article 7.2. specifies the Role of the Project Steering Committee (PSC) as follows: 'At quarterly intervals, the Project Leaders, the RTA and where applicable, representatives of the administrative office and/or the EC Delegation will meet to discuss the progress of the project, verify the achievement of the outputs and mandatory results and discuss actions to be undertaken in the following quarter. The Project Steering Committee will also discuss the draft of the quarterly report submitted to it beforehand, recommend corrections. The responsibility for the organisation of the Project Steering Committee meeting lies with both Project Leaders'14. Highlight. The Project Steering Committee has become an established regular body which once setup of senior members, with only occasional personal modification of its composition, needs good preparatory work done before the meetings, as well as an issuerelated meeting agenda that can facilitate project implementation and raise project effectiveness as well.

The PAO should receive the interim quarterly report at least two weeks before the PSC meets, thus allowing an in depth review of project status.

14

In addition to the provisions under Annex A1, the term 'Steering Committee' is quoted 43 times within the Twinning Common Manual.

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

13

Highlight. The PAO should not achieve knowledge on the progress of the project through the IQR alone, but be in a position to compare the IQR content with its own judgements acquired through an ongoing monitoring process.

The PAO should provide a brief assessment report to the PSC including the specification of the monitoring activities performed. A followup table on decisions made by the PSC should be established with implementation responsibilities specified, and report on execution should be provided during the following PSC meeting. A good orientation for describing project progress and ultimately the reference standard is provided by the template for the twinning final report15.

3.6 Allocation of Monitoring Activities by Intervention Level and Criteria The matrix below summarizes the five tier monitoring system and in which domain each partner has its priorities. The table corresponds to Tool B16. №

Monitoring Activities or Key Monitoring Questions17

MA MQ

Criteria

1

Are overall objective and project purpose (i.e. specific objectives) still consistent with Government Policy and relevant sector programme?

MQ

Relevance

2

As presently designed, is the project necessary?

MQ

Relevance

3

As presently designed, is the project feasible?

MQ

Relevance

4

Is the Log Frame consistent and complete?

MQ

Relevance

5

Is a risk management plan elaborated and applied? If not, a risk management plan needs to be prepared and follow up.

MA

Relevance

6

Have problems emerged and have they been solved?

MQ

Relevance

7

Does the work plan consider all the relevant stakeholders?

MQ

Relevance

8

Are the planned resources available from all parties including BA?

MQ

Efficiency

9

Is the opportunity offered by the Twinning project understood by all involved staff?

MQ

Efficiency

10

Does the team need additional support?

MQ

Efficiency

11

Check consistency of all major Project Documents.

MQ

Efficiency

12

Do the Project Partners possess monitoring skills and capacity?

MQ

Efficiency

13

Assess quality of interim quarterly reports

MA

Efficiency

14

Verify correspondence of status of project implementation with reporting

MA

Efficiency

15

Identify potential weaknesses and discuss findings with partners.

MA

Efficiency

16

Closely liaise with project team (BA and MS partners).

MA

Efficiency

17

Establish a monitoring plan for Benchmarks and Means of Verification.

MA

Effectiveness

18

Define intermediate indicators for benchmarks whenever necessary18.

MA

Effectiveness

19

Does reporting address relevant issues? Is any key problem raised, if yes, which one?

MA

Effectiveness

20

Is there a common view on the progress of the project?

MQ

Effectiveness

21

Verify that projects are solved or entered into agenda of relevant authorities.

MA

Effectiveness

BA/MS

Focal Point

PSC

PAO

TPCG

See Annex C5 to Common Twinning Manual 2007. See Section 4.2 below. 17 Monitoring Activity = MA; Monitoring Question = MQ. 18 Some IQR estimate in % the progress with realization of benchmarks (for example, 50% of the benchmark was finalised). This practice could be acceptable in case intermediate indicators have been predefined; however, the above should be included under the work plan. 15 16

14

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

Monitoring Activities or Key Monitoring Questions

№

MA MQ

Criteria

22

Does the team follow a monitoring plan?

MQ

Effectiveness

23

How can efficiency and effectiveness be raised?

MQ

Effectiveness

24

Are planned beneficiaries using and also benefiting from project results?

MQ

Effectiveness

25

Check communication plan and how visibility rules are enforced

MA

Effectiveness

26

As presently implemented what is the likelihood of wider impact?

MQ

Impact

27

What lessons can be applied in programming current and future twinning intervention?

MQ

Impact

28

Support external monitoring.

MA

Impact

29

How could programming be improved overall?

MQ

Impact

30

Is a sustainability plan elaborated and enforced?

MQ

Sustainability

31

Is local ownership developing?

MQ

Sustainability

32

How can the established partnerships be maintained?

MQ

Sustainability

BA/MS

Focal Point

PSC

PAO

TPCG

3.7 The Twinning Monitoring Matrix for Twinning Projects and Programme in Ukraine Monitoring Activity Programming, ToR Drafting, Contracting

By Whom

PAO

Partnership Assessment

PAO

Routine Internal Monitoring

RTA, RTA Counterpart

Regular Operational Coordination Meeting

IQR Drafting

Interim Quarterly Report Assessment (before PSC Meeting)

FP, RTA, RTA Counterpart, PAO, ECD

RTA, RTA Counterpart PLs

PAO, ECD

[Suggested] Frequency

Objective

Inputs

Methods

Outputs

Annual Twinning Programme, Investigations with BAs

PAO internal meetings

Field visits. Attendance in events and working groups' meetings. Interviews with Stakeholders

Assessment of Staff and Budget Allocation. Strategy. Communication Channels

Data collection. Discussions among partners

Internal database. Consensus

Quarterly

Identify delays and needed corrective actions

Exante Regular

Verify that capacity and commitment necessary for successful project implementation is widespread in BAs and close cooperation exists among project partners

Weekly

Identify needs and assess consequences of alternative actions

Work Plan. Project documentation. Project

Monthly

Review on Activities. Progress in Previous Month, Problems encountered and foreseen activities in next period

Work Plan. Logframe

Review of WP

Monitoring Note (one page) for database + PAO +PSC

Quarterly

Report on progress with recommendations for corrective changes to Logframe

Work Plan. Logframe. Project documentation

Annex C 4 Template

IQR

Work Plan. Previous IQRs. IQR. Project web page

Formal Check, Completeness Check, Consistency Check (see the checklist in the in Annexes), Field Visit

Comments of PAO to PSC

Quarterly

Verify if activities are on schedule and benchmarks are achieved

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

Reporting to TPCG

TPCG Decision (PAO Communication and Reporting)

15

Monitoring Activity

By Whom

PAO, Review Meeting. ECD, External evaluation if BAs, MS, critical problems RTA

Project Steering Committee

External Monitoring

Twinning Programme Co ordination Group

PLs, RTA, RTA Counterpart, FP, ECD, PAO + Invited Stakeholders

Contracted by EC

[Suggested] Frequency

Objective

Inputs

Methods

After first 6 months and then systematically

Identify the validity of the planning, identification of necessary changes or confirmation of Work Plan and objectives

Work Plan. IQRs. PAO Monitoring Reports. External Monitoring Reports if available

Intensive Document Review. Field Visit. Stakeholder Interviews. Monitoring Report of PAO

Minutes of Review Meeting, Review Report Followup Matrix

Quarterly

Review the main issues and take strategic decisions

IQR. PAO comments. Monthly Monitoring Notes

Discussions

Decisions, Minutes of PSC Meetings

Assess relevance and progress in relation to the 5 evaluation criteria

Work Plan. IQR. Minutes of Project Steering Committee meetings. Other work documents. Stakeholder Views.

Intensive Document Assessment. Field Visit. Interviews. Quality Control of Conclusions and Rating

Project Synopsis, Monitoring Report, Background Conclusion Sheet

Monitor Programme overall. Coordinate with other instruments

Twinning Programme updates. Project Summary Reports. External Monitoring Reports. Evaluation Reports

Presentations, Project Summary Report. Discussions

Decisions such as Adoption of Indicative Work Plans; Measures to enhance Twinning programme including audit and evaluation

Once a year

PAO, MDCSU, ECD, Quarterly MoE, MoFA, EC + Invited Observers

Outputs

4. TOOLS 4.1 Key Monitoring Tools As presented in the Common Twinning Manual, several tools can be used to monitor project.s. These are: • Study/analysis/review of documents (including those published in web sites) • Organisation of meetings with relevant counterparts. • Participation in meetings organised by relevant counterparts • Elaboration of, and contribution to, various reports • Visits to project site (including participation in events organised by each twinning projects) When the number of ongoing projects is high, the above activities need to be carefully planned via a detailed monitoring plan which shall also estimate Monitoring Officers' individual workloads. For example, in the hypothesis that for each twinning project the Monitoring Officer is to take part in: • A monthly meeting (two days study/analysis/review of documents, one day meeting, totalling 3 days) • A quarterly Steering Committee (same as above, three days) • Onetwo events per month organised by the twinning project (onetwo days) • And is to prepare a monthly monitoring report for the TPCG (two days) Monitoring each project would absorb, if assisted by appropriate procedures and aid tools (such as checklists), and in case no special issues emerge, min. 6 working days per month. In this respect the Common Twinning Manual suggests: 'Following the receipt by the Commission/AO of the 2nd interim quarterly report, a review will systematically be undertaken. The review will be conducted by the AO (presently the PAO in Ukraine) in coordination with the Commission Delegation.

16

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

The review will be conducted in consultation with the partners. This review can also be discussed during the regular meetings of the Steering Committee, established for each Twinning project. It may lead to reorientation of the project or, in extreme cases, withdrawal of financing, i.e. termination of the project. Where the twinning contract is linked to other contracts (e.g. TA or supply) as part of a larger project, it is highly recommended that the Steering Committee covers the project as a whole and also discusses the other components to ensure good coordination'. Highlight. To review a Twinning project thoroughly after 6 months is a good idea, since at that time (generally one third of project duration) major unforeseen difficulties or potentials have become visible and changes are still possible. It is however necessary to establish a detailed procedure which guarantees that adequate professional input is provided and the readiness to accept conclusions and recommendations must be likely. Just to review the two first interim quarterly reports and check adherence to the work plan is not enough. The timing (six months after commencement) is adequate to verify if the twinning project design is still relevant (which however might require a thorough understanding of the project technical contents) and to identify ways and means to increase the project efficiency. Both internal as well as external conditions as laid down in the assumptions need attention. Section 2 A of the IQR Template includes the assessment of the project assumptions. Risk profiles are to be appraised correspondingly. Specific attention should be given by PAO to the conciseness of that section. The review therefore can be the key instrument to address project quality as well as the quality of project reporting.

The following sections contain: 4.2. Tool A: Checklist for Monitoring of Twinning Projects by Intervention Level 4.3. Tool B: Checklist for Interim Quarterly Report Review 4.4. Tool C: Checklist for Regular Project Monitoring VisitsOn Site 4.5. Tool D: How to Compile a Monitoring Report for TPCG

4.2 Tool A: Checklist for Monitoring of Twinning Projects by Intervention Level The specificity of the Twinning instrument to jointly plan and implement an institution building project with the continuous involvement of the Project MS Partners and other Stakeholders to the reform process leads to the identification of a 'Partnershipbased Monitoring System'. The cohesion of the partnership relationship, the commitment by each side for the provision of monitoring resources, the strategy pursuit and BC ownership, the BAs participation and absorption capacity, are all key factors for success and therefore require monitoring attention. A partnershipbased monitoring system like the one envisaged must be largely integrated in the sense that almost all direct stakeholders have a role to perform in the monitoring process. What can be done better and the achievement of objectives must be a constant concern. Since twinning projects address urgent strategic reform processes of national interest, review of relevance and impact in addition to efficiency and effectiveness needs being systematically carried out. Monitoring focus, tasks, monitoring activities and questions are presented below by partnership and intervention level. Project Team (BA + MS): Focus on EFFICIENCY + EFFECTIVENESS19 • Check consistency of all major Project Documents (11) • Is the Log Frame consistent and complete? (4) • Is a risk management plan elaborated and applied? If not, a risk management plan • Needs being prepared and follow up (5) • Establish a monitoring plan for Benchmarks and Means of Verification (17)

19

Numbers in brackets refer to numbered monitoring questions/activities as listed under Section 3.6 above.

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

17

• Define intermediate indicators for benchmarks whenever necessary (18) • Does reporting address relevant issues? • Is any key problem raised, if yes, which one? (19) • How can efficiency and effectiveness be raised? (23) BA (Focal Point): Focus on RELEVANCE + EFFICIENCY20 • Are overall objective and project purpose (i.e. specific objectives) still consistent with Government Policy and relevant sector programme? (1) • As presently designed, is the project feasible? (3) • Does the work plan consider all the relevant stakeholders? (7) • Are the planned resources available from all parties in particular from the BA? (8) • Is the opportunity offered by the twinning project understood by all involved staff? (9) • Does the team follow a monitoring plan? (22) • Does the team need additional support? (10) Project Steering Committee: Focus RELEVANCE + EFFECTIVENESS21 • Are overall objective and project purpose (i.e. specific objectives) still consistent with Government Policy and relevant sector programme? (1) • As presently designed, is the project necessary? (2) • Have problems emerged and have they been solved? (6) • Is there a common view on the progress of the project? (20) • Are planned beneficiaries using and also benefiting from the project results? (24) PAO: Focus EFFICIENCY + EFFECTIVENESS22 • Check consistency of all major Project Documents (11) • Do the Project Partners possess monitoring skills and capacity? (12) • Assess quality of interim quarterly reports (13) • Verify correspondence of status of project implementation with reporting (14) • Identify potential weaknesses and discuss findings with partners (15) • Closely liaise with project team (BA and MS partners) (16) • Does the team need additional support? (10) • Support External Monitoring (28) • Verify that problems are solved or entered into agenda of relevant authorities (21) • Check communication plan and how visibility rules are enforced (25) Highlight. An internal 5 level monitoring system – addressing in 3 instances (on different levels and by the specific responsible Stakeholders) the Relevance; Efficiency; Effectiveness criteria, and regularly on the top level (TPCG) also the issues of Impact and Sustainability, wherein the ECD participates in the PSC and TPCG, as well as on additional regular operational meetings, the monthly monitoring meetings plus at least one yearly external monitoring mission – can be considered to be fully integrated, comprehensive and subsidiary. A biased agenda e.g. operational issues dominating PSC or TPCG meetings not considering impact and sustainability of projects or no monthly meetings ever held, would demonstrate a suboptimal or deficient system.

TPCG: Focus RELEVANCE + IMPACT + SUSTAINABILITY23 • Are the project purpose and overall objective still consistent with the Government policies and relevant sector programme? (1) • As presently implemented what is the likelihood of wider impact? (26) • What lessons can be applied in programming current and future twinning interventions? (27) • How could be programming be improved overall? (29) • Is a sustainability plan elaborated and enforced? (30) • Is local ownership developing? (31) • How can the established partnerships be maintained? (32)

Ibid. Ibid. 22 Ibid. 23 Ibid. 20 21

18

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

4.3 Tool B: Checklist for Interim Quarterly Report Review 4.3.1 General Remarks The Interim Quarterly Report (IQR) is the key document on project progress. Joint IQR preparation by Twinning partners is a contractual obligation. The IQR has to be comprehensive and structured according to the template provided under the Common Twinning Manual (CTM), Annex C 4. More topics can be added to the IQR, however all topics stipulated by the Common Twinning Manual should be included. The Common Twinning Manual, Section 6.4.2, states that: The Interim Quarterly Reports MUST: 1. Describe progress achieved in the implementation of the Twinning project during the period under consideration, making direct reference to the timetables and benchmarks as set out in the Twinning Work Plan and highlighting any previously unforeseen activities or activities that have been cancelled. 2. Update on the general environment for project implementation. 3. Update on the assumption and risks for project implementation. 4. Make an overall evaluation of the progress achieved, including an explicit judgement on the likelihood of fully completing the project within the remaining time scale and budget. 5. Provide recommendations.

PRACTICAL SUGGESTIONS: First:

Second: Third:

Fourth:

Collect ALL key twinning documents such as Twinning Contract, Work Plan including Log Frame, latest Interim Quarterly Report, Minutes of last Project Steering Committee meeting, thus being able to cross check. Have BOTH templates (IQR Annex C 4 and Final Report Annex C 5) at hand for reviewing the Instructions/Questions given for each heading. READ first the entire IQR and highlight unexpected statements and major gaps. Then READ the previous IQR to identify differential judgements and previous outlooks. Then START assessing the IQR with the help of the checklist presented under Section 4.3.2 in this handbook. Clarify with RTA/RTA counterpart any unclear IQR sections/statements. One completed the check list, date and sign it for the project records.

Annex C 5 (Final Report) is at any moment of the implementation of a Twinning project an additional good reference since it stresses the project impact (Achievement of the Special Objective = Project Purpose) and the sustainability of the project. The IQRs are only steps /stages leading to the achievement of the project as to be described in the Final Report. The questions emphasized in the Final Report template therefore are the best 'measuring rod' for assessing a project. It goes beyond the resources activities benchmark orientation, focus of the IQR.

To be able to assess the project progress, the PAO Project Officer MUST have other sources of information than the IQR to be able to verify the relevance of the statements provided in the IQR. The IQR is the view of the Project Team only, which needs to be balanced by the monitoring process in particular done by the PAO on a regular basis (monthly meetings, regular reviews, external monitoring, etc.).

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

19

4.3.2 Cheklist for Rewiew of Interim Quarterly Report FORMAL AND FACTUAL VERIFICATION AND ASSESSMENT OF PROJECT PROGRESS Project Title: Project No: Reporting period: Report submitted by: Date the report was received: Any amendments to the twinning contract: (if yes, specify dates and check any relevant changes particularly as regards the work plan) All the questions below must be answered with sufficiently detailed answers

Rating: VGvery good, Aadequate, Ppoor, Mmissing (whenever applicable). N/A: not applicable

Action Required

GENERAL FORMAL CRITERIA • Is the required Reporting Template (Annex C4) utilised (Y/N24) • Is the IQR submitted by the prescribed deadline: two weeks before the SC meeting? (Y/N) COMPLETENESS (Are all the specific sections required by the Common Twinning Manual included?) • Correct Cover Page (Y/N) • Section 1: Project data (Y/N) • Section 2A: Background (Y/N) • Section 2B: Achievement of Mandatory Results (Y/N) • Section 2C: Activities in the Reporting Period (Y/N) • Section 2D: Timing and Delays (Y/N) • Section 2E: Assessment (Y/N) • Section 3: Expenditures (Y/N) CONSISTENCY (As required by the Common Twinning Manual) Content 2 A Background • Are both the Overall Objective and Project Purpose with the respective Indicators correctly copied from the Work Plan art. 3? • Are there references to the state of achievement and the encountered problems reported? Are solutions recommended? • Is the Section Policy Development sufficiently targeted to the relevant sector? • Are there any key policy developments affecting the project? • Positively • Negatively, what should be done? • Is the Section Project Assumptions sufficiently elaborated? • Are all Assumptions copied from the Work Plan art. 3? • Is the assessment (re assumptions) conclusive (conclusions clearly defined and unambiguous)? • Are new risks emerging and how are they dealt with? • Does this section contain information on difficulties affecting the project? • Any other comment for the section Content 2 B Achievement of Mandatory Results • Are all benchmarks (OVIs) relevant for the reporting period copied from the Work Plan and listed per mandatory result/component? • Is there reporting on delayed benchmarks (OVIs) from previous periods? • Are benchmarks (OVIs) achieved according to the schedule? • Does the narrative provide clearly defined and unambiguous conclusions? • Are benchmarks stated and discussed? • Is there a monitoring system in place (PSC meeting quarterly, monthly meetings and sixmonth reviews held, etc)? • How have the figures been obtained? • Any other comments on this section? Content 2 C Activities in the Reporting Period • Is progress in line with the Work Plan? • Are specific observations made for major finalised activities (events)? • Any other comments on this section?

24

20

Y/N: Y = yes; N= no.

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

All the questions below must be answered with sufficiently detailed answers

Rating: VGvery good, Aadequate, Ppoor, Mmissing (whenever applicable). N/A: not applicable

Action Required

Content 2 D Timing and Delays • Is the Schedule under this section adherent to the Schedule of Activities in the Work Plan? • Is the section Recuperation of Delays conclusive and likely? • Verify if resources are available to catch up. • Any other comments on this section? Content 2 E Assessment ( should be an elaborate/detailed part) • Does the narrative indicate whether lessons were learned and corrective actions were or will be implemented? • What issues need further discussions or clarifications? • How the relationship between the partners may be described? • Any other comments on this section. Content 3 Expenditures • Checks done by the Contracting Authority. GENERAL CONCLUSIONS Likelihood of fully completing planned project activities within the remaining time and budget. Likelihood of fully achieving the mandatory results within the remaining time and budget. Observations on Effectiveness, Impact and Sustainability if any (to be fully elaborated at the six month review) SUMMARY – OVERALL RECOMMENDATIONS

Date of elaboration Name of Project Manager Signature

4.4 Tool C: Checklist for Regular Project Monitoring Site Visits / OntheSpot Checks 4.4.1 General Remarks Quality Control through monitoring is the task of Project Managers who need to regularly visit project stakeholders in addition to careful review of project progress via studying the interim quarterly reports. The PM remit is to provide information to superiors, Project Steering Committee Members and external monitors. Through interviews to stakeholders the Project Manager may verify the situation assessments and form his/her own qualified opinion based on direct observations and crosschecks. Only through updated insight knowledge of the project implementation, the PM can fulfil support functions such as facilitation, advisory, and coordination. The ongoing attention to project implementation by the PM helps the project team to improve both project implementation and internal project monitoring. Early detection of problems and potentials for improvement become visible and operational. Periodicity: • Regular contact by phone, email etc should be at least on a weekly basis • Once a month some kind of site visit should be routine, e.g. • Attendance of key events (including Project Steering Committee meetings, working group seminars, training) • Interview to Twinning Partners (Beneficiary Administration and Member State), other stakeholders, participants in training events. • Visit to the twinning project premises Highlight. Project Managers are to carefully distinguish between monitoring activities, which are to support twinning projects performance, by inspection and audit. Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

21

Onthespot checks should provide evidence for: Acceptable Logistics No IT or office equipment and furniture is eligible for reimbursement under the twinning budget. However, the availability of adequate office infrastructure and internet access should not become an issue for project implementation. RTAs must concentrate their efforts on the timely implementation of the project activities alone, rather than, as it has been often the case, struggling for basic logistics to become available to the project. This is an area for careful checking. Established Internal Monitoring System Project Managers shall check how internal monitoring is organised, that is if in addition to the IQRs and Project Steering Committee meetings, are there other project meetings organised (for example by project components), how the achievement of mandatory results is internally reviewed, if a risk management plan is in place and it is periodically updated; if all activities done are fully documented and documentation is easily available; if stakeholders are systematically consulted and/or involved in the twinning project, and a system of monitoring through partnership is eventually in place. Effective Internal Communication Twinning is a partnership and therefore the frank exchange of views is essential to achieve progress. Internal communication should be effectively in place both topdown and bottomup but also between departments within the Beneficiary Administration. Project Managers shall check if internal communication is an issue, if intranet information is made available by the twinning partners to all staff involved in the project, how internal communication could be eventually improved. Any language issues should be approached as early as possible and remedial action suggested, such as higher budget allocations for translation and interpretation if project savings allow for these. Well Functioning Thematic Working Groups A great quantity of the twinning work requires the involvement of senior staff and this often is achieved through the establishment of working groups on selected topics. These working groups shall then be supported by Member State counterparts within the framework of a given twinning project component. One may expect that, given heavy workloads and incumbent urgent matters, active commitment of senior staff may be an issue, with the consequence of low absorption capacity (see below) of twinning inputs and project overall limited efficiency. Therefore, a close scrutiny on continuity and participation in working groups is a key instrument for monitoring project progress. Beneficiary's Absorption Capacity Twinning is results oriented. The Beneficiary Administration must be in a position to allocate the number of qualified staff needed for implementing specific tasks. Hence, it is vital to discuss any staffing issue with the hierarchy of the Beneficiary Administration. Project Managers should check if the Beneficiary Administration has any internal training resources or if external training providers could also be involved in the project to increase absorption as well as sustainability prospects. Evidence Backing IQR Statements Project Managers shall check if the IQR is a faithful picture of project status or information is filtered in order to make it acceptable to the hierarchy. If the latter is the case, Project Managers should include factual evidence in their monitoring reports and warn the Programme Authorities on the need of remedial action. Section 4.5 provides guidance in this respect. All Problems Are Known and Addressed Issues Problems can be multiple and grow once not adequately addressed. Some problems such as 'preconditions not met' might exist from the very beginning of implementation including 'infrastructural problems affecting the project management' but many might emerge over time and be much more fundamental. If the work plan is unrealistic, or the Beneficiary Administration commitment is declining, working groups are not active or no procedures are in place allowing full participation of the Beneficiary Administration in the project, the PAO should intervene and provide support for remedial action to be advanced. Identification of problems and provision of support through mediation may prove vital for achieving the project objectives. Effective External Communication Project Managers should look for evidence on visibility activities, by discussing with RTA and RTA counterparts on 22

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

the visibility products prepared, events organised, web site contents (frequency of publication, number of hits) and particularly checking how relation with the media is eventually organised. Highlight. The check list template for onthespot checks presented under Section 4.4.2 below should be improved by Project Managers as long as they progress along the monitoring exercise learning curve.

4.4.2 Cheklist for OntheSpot Cheks Within Twinning Projects Project Title: Project No: Date of site visit: Typology of onthespot checks: • Meeting with RTA/Co RTA • Meeting other Stakeholders (please specify) • Attending a project event (please specify) • Other (please specify) All the questions below must be answered with sufficiently detailed answers

Rating: VGvery good, Aadequate, Ppoor, Mmissing (whenever applicable). N/A: not applicable

Action Required

PERSONS INTERVIEWED • Add names and contact reference of interlocutors • •

CHECKS DONE AT EVENTS • Logistics (including interpretation) • Documents distributed (if any) • Participation (absorption capacity) • Degree of satisfaction by participants • Overall assessment

CHECKS DONE AT PROJECT SITE AND/OR WITH PROJECTS BA/MS • Questions raised by Project Manager on IQRs • Project logistics • Evidence of internal monitoring in place • How are electronic/hard copy files are retrieved • Discussion of project risks and risk management/mitigation plan • Discussion of sustainability plan and prospects • Discussion of visibility actions and issues • How internal communication is working (intranet, other) • Are there issues of internal capacity • Are all problems known • Are all problems addressed • Is local ownership developing • Are stakeholders involved OVERALL RECOMMENDATIONS AND DEADLINES FOR REMEDIAL ACTION

Date of elaboration Name of Project Manager Signature Date the report was communicated to the interviewed persons Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

23

4.5 Tool D: How to Compile a Monitoring Report for TPCG Purpose: The Twinning Programme Coordination Group is the highest level for Twinning Programme monitoring. Its members are interested in common issues affecting the Twinning Programme, therefore monitoring reports on individual projects should be prepared in such a way to back general conclusions with detailed evidence. Hence, monitoring reports on individual projects are to be much focused on context and issues, rather than providing a comprehensive review of each project. A standardised PAO Monitoring Report could be used to provide the most suitable information for this top level supervising audience. The TPCG members should receive at least one week before they meet, a set of concise monitoring reports which summarise the results of PAO monitoring activity for the reference period. The standardised reporting format is intended to allow a rapid review and draw attention on issues which need TPCG discussion, decisions and followup. A centralised monitoring report for use by the TPCG could be derived from the individual reports by collating sections 1, 5 and 6 from each individual monitoring report. Template for standardised PAO Monitoring Report 1. 1.1.

EXECUTIVE SUMMARY (0.5 page) Project Title

Title and number as stated in the Work Plan (or Project Fiche). 1.2 Project Partners Quote the Project Partners – Beneficiary Administration and Partner(s) from Member State(s). List only the main players e.g. only those institutions and individuals who are directly responsible for Project implementation. (e.g. Senior Project Officer, Project Leaders, RTA, etc.). Name, institution, contact address (tel/mail). 1.3 Period Monitored by Project Manager This Monitoring Report has been prepared by …. during the period from … to…. and represent the Project's situation at …. the cutoff date for the Report. 1.4

Financial and contract data

Legal Start ……………….. WP Start …………………… WP End ……………………. Legal End …………………. Any addenda to twinning contract. Brief description of addendum/addenda subject (what was actually change).

2. PROJECT INFORMATION (0.51 page) 2.1 Overall Objective and Project Purpose Use Wider Objective as stated in the Work Plan (or Project Fiche). 2.2 Mandatory Results Use information from the Work Plan. 2.3 Key Assumptions Use information from the Work Plan.

3. 3.1.

PROJECT ACTIVITIES DURING THE MONITORING PERIOD (4 pages) Highlights (1 page)

Four or five headlines (bullet points) that effectively summarise the most significant events or main conclusions reached during the monitoring period. Summarise any followup action requested under previous monitoring report and/or discussed during previous PSC/TPCG meeting, and specify whether and how it has been actually followedup.

24

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

Issues/questions raised at previous PSC/TPCG meetings date(s)

Followup action initiated

Responsibility

Status

Issue 1: Issue 2: Issue n: Report on monitoring activities undertaken during the monitoring period under reference. Type of monitoring activity and date

Main findings, conclusions and recommendations

Please insert type of activity (IQR report, site visit)

Briefly state findings, conclusions and recommendations

3.2.

Summary of activities and benchmark by project component (1.5 pages) (compile section 3.2, 3.3…. 3,n according to the number of project components).

Component Activity Title What was realised during the reporting period Benchmarks achieved

Component Activity Title What was realised during the reporting period Benchmarks achieved

3.3

Planned activities for following monitoring period (1.5 pages).

Component Activity Title What was realised during the reporting period Benchmarks achieved

Component Activity Title What was realised during the reporting period

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

25

4. SUMMARY OF KEY ACHIEVEMENTS (1 page) Component 1 Mandatory Result

Benchmarks

Status of achievement

Problems encountered

Status of achievement

Problems encountered

Status of achievement

Problems encountered

From the work plan From the work plan

Component 2 Mandatory Result

Benchmarks From the work plan

From the work plan

Component 2 Mandatory Result

Benchmarks From the work plan

From the work plan

5. MANAGEMENT ISSUES AND CORRECTIVE ACTIONS DURING THE MONITORING PERIOD (0.5 page) Problem / Issue Identified

Proposed Corrective Actions

Responsibility

Status

1. Summarise the problem/issue in a short sentence.

Briefly explain what action was taken or will be taken to correct the problem.

Who will take responsibility for the action?

What is current status? Has any action already been started? If not provide explanation

2.…..

6. ASSESSMENT AND RECOMMENDATIONS (1 page) 6.1 Assessment Summarise assessment on project Relevance, Efficiency, Effectiveness, Impact, Sustainability. 6.2 Recommendations Two or three recommendations or plans for actions that will correct critical elements mentioned in previous section of the monitoring report.

26

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

5. GOOD PRACTICE / EXAMPLES Project Managers in charge of monitoring activities are recommended to collect at least one (good) example of IQR, Work Plan, and External Monitoring Report.

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

27

ANNEXES Annex 1: Key Factors for Successful Project Implementation Allocate clear responsibilities. Responsibilities for implementing the project work plan should be assigned to named individuals as should the promotion of open dialogue and effective project communication. Establish management and monitoring procedures. Arrangements for managing and monitoring project activities should be established, e.g. by establishing the Steering Committee as a group of senior staff from both (all) partner institutions able to make decisions, according to agreed rules of procedures. Expertly manage the implementation risks. Regularly assess the risks of poor and untimely implementation, and devise robust risk management strategies to prevent and/or mitigate the risks identified or which materialise. Ensure Senior Management Commitment. Senior Management should be strongly committed and support organisational change, in order to secure staff involvement in the project activities. Establish good Coordination. Effective management and interaction should be operational at all levels. Develop effective Project Communication. Ensure discussion between parties involved, by organising internal communication meetings at all project levels and in all phases, to ensure flexibility and provide regular feedback at key project stages. Involve as many staff and stakeholders as appropriate. The allocation of staff within the Beneficiary Administration specifically devoted to project activities, possibly on a fulltime basis, is instrumental to ensure smooth implementation. The more the Beneficiary Administration is large and/or its scope (and consequently the twinning project) is large, the more the twinning project should involve as many staff and stakeholders as possible, including eventually at regional level. Train staff. Experienced and skilled staff with potential and qualities to train other staff should be designated. Followup Procedures. Have followup procedures in place in order to ensure that lessons learned through the project are fed back into the development and improvement of sound monitoring practices and procedures in a coherent manner. Visibility. Make extensive use of the web. Develop and implement different communication tools fine tuned for selected audiences (funding organisations, sectoral experts, media, general public). PostTwinning. Plan to continue with development and training activities after the project ends, and allocate sufficient and suitable resources to the sharing and spreading of knowledge and skills afterwards.

Annex 2: Criteria for Quality Assessment of Twinning Projects Design and Implementation25 The Common Twinning Manual (CTM) in a Nutshell The Common Twinning Manual specifies the rules to be applied for twinning projects summarized below which as it stands are mandatory for all Twinning projects. The listing includes in addition highlighted procedural instructions and definitions. Highlight. It is recommended to use the listings for quality checks and to identify deficiencies which then have to be analysed considering their severity for the project.

25

Monitoring Project Managers are advised to consult the following sections of the European Commission 'Project Cycle Management Guidelines' (2004): Quality Attributes, Criteria and Standards at Identification (Section 4.3.4); Quality Attributes, Criteria and Standards at Formulation (Section 4.3.4); Quality Attributes, Criteria and Standards during Implementation (Section 4.5.7). http://www.interaide.org/pratiques/pages/autres/Europaid%20%20PCM%20Manual%20%202004.pdf

28

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

The mandatory result must be well defined, focused and achievable. The mandatory result must make a specific and direct contribution to Institution Building in the BC. The mandatory result must be concrete, clearly measurable for control purposes, inter alia through adequate indicators, defined as 'benchmarks' in the Common Twinning Manual. The achievement of the mandatory result must remain at the disposal of the BC administration as a sustainable asset. All components of a Twinning project must be clearly and independently identified. Each component and the actions identified to achieve the component must be fully justified in the context of achieving the respective mandatory result. Work plan components must be directly linked to an identifiable component in the budget (see CTM Section 5). The Twinning work plan must specify benchmarks as planned at regular time intervals, which will be used to measure progress. The Twinning work plan must clearly identify timeframes for the project as a whole and for all project components. All significant risks, both internal and external, should be clearly stated and quantified as far as possible. The Twinning work plan should also identify ways of minimising controllable risks. The split of responsibilities must be defined for each component. For components which are a joint responsibility, the Twinning work plan should state which party is the leader. Attribution of responsibility for a particular activity must be directly linked to entitlement to an explicitly identified budget allocation. The Twinning work plan must detail the responsibilities of each of the partners in respect of the individual project components clearly stating who should do what, when and with which resources (as identified in the detailed breakdown of costs). The Commission requires a general mandate (see Annex A8 to the Twinning Contract) from the junior MS partner to the Project Leader appointed by the lead MS, providing the power to commit its administration and take any project implementation decisions on its behalf. The signature of this mandate is a precondition for submitting the Twinning Contract.

Significant staff inputs will be required in the various components of the project: At the broadest and most senior level, there must be political commitment and support within the BC administration for the Twinning project as a whole. Each Twinning project must have a BC Project Leader with sufficient authority to administer and properly implement the project in practice. This person will also be named in, and sign, the Twinning Contract, as the figure ultimately responsible for its implementation. Each Twinning project must have a RTA counterpart in the BC. At project level, BC staff must actively participate (i.e. working with the RTA and short term experts, participating in training activities) to implement a particular project component.

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

29

BC staff inputs should be detailed as far as possible within the Twinning work plan to ensure that there is full recognition of both the practical and political commitments required to achieve the desired results. A change in BC Project Leader must be notified in writing according to procedures outlined under CTM Section 6.6. All infrastructure requirements should be specified in the Twinning Contract, Twinning work plan and budget. Only a provision limited to 5.000 euro for miscellaneous supplies is eligible for EU funding (see CTM Section 5.11). This will only be available in very restricted cases, subject to evidence of the BC's inability to provide the material required. All key staff, including the Project Leader, the RTA, and the principal short and medium term experts must provide full CVs, which will be attached to the Twinning Contract. The profile of the other experts should appear in the Twinning work plan. Experts cannot be included into the project through MS management support organisations, be they mandated bodies or not, but can only come from or through the implementing organisation holding the project leadership, as well as the required expertise (CTM Section 4.5.5). BC and MS administrations will be entirely responsible for their respective part of the general management, logistical and financial management in relation with a Twinning project. The requirements for logistical and accounting support should be clearly taken into account by the MS preparing a Twinning work plan. RTAs will be expected to manage their own allowances and claims for statutory reimbursements but they should ensure adequate documentation for the purpose of the Final report and Audit certificate. EU funds will not be used to fund BC running costs. For the sake of longterm sustainability and in order to ensure that systems are established which are commensurate with BC future funding capacity, BCs are expected to commit their own resources to Twinning projects. See also Section 9.2 on Sustainability. The Project Leader of the MS will be required to submit formal reports and the Project Leader of the BC will be fully involved in this process and will be required to cosign them (see CTM Section 6.4). The proper implementation of Twinning projects may be assessed in the framework of monitoring bodies set up by the respective EU programme, in order to review results and achievements of the project against those set out in the work plan. Twinning contract modifications can only be made within the period of implementation of the Contract. The modifications will only apply to subsequent implementation and cannot apply retroactively.

The following changes to a Twinning Contract thus require a formal addendum: The relevant field of cooperation with the EU and the Community Acquis related to the project. (CTM Work Plan Template Article 2). Mandatory results (CTM Work Plan Template Article 3). MS administration involved in the Twinning project as mentioned in Article 5 of the Twinning Contract. Implementation period of the Action (Twinning Contract Article 2) Definition of the mandatory results and the benchmarks to be achieved (CTM, Work Plan Template Articles 3 and 4). For changes concerning the means used for implementation, the time schedule and dates, and the identity of nonkey MS' short term experts an addendum is not needed.

30

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

Identity of the MS and BC Project Leaders, the RTA and the main MS shortterm experts (CTM Work Plan Article 6). Interruption or termination of the Twinning contract before completion. Suspending the funding or lifting the suspension of funding for a Twinning Project. Major reallocations beyond 15% of the total Twinning Budget.

If the change sought does not concern one of the points listed above, side letters are sufficient where there are: Changes which do not affect the basic purpose of the project. Reallocations below 15% of the total Twinning budget. The Twinning Contract is considered to have been changed on the date of the latest notification. Changes must be notified before their implementation, even if it is only the day before. A side letter (administrative order), is not an addendum in legal terms. Costs for changes notified after their implementation cannot be reimbursed. Minor changes such as changes of address, changes of bank account and changes of auditor may simply be notified and signed by the two Project Leaders (MS and BC), although this shall not affect the Contracting Authority's right to question the signatory Member State's choice of bank account or auditor.

The following principles apply to all budgetary changes whatever their impact: The overall budget for a Twinning project cannot be increased, and so an activity must be reduced or cancelled first in order to finance a new one. The unit costs (fees, daily allowances, etc.) must respect the rates set in this Manual; Each Twinning project must include a fulltime Resident Twinning Adviser, resident in the BC for a minimum of 12 consecutive months. Transfers of budgetary resources may not jeopardise this requirement. Introduction of a new activity must be justified by showing that it will be of real use in achieving the mandatory results targeted by the Twinning Contract; the mere availability of funds (following savings under or cancellation of activities originally foreseen) is not sufficient to justify the financing of new activities. Twinning Contracts can only be modified within the life time of the Contract. Modifications cannot be done retroactively. The budget should be formally reallocated before the new activity can be implemented. Activities implemented before being officially entered into the budget will not be financed. Specific budgetary changes through addenda: Budgetary changes are summed up after each side letter. Once the total amount of modifications reaches 15% of the total budget, any further modification (irrespectively of its size) to the Twinning budget should be done via an addendum to the Twinning Contract, including full approval of all the signatories of the contract and approval by Commission.

Annex 3: Description of the Five Evaluation Criteria External ResultOriented Monitoring is based on 5 criteria which are commonly used throughout EuropeAid assistance for assessing projects. This Annex provides the reader with a full description of the five evaluation criteria: relevance, efficiency, effectiveness, impact and sustainability. For Twinning projects the participatory approach throughout the planning and implementation phases is key and therefore the involvement into the monitoring process forms an integral part of it.

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

31

1. Relevance The relevance of a project relates primarily to its design and concerns the extent to which its stated objectives correctly address the identified problems or real needs. It needs to be kept under review throughout the life of the project in case changes occur either in the nature of the very problems originally identified, or in the circumstances – whether physical, political, economic, social, environmental, institutional or policy – in which the project takes place, necessitating a corresponding change of focus. In other words, relevance concerns the appropriateness of the project design to the problems to be resolved at two points in time: when the project was designed, and at the time of the evaluation. However, the flexible handling of any changes needed to keep a project relevant forms part of the definitions of the other evaluation criteria, given below.

Analysis of Relevance systematically focuses on the following: • Identification of real (as distinct from perceived) problems or needs, and of the correct beneficiaries, and how well the project's initial design addressed the problems and needs in relation to the intended beneficiaries; • Quality of assessment of local absorption capacities; • Quality of assessment of local implementation capacities; • Preparatory activities jointly undertaken (policy assessments, sector reviews, (pre) feasibility studies including financial and economic analysis, planning workshops, etc.), by whom, how well the findings were incorporated into the final project document, and any obvious omissions; • Appropriateness of initial consultations with, and participation by, local key stakeholders including the Delegation, national authorities, intended beneficiaries, and other donors (the lastmentioned especially on complementarity aspects) before the design was confirmed and implementation started; • Complementarity and coherence with related activities undertaken elsewhere by government or other donors, at the same level or at a higher level, rather than duplication, overlapping or conflict; • The quality of the entries in the assumptions, risks and conditions column of the Logframe at the appropriate levels; • Overall design strengths and weaknesses including: quality of the Logframe (or Logframes if a multi component programme); clarity and internal consistency of the stated overall objectives, purpose and results; whether the objectivelyverifiable indicators of achievement (OVIs) were well chosen and widely agreed; realism in choice and quantity of inputs; overall degree of flexibility and adaptability to facilitate rapid responses to changes in circumstances.

2. Efficiency The efficiency criterion concerns how well the various activities have transformed the available resources into the obligatory mandatory results (sometimes referred to as outputs), in terms of quantity, quality and timeliness. A key question it asks is 'have things been done right?' and thereby also addresses valueformoney, e.g., whether similar results could have been achieved better by other means at lower cost within the same amount of time. The analysis of Efficiency will therefore focus on: • The quality of daytoday management, for example in: management of the budget (including whether an inadequate budget was a factor); management of personnel, information, property, etc; whether management of risk was adequate, i.e. whether flexibility was demonstrated in response to changes in circumstances; relations/coordination with local authorities, institutions, other donors; meeting established deadlines. • Costs and valueformoney: how far the costs of the project were justified by the benefits – whether or not expressed in monetary terms – that they generated, in comparison with similar projects or known alternative approaches, taking account of contextual differences; 32

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

• Country contributions from local institutions and government (e.g. offices, experts, tax exemption, as set out in the Logframe resource schedule), target beneficiaries and other local parties: were they provided as planned, could reallocation of responsibilities have improved performance, has communication been good? • Commission HQ/Delegation inputs (e.g. procurement, training, contracting, either direct or via consultants/ bureaux): key questions as for local/government inputs (above); • Technical assistance/twinning: how well did it help to provide appropriate solutions and develop local capacities to define and produce results? • Quality of monitoring: its existence (or not), accuracy, flexibility and utility; adequacy of baseline information; • Whether the chosen indicators of efficiency were suitable and, if not, whether management amended them; • Did any unplanned results arise from the activities?

3. Effectiveness The effectiveness criterion, in the Logframe terminology, concerns how far the project's results were used or their potential benefits were realised – in other words, whether they achieved the project purpose. The key question is what difference the project made in practice, as measured by how far the intended beneficiaries really benefited from the products or services it made available.

The analysis of Effectiveness will therefore focus on: • Whether the planned benefits have been delivered and received, as perceived mainly by the key beneficiaries, but also taking account of the views of donor management, the responsible national Government authorities, and other concerned parties (NGOs, business associations etc); • The appropriateness of the indicators of benefit used in the above assessment to measure achievement of the project purpose (this is also relevant to cost effectiveness analysis as referred to in the footnote on the previous page); this should include a judgement on how promptly and effectively the project management reacted to any changes that occurred following the initial design by amending indicators found no longer to be appropriate; • In institutional reform projects, whether behavioural patterns have changed in the beneficiary organisations or groups at various levels; and how far the changed institutional arrangements and characteristics have produced the planned improvements (e.g. in communications, productivity, ability to generate actions which lead to economic and social development); • If the assumptions and risk assessments at results level turned out to be inadequate or invalid, or unforeseen external factors intervened, how flexibly management adapted to ensure that the results would still achieve the purpose; and how well it was supported in this by key stakeholders including Government, Commission (HQ and locally), etc.: in summary, 'were the right things done' to ensure that the potential beneficiaries actually benefited? • Whether the balance of responsibilities between the various stakeholders was correct, which accompanying measures were or should have been taken by the partner authorities, and with what consequences; • How unplanned results may have affected the benefits received; • Whether any shortcomings at this level were due to a failure to take account of crosscutting or overarching issues such as gender, environment and poverty during implementation.

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

33

4. Impact The term impact, sometimes referred to as outcome, denotes the relationship between the project's purpose and overall objectives, that is the extent to which the benefits received by the target beneficiaries had a wider overall effect on larger numbers of people in the sector or region or in the country as a whole. The analysis, which should be both quantitative and qualitative wherever possible, will need to take account of the fact that, at this level, the project will normally be only one of a number of influences contributing to the wider outcome.

At Impact level the analysis examines such aspects as: • To what extent the planned overall objectives have been achieved, and how far that was directly due to the project; • In institutional reform projects, to what extent enhanced economic and social development have resulted from improved institutional capabilities and communications; • In infrastructuretype projects, to what extent have they also enhanced economic and social development beyond the level of their immediate users? • How flexibly donor management and the Government responded and reacted to unforeseen external shocks; • If there have been unplanned impacts, how strongly/to what extent they have affected the overall impact; • Whether the project's Logframe indicators at this level were suitable and, if not, whether management amended them; • Where appropriate, all genderrelated, environmental and povertyrelated impacts and any lack of overall impact resulting from neglect of these issues; • Whether overall the desired wider impact could have been better achieved otherwise; • To what extent the economic effects have been spread between economic growth, salaries and wages, foreign exchange, and budget, and to what extent this is related to the achievement of the project's overall objectives.

5. Sustainability The fifth and final – and often most important – criterion, sustainability, relates to whether the positive outcomes of the project at purpose level are likely to continue after external funding ends, and also whether its longer term impact on the wider development process can also be sustained at the level of the sector, region or country. An analysis of sustainability will therefore focus on the aspects below. Their relative importance will depend on the nature of the project; it is useful to examine how strongly concern for, or neglect of, one or other of the factors may have affected achievement of a sustainable outcome: • Ownership of objectives and achievements, e.g. how far all stakeholders were consulted on the objectives from the outset, and whether they agreed with them and remained in agreement throughout the duration of the project; • Policy support and the responsibility of the beneficiary institutions, e.g. how far donor policy and national policy corresponded, and the effects of any policy changes; how far the relevant national, sectoral and budgetary policies and priorities affected the project positively or adversely; the level of support from governmental, public, business and civil society organisations; and whether national bodies to provided resources;

34

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

• Institutional capacity, e.g. the degree of commitment of all parties involved, such as Government (e.g. through policy and budgetary support) and counterpart institutions; the extent to which the project is embedded in local institutional structures; if it involved creating a new institution, how far good relations with existing institutions were established; whether the institution appears likely to be capable of continuing the flow of benefits after the project ends (is it wellled, with adequate and trained staff, sufficient budget and equipment?); whether counterparts were properly prepared for taking over, technically, financially and managerially; • The adequacy of the project budget for its purpose.

Annex 4: Using the LFA during the Project Cycle. Strength and Common Difficulties A brief summary of how LFA (particularly the matrix, activity and resource schedules) can be used during project formulation, implementation and evaluation is provided below: Use of LFA during the Project Cycle Project Cycle Stage Preparation

Use of LFA • The Logframe Matrix provides a summary of key project elements in a standard format, and thus assists those responsible for appraising the scope and logic or proposed investments. • The tools that make up LFA can be applied during exante evaluation to deconstruct the proposed project, to further test its relevance and likely feasibility. • The objectives specified in the Logframe, combined with the activity, resource and cost schedules, provide information to support costbenefit analysis. • The costschedules allow cashflow implications to be assessed (including the contributions of different stakeholders), and the scope of Financing Agreements to be determined. • The Logframe provides a basis on which contracts can be prepared – clearly stating anticipated objectives, and also the level of responsibility and accountability of project managers and other stakeholders. • The Logframe and associated schedules provide the basis on which more detailed operational work plans can be formulated.

Implementation

• The Benchmarks and Means of Verification provide the framework for a more detailed Monitoring and Evaluation Plan to be designed and implemented by project managers. • The Assumptions provide the basis for elaborating a risk management plan. • The Mandatory Results, Benchmarks and Means of Verification (together with activities, resource and costs) provide the framework for preparing reports on project progress, to compare inter alia what was planned with what has been achieved.

Evaluation and Audit

• The Logframe provides a framework for evaluation, given that it clearly specifies what was to be achieved (namely specific objectives and mandatory results), how these achievements were to be verified (Benchmarks and Means of Verification) and what the key assumptions were. • The Logframe provides a structure for preparing ToR for Evaluation studies and for performance audits.

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

35

Strengths and Common Difficulties with the Application of the LFA Element Problem analysis and objective setting

Strengths

Common Difficulties

• Requires systematic analysis of problems, including cause and effect relationships. • Provides logical link between means & ends. • Places the project within a broader development context (overall objective and purpose).

• Getting consensus on priority problems. • Getting consensus on project objectives. • Reducing objectives to a simplistic linear chain. • Inappropriate level of detail (too much/too little).

• Encourages examination of risks and management accountability for results. Benchmarks and source of Verification

• Requires analysis of how to measure the achievement of objectives, in terms of both quantity and quality.

• Finding measurable and practical indicators for higher level objectives and for projects with 'capacity building' and 'process' objectives.

• Helps improve clarity and specificity of objectives.

• Establishing targets too early in the planning process (such targets could prove unrealistic).

• Helps establish the monitoring and evaluation framework.

LGM Format and application

• Links problem analysis to objective setting. • Emphasises importance of stakeholder analysis to determine 'whose problems' and 'who benefits'. • Visualizes concepts thus being relatively easy to understand.

• Relying on 'project reports' as the main 'source of verification', and not detailing where the required information actually comes from, who should collect it and how frequently. • Prepared in a mechanical manner as a bureaucratic 'boxfilling' requirement, not linked to problem analysis, objective setting or strategy selection. • Used as a means of topdown control too rigidly applied. • Can alienate staff not familiar with the key concepts. • May become a 'fetish' rather than a help.

Annex 5: Glossary of Key Terms Terms related to monitoring 27 External monitoring is distinguished from 'internal monitoring' because it involves external agents (e.g. donor officials or contracted consultants), and the use of donor designed/approved monitoring methods and reporting formats, which are designed primarily to meet the donor's own reporting and accountability requirements. The Results Oriented Monitoring (ROM) system is a key example of an 'external monitoring' and reporting requirement. Internal project monitoring refers specifically to monitoring that is undertaken by the project's implementing partners, using their own (local) systems and procedures, to meet their own management information needs. Monitoring by ECD Task/Sector Managers refers to the role of Managers with respect to collecting, analysing and using information about project progress and performance, whether it is sourced from 'internal' or 'external', formal or informal sources. Project Monitoring refers generally to the ongoing collection, analysis and use of information about project progress and the results being achieved. It supports effective and timely decision making, learning by project stakeholders and accountability for results and the resources used. Regular review refers to a structured process of review, reflection and decision making that is undertaken by the project implementing partner and other stakeholders on a regular basis. It is therefore an element of the broader monitoring process which gives particular focus to sharing information among stakeholders, and to making 27

As used in: EuropeAid: Strengthening project internal monitoring  How to enhance the role of EC task managers: Tools and Method Series Reference Document No. 3 June 2007. http://ec.europa.eu/europeaid/multimedia/publications/documents/tools/europaid_strengthening_project_internal_monitoring_en.pdf

36

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

decisions about followup actions required. Project Steering Committees (or similar) might often be one of the key fora in which such regular reviews are undertaken. Other relevant terms Baseline – state of the economic, social or environmental context, at a given time (generally at the beginning of the project), and against which changes will be measured. The basic situation is described by context indicators which describe the economy, socioeconomic environment, concerned groups etc. Benchmark – qualitative and quantitative standard for comparison of the performance of an intervention. Such a standard will be often the best in the same domain of intervention or in a related domain. Benchmarking is facilitated when, at the national or regional level, there is comparative information of good and not so good practice. Implementing partners – refers to the organisations which have direct responsibility and authority for project implementation, including management of the available resources, implementation of activities and achievement or results. A project may have one or more implementing partner, and such partners may include government agencies, nongovernment organisations and/or private contractors. Lesson Learned – knowledge and experience, positive or negative, derived from actual incidents as well as observations and historical studies of operations. Milestone – a scheduled event signifying the completion of a major deliverable or a set of related deliverables. A milestone has zero duration and no effort – there is no work associated with a milestone. It is a flag in the work plan to signify some other work has completed. Usually a milestone is used as a project checkpoint to validate how the project is progressing and revalidate work. Milestones are also used as highlevel snapshots for management to validate the progress of the project. In many cases there is a decision to be made at a milestone. Objectively Verifiable Indicators – measurable indicators that will show whether or not objectives have been achieved at the three highest levels of the logframe. OVIs provide the basis for designing an appropriate monitoring system. Project Cycle Management – a methodology for the identification, preparation, implementation, monitoring and evaluation of projects and programmes. PCM methodology is based on the principles of the Logical Framework Approach. It helps specify key tasks, quality assessment criteria, roles and responsibilities and decision making options to support effective management, the achievement of desired results and learning from experience. Project Evaluation – refers to the periodic assessment of the efficiency, effectiveness, impact, sustainability and relevance of a project in the context of stated objectives. It is frequently undertaken at or after completion and usually involves independent evaluators with a primarily purpose of learning lessons to guide future decision making, design and implementation of other projects, future programming and policy making. Results – this term is used generally to refer to the outputs, deliverables, outcome and/or impact of a project (as distinguished from the inputs and activities). This is different from the way the term is used in the EC's PCM Handbook – where 'results' refers specifically to one level of the Logframe Matrix hierarchy of objectives – namely just the outputs. Risk – the threat or probability that an action of event will adversely affect an organisation's ability to achieve its objectives. Stakeholders – individuals, groups or organisations with an interest in the monitored/evaluated operation, particularly: authorities who decided on and financed the operation, managers, and operators and spokepersons of the public concerned. Sector/Project/Task Manager – refers to an EC officer who has an operational responsibility for overseeing and supporting the effective formulation, implementation and/or monitoring of specific development projects or programmes financed by the EU. Most Sector/Project/Task Managers are now based at Delegations and usually have responsibility for a broad (and often diverse) portfolio of 'activities'.

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009

37

Annex 6: Web References There are 1,720,000 quotes in the web for 'Project Cycle Management Handbook'28 and several versions of PCM Handbooks and Guidelines downloadable from the EuropeAid homepage; however the 2004 version (see reference nr. 4) is methodologically much clearer and therefore recommended for studies.

28

1

European Commission, Common Twinning Manual 2009

http://ec.europa.eu/enlargement/howdoesitwork/ financialassistance/institution_building/twinning_en.htm

2

European Commission, EuropeAid Cooperation Office Handbook for ResultsOriented Monitoring of EC External Assistance (projects and programmes) April 2008

http://ec.europa.eu/europeaid/how/ensuringquality/rom/ documents/handbook_rom_system_final_en.pdf

3

European Commission, external cooperation programmes, various guidelines, see in particular 'Strengthening Internal Project Monitoring', 2007

http://ec.europa.eu/europeaid/multimedia/publications/ index_en.htm http://ec.europa.eu/europeaid/multimedia/publications/ publications/manualstools/t108_en.htm

4

European Commission, Common Twinning Manual http://ec.europa.eu/enlargement/pdf/financial_assistance/ institution_building/final_version_of_the_manual_060707.pdf 2007

5

European Commission, Guidelines on Aid Delivery http://www.interaide.org/pratiques/pages/autres/ Methods, volume 1, Project Cycle Management, Europaid%20%20PCM%20Manual%20%202004.pdf February 2004 see pages 150151 useful web references

6

EuropeAid's Project Cycle Management Handbook March 2002

http://www.stgm.org.tr/docs/1123450143PCM_Train_Hand book_ENMarch2002.pdf

7

Tearfund's Project Cycle Management tools

http://tilz.tearfund.org/Publications/ROOTS/Project+cycle+ management.htm

8

A portal listing 25 M&E resource material

http://portals.wi.wur.nl/ppme/?page=2136

As of September 2009.

38

Handbook on Internal Monitoring of Twinning Operations in Ukraine, 10/2009