Performance Evaluation Methodology And Systems

Budget Performance Evaluation Methodology by Tarun Das

Budget Performance Evaluation (BPE)Methodology, Systems and Management Professor Tarun Das1 The Lexicon of Performance “Wherever there is government, there is government performance. On the basis of such obvious statements, modest reputations are earned! However, what constitutes performance within government is more complicated, pluralistic, value laden, and controversial than is true with the performance of private firms, where performance yardsticks are more limited and more universally accepted. In government, performance is regarded as progress toward goals and objectives, but measurement is complicated by the fact that the outcomes are often multiple, vague, shifting and even conflicting; as they emerge out of wider political process and public debate.” “In diverse countries such as Canada or Australia, individuals, groups, and regions will often disagree strongly over what constitutes good performance. Also, government performance is more subject, than corporate performance, to continuous and critical scrutiny by opposition parties, interest groups and the media. So, for governments, appearances matter as much or even more than the reality of performance. Maintaining legitimacy and support for the government policies and programs is also a legitimate performance goal. In short, performance is a many splendoured thing, and good performance is partly in the eyes of the beholder.” Source: Paul G. Thomas (2006) Performance reporting, obstacles and accountability- Recent trenda and future directions, The Australian National University, Canberra ACT 0200

Ministry of Finance Government of Mongolia ADB Capacity Building Project On Governance Reforms Ulaanbaatar, Mongolia 20 May 2008 _______________________________________________________

Glocoms USA Inc. Strategic Planning Expert. Formerly, Economic Adviser, Ministry of Finance and Planning Commission, Government of India and Professor (Public Policy), Institute for Integrated Learning in Management (IILM), New Delhi, India. 1

MOF, Govt. of Mongolia

1

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Budget Performance Evaluation (BPE) Methodology, Systems and Management Professor Tarun Das Contents 1. Introduction, Scope and Objectives 1.1 1.2

1.3 1.4

The Budget and Public Sector Performance Performance Based Budgeting Integration of Strategic Plan with Program Budget Performance Monitoring and Evaluation

2. Ideal Performance Evaluation Systems

2.1 2.2 2.3

Characteristics of an ideal performance evaluation system Various Models of Performance Evaluation Choice of Particular Performance Evaluation System

3. Budget Performance Indicators 3.1 3.2 3.3 3.4

Performance Measures versus Performance Indicators Types of Performance Indicators Characteristics of an Ideal Performance Indicator Use of Performance Measures

4. An Operational Budget Performance Evaluation System for the Government of Mongolia 4.1 Hierarchy of Activities, Functions and Output Classes 4.2 Types of Budget Performance Evaluation 4.2.1 Input/ resource use evaluation (compliance evaluation) 4.2.2 Output/ service evaluation (efficiency evaluation) 4.2.3 Clients’ satisfaction evaluation (efficiency evaluation) 4.2.4 Outcomes/ Community evaluation (effectiveness evaluation) 4.3 Benchmarks 5. Budget Performance Evaluation Methodology 5.1 5.2

5.3 5.4 5.5

Analysis of strategic plan and baseline profile Diagnostic scan and SWOT analysis Strategic and Operational Performance evaluation Budget Compliance, Efficiency and Effectiveness Evaluation Overall Assessment and Score

MOF, Govt. of Mongolia

2

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

5.6

Major findings and recommendations

6. Budget Performance Evaluation Management Selected References Annex-1: Template for Review of Strategic Business Plan and Baseline Profile of the Line Ministry/ Budgetary Agency Annex-2: Template for Strategic Review and Operational Review Annex-3: Template for Budget Compliance, Efficiency and Effectiveness Evaluation Annex-4: Methodology for calculation of composite score and grading the Line Ministry/ Budgetary Agency Glossary: Definition of Terms List of Tables: Table-1: Characteristics of an Effective Budget Table-2: Characteristics That Adversely Impact Budget Performance Table-3: Characteristics of an Ideal Performance Measurement System Table-4: Budget Performance Indicators Table-6: Broad Categories of Budget Performance Evaluation Table-6: Scope of Performance Review and the Initial Profile Table-7 Strategic Review and Operational Review Areas Table-8A: Typical Best Practices for Strategic Review Table-8B: Typical Best Practices for Operational Review Table-9 SWOT Analysis for the Ministry of Finance Table-10: Scores for Strategic and Performance Evaluation Table-11: Rating of an Agency on the basis of Performance Scores Table-12: Marks for Budget Compliance Evaluation Table-13: Marks for Budget Efficiency Evaluation Table-14: Marks for Budget Effectiveness Evaluation Table-15: Weights for Various Types of Evaluation Table-16: Rating of an Agency on the Basis of Overall Scores Table-17: An Example of Performance Scores for three Agencies Table-18: Estimation of Overall Rating for three Agencies Table-19: Deadlines for various steps in Budget Performance Evaluation List of Flow Charts Flow-Chart-1: Strategic Plan and Performance Based Budgeting Cycles: Flow Chart-2: Integration of Strategic Business Plan, Program Budget and Performance Evaluation Flow Chart-3: Pyramidic Structure of Performance Evaluation Flow Chart-4: Deadlines for various steps in Budget Performance Evaluation

MOF, Govt. of Mongolia

3

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Budget Performance Evaluation (BPE) Methodology, Systems and Management Professor Tarun Das 1. Introduction, Scope and Objectives “Would you tell me please, which way I ought to go from here?” asked Alice. “That depends a great deal on where you want to get to,” said the cat. “I don’t much care where ….,” said Alice. “Then it does not matter which way you go,” said the cat. --- Extract from Alice in Wonderland, Lewis B. Carroll

Eventually Alice in Wonderland realized that it matters a great deal to know “where to go” and “how to get there”. Similarly, in government budget it is important to know the basic goals and objectives and the overall scope of budgeting in terms of exact outputs and outcomes, and how to achieve these goals, objectives, outputs and outcomes in time and with least cost. Strategic Planning, Budget Formulation, and Performance Management Cycle is an interactive on-going process to facilitate sound government business and financial plan. Budget Performance Evaluation (BPE) makes a detailed and critical analysis of the multi stakeholders’ requirements, develops action plans and allocates resources by integrating budget and performance goals. Government resources are not unlimited and there are constraints on public resources and competing demands by various social sectors, public goods and services. Consequently, an optimal resource planning and budgeting requires that strategic planning, budget and performance management must be integrated in a comprehensive and time bound plan. 1.1 The Budget and Public Sector Performance Performance budgeting is primarily a management, transparency and financial accountability tool rather than a mere budgetary instrument. Effective and efficient delivery of public goods and services by budgetary agencies requires performance management, under which an Agency establishes its service objectives and monitors performance towards the attainment of those objectives. Meyers (1996) has identified 11 characteristics of an effective budget which supports performance. These characteristics are listed in Table-1

MOF, Govt. of Mongolia

4

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Table-1: Characteristics of an Effective Budget Term 1. Accountability

Characteristics Detailed controls should ensure that government policy directives are carried out by managers, contractors and all concerned parties.

2. Comprehensiveness Includes all uses of government’s financial resources. 3. Constraint

Contains limitations on the amount of money that can be raised and spent by government.

4. Cooperation

Budgeting should exist in harmony with other decision processes and should not be dominant.

5. Honesty

Contains unbiased projections and allocations under different programs and heads of accounts.

6. Judgment

Encourages participants to seek the most effective ways to achieve the least costs.

7. Legitimacy

The budget process should reserve important decisions to legally appropriate authorities.

8. Perception 9. Responsiveness

Considers both near- and long-term plan. The budget must adopt policies that match public preferences.

10. Timeliness

A budget process should complete regular tasks within expected time period.

11. Transparency

The budget and budget procedures should be understood by participants and outside stakeholders.

Source: Meyers, Roy T. (1996).

Some of these characteristics for an effective budget relate to the process of budget prioritization (comprehensiveness, perception), while others relate to operational efficiency (cooperation, timeliness). Major factors that might affect adversely the budget performance are summarized in Table-2.

Table-2: Characteristics That Adversely Impact Budget Performance MOF, Govt. of Mongolia

5

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Term

Characteristic

1. Extra-budgetary Funds

Extra-budgetary funds lead to dual budgeting and do not allow tradeoffs between competing priorities. Extra-budgetary funds generally circumvent central controls, thereby violating the comprehensiveness standard. Earmarked funds are dedicated to particular uses or purposes. Agencies with earmarked funds may have little or no incentive to economize. Earmarking may impede the government’s ability to be flexible in managing its budget or in allocating resources. Inaccurate budget estimates, arbitrary budget cuts, and non-timely fund disbursements will create substantial budget unpredictability.

2. Earmarked Funds (Earmarking)

3. Unpredictability

4. Unreliable Information

Inaccurate and unreliable information will conspire against enhancing budget performance.

Source: Meyers, Roy T. (1996).

1.2 Performance Based Budgeting A performance budget is an integrated annual performance plan that shows the relationship between program funding levels and expected results. It indicates that a goal or a set of goals should be achieved at a given level of funding. An effective performance budget does more than act just as a program budget with anticipated outcomes. It identifies and explains the relationships between money spent and results. Such explanation is key to manage a program effectively. Whenever there are variances between plans and actual results, managers examine the resource inputs and how they relate to outcomes to determine program effectiveness and efficiency. A program performance budget defines all activities, direct and required by a program, in addition to estimating activity costs. The figure outlines this process. By tracking the cost and number of each activity, output, and outcome, unit cost information also generated. Money

⇒ Activity

⇒ Output

⇒ Outcome

indirect, following units for may be

⇒ Strategic Goals

Regardless of agency level, managers must know accurate and timely cost and performance information to manage their resources most effectively. This applies to both administrative officials and program managers. In certain MOF, Govt. of Mongolia

6

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

cases (e.g., computer services), managers require this information to establish unit prices. In other cases, managers simply want to ensure that every unit of money supporting a program is spent wisely. Recently many countries have focused their attention to governance reforms and the role of performance based budgeting. As in the case of many aspects of Budget Modernization techniques, Australia and New Zealand have developed over the years very extensive performance based budgeting methodology and systems. The common principles in Australia and New Zealand relate to governance reforms and the institutionalization of performance standards in budgets: Budget transparency is the key issue. In Australia, for example, there is a requirement for three-year forward estimates and for reporting of budgetary versus actual budgetary allocations for the previous year. Under the Public Sector Management and Finance Act (PSMFA, 27 June 2002) government of Mongolia has also adopted the same requirement2.

(i)

Decentralization of operating functions is another important development. There has been a substantial devolution of management functions to operating line agencies to perform their tasks. Under the system of portfolio budgeting, line agencies are required to prepare multiyear Strategic Business Plans indicating vision, mission, objectives and priorities, and identifying spending and revenue measures necessary to meet aggregate fiscal targets set by the Head Quarter. PSMFA of Mongolia also mandates the same requirements3.

(ii)

The primary difference between the Australia and New Zealand performance based budget frameworks is a tradeoff between a focus on outputs and a focus on outcomes. For example, in New Zealand, the focus is on technical efficiency in the delivery of outputs, whereas in Australia the focus is on outcomes (i.e., the impact of outputs on beneficiaries). Mongolian PSMFA puts emphasis on both technical efficiency and outcomes. The main reason for the diverse paths is due to the underlying conditions previous to the reforms. In New Zealand, the pre-reform state was characterized by substantial budget overspending and pre-dominant role of the public sector services. An important component of the reforms program was the substantial privatization of these services. So in New Zealand, performance is based on formal accountability mechanisms and sanctions. On the other hand, Australia did not have an extended public sector. The budget process evolves a system where political leaders set strategic priorities, and operating line managers are given considerable discretion to manage their agencies and programs to achieve the intended outcomes. In 2 3

Articles 26.1, 26.2 and 26.3 of the Public Sector Management and Finance Act (PSMFA, 27 June 2002). Articles 26.1, 26.2 and 26.3 of the Public Sector Management and Finance Act (PSMFA, 27 June 2002).

MOF, Govt. of Mongolia

7

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

the Australian system there are no formal output based contracts. Public accountability is achieved through performance reporting and evaluations. 1.3 Integration of Strategic Plan with Program Budget To accomplish its strategic objectives effectively, an Agency must link outcomes, strategy, budget, and performance into an integrated management system. This management system, based on a model of continuous improvement, is shown in Flow Chart-1 below:

Strategic Business Plan

Broadly defines strategic goals, outcomes, outputs of an Agency and the methods to achieve them.

Performance Assessment and Improvement Plan

Compares actual to target performance and benchmarks. Determine changes that will produce the best value.

Program Budget and Performance Planning Program Budgets- Funds allocated to specific programs to achieve desired goals, outputs, and outcomes with least cost. Performance - specifically designed results Value- achieving value for money Establish long-term and annual targets for spending, performance and value.

Performance Monitoring Track the progress, expenditure, and value for money for achieving outcomes.

Flow-Chart-1: Strategic Plan and Performance Based Budgeting Cycles: Source: John Mercer, regarded as the father of the USA GPRA, 1993.

The process begins with an understanding of important national priorities and outcomes, which are then translated into broadly defined goals and intended results for the Agency. These become the Agency’s strategic goals and strategic objectives. Outcomes and Outputs related to these strategic goals and objectives are then articulated. Programs are developed to achieve these outcomes and outputs with least resources, and then performance measures and indicators are identified to provide the means to assess the progress made during the budget year and to suggest improvements for the next year’s budget. Flow Chart-2 below explains the relationship between an MOF, Govt. of Mongolia

8

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Agency’s Medium Term Strategic Business Plan and its Annual Program Budget and performance evaluation.

Strategic Business Plan

Next Year’s Budget

Vision, Mission, Strategy and Objectives

Improvement Plan

Strategic Goals

Performance Monitoring and Evaluation through PART

Performance Indicators And Measures

Performance Indicators And Measures

Strategic Outcomes And Outputs

Program Outcomes And Outputs

Activities and Processes

Program Budgets

Inputs (Staff, Funds, Goods, Services, ICT)

Resources for The Budget Year

Flow Chart-2: Integration of Strategic Business Plan, Program Budget and Performance Evaluation 1.4 Performance Monitoring and Evaluation The latest stage of development as regards budgeting is the performance (based) budgeting. Most of the OECD countries (e.g. Australia, Canada, New Zealand, The Netherlands, Sweden, Finland, UK, and USA) have rewritten their financial legislation or legal frameworks to allow for performance (based) financial information systems, including budgets. It is important to understand that performance-based budgeting is not simply the use of program performance information in developing a budget. It does more than just informing the resource allocations of a traditional type of budget. In other words, it is not just "budgeting based on performance." Instead, it is the process and integrated systems framework by which a particular type of budget (such as output budget or program budget) is developed. To design an effective system of performance-based budgeting, it MOF, Govt. of Mongolia

9

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

is vital to understand first what the end product should be, what it should contain, and how efficiently it can be achieved. The pyramidic structure of performance budgeting is illustrated in Flow Chart-3.

Flow Chart-3: Pyramidic Structure of Performance Evaluation Source: John Mercer, an expert on Performance Budget and also regarded as the father of the USA “Government Performance and Results Act (GPRA, 1993)”.

A true Performance Budget is not simply an output budget with some program goals attached. It indicates how the budgeted resources are expected to turn into results, by outlining a general chain of cause and effect. The most effective Performance Budget shows day-to-day activities and financing for each program area, and how these activities generate certain outputs, and what outcomes should then be the result. A Performance-based output Budget differs from an input-based budget in a fundamental way. An input-based budget shows how budgeted fund is spent on salaries, social benefits, goods, office supplies, travel, utilities, equipment, etc. The Performance Budget shows what expenditure will accomplish: establishment of a primary school, city hospital, building roads, houses etc. and reviews a compliance activity. However, every output can also be associated with input costs. 2. Ideal Performance Evaluation Systems MOF, Govt. of Mongolia

10

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

2.1 Characteristics of an Ideal Performance Evaluation System Paul G. Thomas (2005) has mentioned the following properties of an ideal perforance measurement system: It has clearly defined purposes and uses. It focuses on outcomes, not just on inputs and outputs. It employs a limited, cost effective set of measures. It uses measures which are valid, reliable, consistent, comparable and controllable. • It produces information which is relevant, meaningful, balanced and valued by the leaders/ funders of the organisation. • It is integrated with the planning and budgetary processes. • It is embedded in the organisation, is stable and is widely understood and supported. • • • •

In somewhat less abstract terms, the Canadian Comprehensive Auditing Foundation (CCAF) has developed nine principles 'to provide direction for performance reporting in Canada. Box-1 presents these nine principles. The first five principles provide guidance about what governments should report, while the remaining four relate to how governments report. The principles start with as ideals, the ‘ceiling’ that reporting aspires to reach, but over time they become 'standards’, the floor below which reporting should not sink. Taken as a set, the nine principles are meant to provide a framework for performance reporting. 2.2 Various Approaches to Performance Measurement Box-1 — Nine Principles of Better Performance Reporting 1. 2. 3. 4. 5. 6. 7. 8. 9.

Focus on the few critical aspects of performance. Look forward as well as back. Explain key risk considerations. Explain key capacity considerations. Explain other factors critical to performance. Integrate financial and non-financial information. Provide comparative information. Present credible information, fairly interpreted. Disclose the basis for reporting.

Source: Canadian Comprehensive Auditing Foundation, Reporting Principles: Taking Public Performance Reporting to a New Level. Ottawa, 2002.

Performance measurement has become so widespread that it is impossible to know all what is taking place within public sector around the world. There is no single, ‘one best’ approach to performance measurement. A government needs to develop an approach that fits with its constitutional/ legal MOF, Govt. of Mongolia

11

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

requirements, arrangements, political ideology, its size, administrative and institutional capabilities, and, not least important, what it can afford. The general tendency for governments has been to apply a single and uniform approach to all ministries, agencies and programs. This ‘across-the-board’ approach may have apparent virtues of consistency, comparability and fairness, but it is not without problems. It is interesting to know how the system evolved in Canada over the past decade. The federal and provincial governments in Canada developed two broad approaches to performance measurement. In Alberta, Nova Scotia, Ontario and Quebec, governments reported on the performance of the entire government in terms of the impacts of their programs on the society. This ‘social-indicator’ type of approach supports external and political accountability. However, the selection of indicators included in ‘report cards’ to citizens was inherently arbitrary (Thomas 2006). Other provinces and the Government of Canada began their performance measurement efforts by requiring individual departments to prepare business plans and performance reports. The ‘business-line’ approach is more of a managerial tool than something which is normally liked by politicians and the public. However, these two broad approaches viz. the system-wide and the business-line could be pursued simultaneously and complement one another. This has been the evolution of the performance reporting system in the Government of Canada. It began by publishing performance reports on a departmental basis and now more than 80 such documents for departmental and non-departmental bodies are tabled in Parliament on a annual basis. Governments have developed a number of frameworks to identify successful programs. Probably the most common framework involves the so-called ‘three big Es’: economy, efficiency and effectiveness as described below. Economy

Have inputs been acquired at least cost?

Efficiency

Are the inputs (people, money, supplies) being combined to produce the maximum volume of outputs (goods and services)? Are the goals of the organization/ program being met, without undue unintended consequences?

Effectiveness

These elements had been used by both the public and private sector management over the past four or five decades. But, the framework misses another important E i.e. equity, which deals with the distributional impacts of performance. Omitting equity may have an adverse impact on another big E in government i.e. electability. In its earlier program evaluation scheme, the Government of Canada considered a program to be well performing when it was: MOF, Govt. of Mongolia

12

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Relevant Successful Cost-effective

Consistent with government and departmental priorities. Achieves the intended outputs and outcomes. Involves the most efficient means to achieve goals.

This framework deals with the desirability of continuing a program, but does not address the question: Does the organisation have the capability and capacity to deliver the desired results? Organisational report cards represent another type of performance measurement and reporting. Box-2 presents one interpretation of the requirements for such report cards. Box-2: Organisational Report Cards — Criteria for Design (a) (b) (c) (d) (e) (f)

Validity- satisfies legal requirements Comprehensiveness- covers all aspects of a budget Comprehensibility- easy to understand Relevance- appropriate for strategic objectives Reasonableness- can be achieved within time and at reasonable cost Functionality- operational and realistic

Source: William T. Gormley and David L. Weimer, Organisational Report Cards. Cambridge, Mass.: Harvard University Press, 1999. pp. 36-37.

In 1987 the Canadian Comprehensive Audit Foundation (CCAF) published a report on ‘organisational effectiveness’ which mentioned the following attributes of an effective organisation management: Box-3: Attributes of an effective organisation management (a) Relevance (b) Appropriateness (c) Achievement of purpose (d) Acceptance (e) Secondary Impacts (f) Costs and Productivity (g) Responsiveness (h) Financial Results (i) Working Environment (j) Monitoring and Reporting

Several Agencies have since applied this framework. But, there may be conflict in practice among the attributes — e.g., cost efficiency may reduce responsiveness. Besides, assigning weights to these attributes and constructing a weighted overall index is a challenging job. The Office of the Auditor General of Canada (OAG) has recommended another performance framework with six components of performance: MOF, Govt. of Mongolia

13

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

1. 2. 3. 4. 5. 6.

Mission statements Results statements Performance indicators/measures Performance expectations/targets/commitments Strategies/activities Performance accomplishments/achievements

This framework emphasises the desirability of integrating performance planning, budgeting, monitoring and reporting, and also stresses external accountability for results. 2.3 Choice of Particular Performance Evaluation System Regardless of the approach adopted, a sound performance measurement and evaluation system must have three qualities: it must be technically valid, it must be functional, and it must be legitimate. Table-3 presents one generic listing of the ‘ideal’ qualities of such a system. Table-3: Characteristics of an Ideal Performance Measurment System Characteristic

Definition

Clarity

Performance indices should be simple, well defined, and easily understood.

Consistency

Definitions of indicators should be consistent over time and across agencies. One should compare like with like. A Mmanager’s performance should only be measured for areas over which he/she has control. Performance is not independent of the environment within which decisions are made. Covers all aspects of a budget. Consider a limited number of performance indices which provide the biggest pay-off. Performance indicators are relevant to the special needs. Targets are based on realistic expectations.

Comparability Controllability Contingency Comprehensive Bounded Relevance Feasibility

Source: Peter M. Jackson, Measures for Success in the Public Sector.

3. Budget Performance Indicators A budgetary agency has the responsibility to ensure that its programs meet the stated objectives and are cost effective. By measuring the output, outcome and efficiency of a program, performance indicators can guide an agency for optimal resource allocation and communicate the results of public MOF, Govt. of Mongolia

14

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

programs to the stakeholders. It is necessary to review progress regularly toward achieving outputs and outcomes, and continuously improve by planning, executing, evaluating, and adjusting actions to achieve desired results using an integrated performance evaluation system. 3.1 Performance Measures versus Performance Indicators Most government manuals on performance measurement make a distinction between performance measures and performance indicators. Ideally, performance measures report unambiguously on the relationships between program activities and the outputs and outcomes associated with them. However, various economists have pointed out that most relationships between programs and societal impacts are imperfectly understood and subject to change over time. For example, if we want to measure the relationship between safety regulation in the transportation field and the reduction of fatalities and injuries, we must control the impact of other influences within the program environment. Thus, performance measures usually provide only a proxy indication of the performance of a program or policy system. While performance measures might be likened to numbers on a gauge, performance indicators might be compared to alarm bells. Like the bell on the proverbial cat in the famous fable, performance indicators tied to specific programs can warn when unpleasant surprises are on the way. Given the current state of our knowledge about many programs, the distinction between true measures and approximate indicators is somewhat artificial, and subject of public debate rather than of automatic acceptance. There is strong push within the performance measurement movement to produce comparative evidence on performance. The comparisons can be made with regards to other programs within a sector over time. ‘Benchmarking’ and the adoption of the ‘best practices’ of leading organisations is a part of this trend. Here again there is no simple answer as there are various techniques for benchmarking and best practices4.

For practical solutions, it would be wiser for agencies to pursue a ‘smart practice’ rather than a ‘best practice’ approach (Thomas 2006). Smart practice recognises that in statistical terms all organisations cannot be ‘the best’ and an organisation cannot remain best for all times. Organisations deemed to be the best at one point in time See Tarun Das and E. Sandagdorj (2007e and 2007f) on measures for benchmarks and best practices. 4

MOF, Govt. of Mongolia

15

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

by one set of measures often lose that status in the near future. Pushing organisations to strive for ‘the best in class’ status may ignore the practical constraints in an agency, such as lack of able leadership and skilled manpower, inadequate infrastructure, unsupportive organisational culture etc. Developing reliable and consistent information for comparison purposes is a worthy goal, but it has to be balanced by the costs involved and the potential gains. 3.2 Types of Performance Indicators Performance indicators measure what an Agency did in the fiscal year. There are many kinds of performance indicators ranging from quantities or value of goods and services produced in a given period (such as the number of crimes or breaking of traffic rules detected by the police) to more complex indicators such as efficiency and effectiveness of service delivery. Nayyer-Stone (1999) has mentioned four primary types of performance indicators: input, output, outcome and efficiency, which are described in Table-4. In general, input indicators address the amount of resources used in providing a service, whereas output indicators describe the activities undertaken in providing a service. Outcome/ effectiveness indicators are used to evaluate the quality and effectiveness of public services. Outcome/ effectiveness indicators are generally measured in terms of the number of people served and the quality of the service delivered. For example, a relevant outcome indicator for policing is the number of crimes committed per capita. Outcome indicators provide an indication of how effectively community services are provided. However, outcome/ effectiveness indicators are sometimes difficult to use due to the inability to determine a direct correlation between the service provided and the results measured, and these are generally time consuming measurement techniques. A final indicator is efficiency, which relates inputs to output or outcome- for example, the cost per liter of water delivered to a household. When efficiency indicators are used over time, they provide evidence of productivity trends. These indicators and other indicators are explained in more details in the next section on an operational performance evaluation system for Mongolia.

Table-4: Budget Performance Indicators Type of Indicator Input Indicator

MOF, Govt. of Mongolia

Definition Measure of Resources Employed

16

Example • Employees Required • Goods and Services Used • Equipment Needed

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Output Indicator

Effectiveness/ outcome Indicator

Efficiency Indicator

Quantity of Goods and Services Provided The degree to which the intended objectives of the services are being met. Cost per unit of output

Number of projects Number of outputs Number of people served Increase in literacy rate Increase in employment Decrease in crime rate Reduction of poverty. Reduction of maternal and child mortality rate • Cost/ liter of water delivered. • Cost of garbage collected. • • • • • • • •

• Cost per student in schools Source: Adapted from Hatry, Harry P. (1977).

3.3 Characteristics of ideal performance indicators Like any statistical measure, performance indicators must satisfy a number of criteria. In general, an ideal performance indicator should be S.M.A.R.T. (i.e. simple, measurable, achievable, relevant and timely) and C.R.E.A.M. (i.e. clear, relevant, economic, adequate and monitorable). S.M.A.R.T.

C.R.E.A.M.

• • • • •

Simple- easily defined Measurable- easily quantifiable Achievable – can be achieved, not a wish list Relevant- Appropriate for the strategic objectives Timely- can be achieved in time • Clear- Precise, unambiguous, tangible and quantifiable • Realistic- achievable with reasonable cost and in time • Economic - Available at reasonable cost and in time • Adequate- Provides sufficient basis to access performance • Monitorable- Amenable to impartial/ objective evaluation 3.4 Use of Performance Measures

Performance measures can be used in several ways, including the following: Controlling costs – enabling agencies to identify costs which are much higher or lower than average and determine why these differences exist. b) Comparing processes – analyzing performance of services provided with a particular technology, approach or procedure. c) Maintaining standards – monitoring service performance against established performance targets or benchmarks.

a)

MOF, Govt. of Mongolia

17

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

d) Comparing sectors – comparing costs of public sector delivery to costs of private sector delivery of the same type of goods and services.

Box-4 — The Aims of Performance Measurement • To help clarify organisation goals, directions and expectation. • To help organisations learn how to accomplish goals more effectively. • To communicate the priorities of the organisation. • To support strategic/business line planning by linking broad statements of direction to specific operational outputs and outcomes. • To support budgetary planning and resource allocation processes. • To monitor the operation of programs and to make continuous improvements. • To motivate public servants and to restore pride within the public service that it is making a positive contribution. • To enable citizens to make better informed decisions in the use of public programs. • To restore public confidence that they are receiving value for money in public spending. • To assess whether the organisation is achieving its goals. • To strengthen internal administrative and external political accountability. Adapted from Thomas, Paul G. (2006) Performance Measurement, Reporting, Obstacles and Accountability -Recent Trends and Future Directions, Research School of Social Sciences, The Australian National University, Canberra ACT 0200.

4. An Operational Budget Performance Evaluation System For the Government of Mongolia Budget Performance Evaluation System provides an overview of what a budgeting Agency has done and what it has accomplished with the budgeted resources at its command. This involves describing the resources used, the goods and services produced and the resulting benefits to the citizens and the community at large. The evaluation process provides a coherent account of the full range of goods and services provided by an agency. The resulting report helps citizens understand how effectively and efficiently goods and services are being provided by the agency. 4.1 Hierarchy of Activities, Functions and Output Classes Performance evaluation classifies government services in terms of policy areas, functions and output. To take into account variations in size and complexity of agencies, performance evaluation may deal with sub-classified areas of broad functions of an agency. For example, the broad functional area of Law, Order and Public Safety of the Ministry of Justice and Internal Affairs (MOJIA) could be subdivided into the provision of: police services; court services; corrective services; and emergency services. MOF, Govt. of Mongolia

18

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Broad Functional Area: Separate Functions:

Law, Order & Public Safety a) b) c) d)

Police Services Courts Services Corrective Services Emergency Services

Activities within each functional area are grouped together as “output classes” based on a common characteristic or purpose. These groupings describe how an agency fulfills its role. For example, the function of Police Services could be sub-divided into sub-functions such as community policing, crime investigation and traffic control. Functional Area: Sub-Functions:

a) Police Services (i) (ii) (iii)

Community policing Crime Investigation Traffic control

4.2 Types of Budget Performance Evaluation Broadly, there are three types of Budget Performance Evaluation- Budget Compliance, Budget Efficiency and Budget Effectiveness indicators. Budget Compliance examines whether the actual expenditure for a program or output is within the budgeted funds and whether there had been expenditure over-runs. Budget Efficiency means that the actual outputs are at least equal to the budgeted outputs, and Budget Effectiveness implies that the actual outcomes are at least equal to the program outcomes. Types of Performance Evaluation Type of Performance Budgeted Actual Evaluation Compliance Budgeted finance → ← Actual expenditure Efficiency Budgeted outputs → ← Actual outputs Effectiveness Budgeted outcomes → ← Actual outcomes As discussed in section-2 there are four types of performance indicators viz, input indicators, output indicators, efficiency and effectiveness indicators. Other types of indicators such as resource indicators, service indicators, clients’ satisfaction indicators and community indicators have also been suggested in the literature. For practical purpose, input indicators are the same as resource indicators, output and service indicators are the same as efficiency indicators, and outcomes and community indicators are the same as effectiveness indicators. Depending on these performance indicators, we can propose four types of budget performance evaluation as follow: Table-5: Broad Categories of Budget Performance Evaluation MOF, Govt. of Mongolia

19

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

(i) Input/ Resource indicators (compliance evaluation) quantifying levels of expenditure (or unit costs) on the delivery of goods and services, the number of staff employed and the value of assets owned; (ii) Output/ Service indicators (efficiency evaluation) measuring the quantity, quality, accessibility and timeliness of services provided within defined output classes. Such indicators include measures of the outputs of services and their outcomes against an agency’s publicly announced goals, targets or standards. (iii) Clients’ Satisfaction indicators (efficiency evaluation) measuring the personal assessment of services by clients and community stakeholders based on their own expectations. (iv) Outcomes/ Community indicators (effectiveness evaluation) measuring broad social, economic and environmental standards achieved in the community. 4.2.1 Input/ Resource indicators (compliance evaluation) Resource indicators (or inputs) describe the quantity of resources the Agency puts into each output, and where possible, the resources used on each function. These indicators primarily measure inputs in terms of total expenses, staff, goods and services. Unit costs of specific goods and services can also be measured, if possible (e.g. average annual cost per child in a kindergarten school or for educating a student in a primary, secondary and high school etc.).

Expenses should take into account the full cost of services, funded not only from the Government's Consolidated Fund, but also by agencies' own revenues from service charges. Staff data should be presented as average equivalent full-time (EFT) staffing for standardization during each year. Ideally, the value of fixed assets, not fully reflected in recurrent expenditure figures, particularly in the areas of investments on land, infrastructure, buildings, plant and equipment may also be included. But, this may not be feasible at present for Mongolia. We have recommended earlier that the investment cost for public infrastructure such as rails, roads, airports, bridges etc. may be taken as sunk cost. Even capital cost (comprising depreciation and capital charges) may not be included as a part of output cost and may be postponed for an indefinite period until all the assets have been listed and evaluated5.

4.2.2 Output/ Service indicators (efficiency evaluation) 5

For detailed discussion on Output Costing, see Tarun Das and E. Sandagdorj (2007b).

MOF, Govt. of Mongolia

20

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Output indicators measure activities or processes in terms of the outputs they produce. For example, an output indicator of teaching would be the number of students taught in a year. Final outputs are the end products of a chain of activities and are delivered to external clients. Other outputs may be produced for internal clients or as steps towards providing services for external clients. Such outputs are vital to determine the efficiency of supply chains. For example, significant government agency activity may be devoted to producing public policies, developing corporate or strategic plans, providing advice to the Ministers, maintaining computer systems, managing finances, and recruiting, training, managing and appraising staff. These outputs may be costed for output budgeting but may not be delivered to the people. Yet they are vital to the provision of government services. These indirect costs may be apportioned among various outputs either on pro-rata basis or according to some allocation principles. (a) Costing outputs Unit costs are important measures of efficiency, but several factors stand in the way for calculating unit costs for the delivery of public services. In the global budgeting system, many expenses (e.g. corporate overheads) are notionally or pro-rata apportioned among various output classes. Sometimes, cost estimates are based on crude assumptions, such as the cost of indirect overheads could be distributed between different outputs according to the direct cost of their production or their staffing numbers. It is necessary to prepare detailed guidelines for agencies for a uniform costing methodology6. 4.2.3 Clients’ Satisfaction indicators (efficiency evaluation) Client views are critical to evaluating services and it is necessary to collect data on client satisfaction and feedback. Performance evaluation should place importance on clients’ and stakeholders’ perceptions and should report client feedback on the quantity, quality, accessibility and timeliness of services. Other stakeholders might also provide valuable feedback about government service provision. Various techniques such as conducting surveys and stakeholders seminars may be used to assess stakeholders’ perceptions.

6

See Tarun Das and E. Sandagdorj (2007g).

MOF, Govt. of Mongolia

21

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

4.2.4 Outcomes/ Community indicators (effectiveness evaluation) Where possible, performance reports should identify and measure the outcomes of public services i.e. the extent to which service outputs achieved announced goals, targets or standards set by the agency. It is necessary to adopt appropriate performance standards that define the quality of outputs or which set targets for service delivery such as timeliness or accessibility. However, it is often difficult to identify outcomes that are the direct results of outputs produced. Due to the significant influences of external factors on the output, it is a matter of judgments, as to which outcomes, agencies could be held accountable for. For example, the number of students completing vocational training who find employment reflects not only the quality of the training but also the availability of jobs in the market. Thus, it is not fair to judge vocational courses by the outcome of “percent of graduates who find related employment”. Alternatively, vocational courses could be judged by assessment of competencies acquired through such courses, although this outcome might be influenced by other factors such as student motivation. Community indicators measure aspects of community life relevant to the government. These statistics are used for a variety of purposes, such as assessing the need or demand for government intervention, and assessing the impact of government services on the community. Community indicators also influence public opinion on priorities to which the government might respond with policy and additional resources. For instance, statistics on the unemployment and poverty might induce government to introduce new employment generation and poverty alleviation programs. 4.3 Benchmarks7 Benchmarks set the standard or point of comparison against which a measure could be assessed. Often this standard is the average measure achieved by comparable jurisdictions (such as government agencies in all Line Ministries or Aimags) or the best result achieved over a period of time by an agency. Benchmarking draws comparisons between like indicators, for example, between Line Ministries or Aimags regarding the levels of resources allocated for comparable services, the type and quantity of services provided per 1,000 population or per 1,000 sq. kms or the level of client satisfaction for a particular goods or services.

7

For comprehensive discussion on benchmarks setting, consult Tarun Das and E.

Sandagdorj (2007e and 2007f). MOF, Govt. of Mongolia

22

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

5. Budget Performance Evaluation Methodology Budget Performance Evaluation involves five main steps as the following:

1. Summary of Baseline Scenario and Budget Profile 2. Diagnostic Scan and SWOT Analysis

3. Budget Compliance, Efficiency and Effectiveness Evaluation 4. Performance Evaluation 5. Major Findings and Recommendations 5.1 Review of Strategic Plan and Baseline Profile In undertaking a performance evaluation, it is necessary to start with a baseline and initial profile and to identify the key issues on which the performance evaluation is to be focused. Table-6: Scope of Performance Review and the Initial Profile Scope of Performance Review: 1. Strategic Business Plans

Scope of Initial Profile (say for the Budget year 2008):

1. Scope, Governance, Vision, Mission and objectives

2. Scope of review

3. Review steps and key

2. Main functions, programs and activities 3. Structure, staffing and time schedules

milestones 4. Preliminary assessment

4. Program-wise Budgeted funds

5. Focus of review

5. Output costs, benchmarks and performance parameters

This assessment allows an agency to take a detailed look at their current business activities and how they wanted to perform in the budget review year. Agencies will be asked to provide a brief description 8 of their Strategic Business Plan with vision, mission, objectives and goals. They will also be asked to provide summary of their program budgets, which is being reviewed, with budgeted resources, outputs and outcomes. Agencies will be required to provide details of workforce size, their functions and skills, workload volume and contributions to the strategic planning. Annex-1 provides the Template for Review of Strategic Plan and Baseline Profile.

8

Agencies are not supposed to hand over the original copies of the Strategic Business Plan (SBP) and Budget document which they had earlier supplied to the MOF. They are required to make a summary of the SBP and budgeted programs. Detailed Templates are provided in Annex-1 of this report. MOF, Govt. of Mongolia

23

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

5.2 Diagnostic Scan and SWOT Analysis A diagnostic scan of an Agency is necessary before starting a performance review, because actual performance is influenced by constraints on resources, technical manpower and the ICT system. There are basically two types of review- strategic review and operational review. Strategic Review: How well an Agency manages its external environment by delivering relevant and effective services. Operational Review: How well an Agency manages its internal environment by using its resources efficiently and prudently. Both desktop and field scans are required to determine the following aspects: whether best practice techniques were attempted; whether the practice was documented; and whether it was widely applied within the agency. The desktop scan involves checking the existing material on strategic plans and program/ output budgets already submitted by the Agency to the Ministry of Finance, whereas field scans involve conducting surveys and interviewing key stakeholders (clients, community groups, staff and management), to obtain their views on how internal management tools are working in practice. There are 8 possible strategic review areas and 8 operational review areas as indicated in Table-7. Table-7 Strategic Review and Operational Review Areas Strategic review areas

Operational review areas

1. Strategies

9. Work Culture

2. Environment

10. Communications

3. Clients

11. Organisation structure

4. Other stakeholders

12. Reporting Lines

5. Regulation

13. Human resources

6. Policy regime

14. Processes and systems

7. Service delivery

15. Controls

8. Reviews

16. Cost and Asset management

For each of these 16 areas, it is necessary to test whether the agency has applied any typical best practice management techniques. For example, MOF, Govt. of Mongolia

24

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

when examining “Clients”, agencies would be asked whether they have applied any of the following types of management practices e.g. client needs analysis, client segmentation, clients’ satisfaction surveys, grievances and complaints handling and so on. When examining “Controls and Cost and Asset Management”, agencies would be asked if they use the following practices e.g. financial information system, management information system, asset management plan and corporate overhead costs analysis, etc. Table-8A: Typical Best Practices for Strategic Review Strategic Review Areas (number of sub-areas)

Typical Best Management Practices

1. Strategies (2)

Strategic Business Plan, Master Plan

2. Environment (2)

Socio-Economic-Political Environment Analysis, SWOT Analysis.

3. Clients (2)

Clients Needs and Satisfaction Surveys, Grievances and Complaints Handling.

4. Other Stakeholders (2)

Stakeholders Consultation, Focus Groups

5.Regulation (2)

Regulatory Review, Parliamentary Consultative Committee Review

6. Policy (2)

Ministerial Review, Donors Review

7. Service Delivery (2)

Service Charter, Benchmarking

8. Review plan (2)

Performance Agreements, External Audits

Table-8B: Typical Best Practices for Operational Review Operational Review Areas Typical Best Management Practices 9. Work Culture (2)

Code of Conduct, Regular Staff Meetings

10. Communications (2)

Annual Report, Website for Public

11. Organisation Structure (2)

Organisation Chart, Job Descriptions

12. Reporting Lines (2)

Delegation of Powers, Chinese Walls

13. Human Resources (2)

H/R Manual, Training and Development Programs.

14. Process & Systems (2)

Rules and Procedure Manuals, ICT development Plans.

15. Controls (2)

Financial Information System, Management Information System

16. Expenditure and asset management (2)

Asset Management Plan, Agency Overheads Analysis

MOF, Govt. of Mongolia

25

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

SWOT Analysis After diagnostic scan, a SWOT analysis may be undertaken to determine the strengths, weakness, opportunities and threats of the Agency. For example, a SWOT analysis for the Ministry of Finance is illustrated in Table-8. Table-9 SWOT Analysis for the Ministry of Finance

•

Sustained high GDP growth averaging around 8% since 2004. • Strong performance by minerals and services sectors. • Manageable fiscal deficit, with fiscal balance generating surpluses in 2005-2006. • Declining and moderate inflation rates since 2005 • Declining interest rates of the central bank bills • Declining ratios of public debt to GDP • Declining ratios of domestic debt to GDP

Internal environment

Strengths

Weaknesses

• • • • • •

• • • •

External environment

Opportunities



   

Threats

MOF, Govt. of Mongolia

High dependence on mineral revenues Declining trend of the non-mineral revenues as percent of GDP Very high growth rate of money supply which could revive inflationary pressures Low levels of domestic savings Large proportion (nearly 80%) of current expenditure in total public spending leading to low levels of public investment High prevalence of the poverty ratio (32.6 percent in 2006) Economic growth remains vulnerable to unfavourable weather shocks and recurrence of dzuds in future Herd size and composition may put constraints on the growth of agriculture and allied sectors Degradation of environment due to over use of forestry Rise of government expenditures due to elections in 2008 Located between two large and fast growing neighbors (China and Russia) with positive pulls and pushes on the Mongolian economy Surplus in the external current account during 2004-2006 Comfortable foreign exchange reserves (equivalent to 3.4 months of imports at the end of 2006) Declining ratios of external debt to GDP Very low external debt service ratios indicating sustainability of external debt over time  Overall economic growth and government revenue remain vulnerable to terms of trade shocks trigged by sharp declines in international prices of copper and gold in future  Balance of payments remains vulnerable to hardening of global oil prices in future

26

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

MOF, Govt. of Mongolia

27

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

5.3......䈘

Strategic and Operational Performance Evaluation

Annex-2 provides a Template for Strategic & Operational Evaluation. An agency’s performance can be assessed in relation to the 16 performance factors listed in Tables-7A and 7B. Each factor can be given scores on a scale of 0 to 5 by using an approach adapted from the Australian Quality Council: Table-10: Scores for Strategic and Performance Evaluation 0

Approach had not been considered or attempted or does not exist.

1

Some evidence of individual initiative and unsystematic efforts.

2

Approach is planned and introduced in some areas in a limited way.

3

Systematic approach has been implemented in some areas and results are under examination.

4

Approach has been implemented in some areas and results/outcomes have been used to improve the planning and budgeting.

5

Approach used in most of the areas and results/outcomes have been used to improve the planning and budgeting.

Then a “Borda Index” (i.e. sum of all ranks for all factors) can be estimated. This will provide a composite index for rating performance of agencies. There are 32 (=16X2) sub-areas. So a maximum 160 marks can be scored by an Agency. Total score can be expressed as a percentage of 160 marks. Percentage can also be calculated separately for strategic performance and operational performance. Then, total marks for each category can be expressed as a percentage of 80 marks. It is most unlikely that an Agency will be able to score 100% marks. On the basis of percentage of marks, the strategic performance or operational performance, or the combined strategic and operational performance of an Agency could be rated as follows: Table-11: Rating of an Agency on the basis of Performance Scores Range of Performance Scores (in percentage)

Rating of Agency

(a) Effective (EF)

85 – 100

(b) Moderately Effective (ME)

70 – 84

(c) Adequate (AD)

50 – 69

(d) Ineffective (IN)

0 – 49

5.4 Budget Compliance, Efficiency and Effectiveness Evaluation Under Budget Compliance Evaluation, program and sub-program wise budgeted expenditure are compared with the actual expenditure, and the following marks are assigned to each program: MOF, Govt. of Mongolia

28

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Table-12: Marks for Budget Compliance Evaluation 0

If actual expenditure exceeds budgeted expenditure by more than 10%.

1

If actual expenditure exceeds budgeted expenditure by more than 7.5 per cent but less than 10 per cent.

2

If actual expenditure exceeds budgeted expenditure by more than 5 per cent but less than 7.5 per cent.

3

If actual expenditure exceeds budgeted expenditure by more than 2.5 per cent but less than 5 per cent.

4

If actual expenditure exceeds budgeted expenditure by less than 2.5%.

5

If actual expenditure is within the budgeted expenditure.

Under efficiency evaluation, budgeted outputs are compared with actual outputs, and the following marks are assigned to each program. Table-13: Marks for Budget Efficiency Evaluation 0

If actual output falls short of budgeted output by more than 10 per cent.

1

If actual output falls short of budgeted output by more than 7.5 per cent but less than 10 per cent.

2

If actual output falls short of budgeted output by more than 5 per cent but less than 7.5 per cent.

3

If actual output falls short of budgeted output by more than 2.5 per cent but less than 5 per cent.

4

If actual output falls short of budgeted output by less than 2.5 per cent.

5

If actual output is at least equal to the budgeted output.

Under effectiveness evaluation, budgeted outcomes are compared with actual outcomes, and the following marks are assigned to each program. However, one has to wait a number of years before the outcome results are available. Therefore, for the next three years, effectiveness evaluation may not be feasible. Table-14: Marks for Budget Effectiveness Evaluation 0

If actual outcome falls short of budgeted outcome by more than 10%.

1

If actual outcome falls short of budgeted outcome by more than 7.5 per cent but less than 10 per cent.

2

If actual outcome falls short of budgeted outcome by more than 5 per cent but less than 7.5 per cent.

3

If actual outcome falls short of budgeted outcome by more than 2.5 per cent but less than 5 per cent.

4

If actual outcome falls short of budgeted outcome by less than 2.5%.

5

If actual outcome is at least equal to the budgeted outcome.

After assigning marks for all sub-programs, actual marks obtained for all programs of an Agency will be expressed as a percentage of total possible marks. The following MOF, Govt. of Mongolia

29

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das Box indicates the calculation of marks for an agency for compliance and efficiency evaluation.

Total number of sub-programs = N Maximum possible marks for both compliance and efficiency= 10N Total marks obtained for compliance = M1 Total marks obtained for efficiency = M2 Percentage of Marks assigned to the Agency for compliance evaluation = 100 * M1 / 5N = P1 Percentage of Marks assigned to the Agency for efficiency evaluation = 100 * M2 / 5N = P2 Percentage of marks assigned for both compliance and efficiency evaluation = (P1+P2)/2 = 100 * (M1 + M2)/ 10N For an example, Annex-3 provides the templates for the compliance and efficiency evaluation for the program budget of the Ministry of Education, Culture and science (MoECS) for the budget year 2008. However, it may be noted here that MOECS Program budget has confused between outputs and outcomes9. Immediate results of the Programs are called “outcomes”. But, these are, in fact, “outputs” of the program, and not “outcomes”. To reiterate the exact definitions, outputs are the immediate or end results of a project, whereas outcomes are the long term impact of the project on the society after the completion of the project. For example, literacy rate is an outcome, number of students is an output and number of teachers is an input. So, I have replaced outcomes by outputs in the Template. While preparing the next Program Budget, the line ministries and the World Bank ECTAC Team may please keep this distinction in mind.

For definitions of inputs, outputs and outcomes, consult Tarun Das and E. Sandagdorj (2007a). Also consult the Budgets of the Australian Government for any Portfolio. 9

MOF, Govt. of Mongolia

30

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

5.5 Overall Assessment and Score Thus, we have the following three broad evaluations: (1) Strategic Plan and Baseline Profile Evaluation- Annex-1 provides 20 basic questions each carrying 5 marks. Total marks obtained will be expresses as a percentage which will be the score for strategic plan evaluation. (2) Strategic and Operational Performance Evaluation – It has been explained in section 4.3 above and the template is given in Annex-2. Total marks obtained will be expresses as a percentage which will be the score for strategic plan evaluation. (3) Budget Compliance and Effectiveness Evaluation – It has been explained in section 4.4 above and the template is given in Annex-3. Total marks obtained will be expresses as a percentage which will be the score for strategic plan evaluation.

For overall assessment a weight of 30 per cent will be given for strategic plan and baseline evaluation (Annex-1), a weight of 20 percent will be given for strategic and operational performance evaluation (Annex-2) and 50 per cent will be given for budget compliance and effectiveness evaluation.

1-A 1-B 1-C 2-A 2-B 3-A 3-B

Table-15: Weights for Various Types of Evaluation Type of Evaluation Weights Strategic Plan Evaluation Weight: 10% Systems Development Weight: 10% Human Resource Development Weight: 10% Strategic Performance Evaluation Weight: 10% Operational Performance Evaluation Weight: 10% Program Budget Compliance Weight: 25% Program Budget Effectiveness Weight: 25% Total 100%

Translating Performance Scores into Ratings: Finally, the overall performance scores will be converted into qualitative ratings using the scoring bands given in the following table: Table-16: Rating of an Agency on the Basis of Overall Scores Range of Performance Scores Rating of Agency (in percentage)

(e) Effective (EF)

85 – 100

(f) Moderately Effective (ME)

70 – 84

(g) Adequate (AD)

50 – 69

(h) Ineffective (IN)

0 - 49

There will be another category called “Results Not Demonstrated” when an Agency does not have performance measures that have been agreed by MOF either for baselines or for the assessment year.

An Example: MOF, Govt. of Mongolia

31

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

To provide an example, let us assuming that we are evaluating budget performance for three Agencies- A, B and C. The results are given in the following tables: Table-17: An Example of Performance Scores for three Agencies Type of Evaluation 1-A Strategic Plan Evaluation 1-B Systems Development 1-C Human Resource Development 2-A Strategic Performance Evaluation 2-B Operational Performance Evaluation 3-A Program Budget Compliance 3-B Program Budget Effectiveness Total

Weight

Performance Scores (%) Agency Agency Agency -A -B -C

Weighted Scores Agency Agency Agency -A -B -C

0.1 0.1

50 50

60 50

85 75

5 5

6 5

8.5 7.5

0.1 0.1

60

50

70

6

5

7

50

50

85

5

5

8.5

60

70

70

6

7

7

0.25

40

70

70

10

17.5

17.5

0.25 100%

40 47

70 63

70

73.5

10 47

17.5 63

17.5 73.5

0.1

On the basis of these scores, the Agencies would be graded as given in the following table: Table-18: Estimation of Overall Rating for three Agencies Type of Evaluation (1) 1-A Strategic Plan Evaluation 1-B Systems Development 1-C Human Resource Development 2-A Strategic Performance Evaluation 2-B Operational Performance Evaluation 3-A Program Budget Compliance 3-B Program Budget Effectiveness

Weight (2)

Performance Scores (%) Agency Agency Agency -A -B -C (3) (4) (5)

Rating of an Agency Agency Agency Agency -A -B -C (6) (7) (8)

0.1 0.1

50 50

60 50

85 75

AD AD

AD AD

EF ME

0.1 0.1

60

50

70

AD

AD

ME

50

50

85

AD

AD

EF

60

70

70

AD

ME

ME

40

70

70

IN

ME

ME

0.1

0.25

40 70 IN ME ME 70 47 63 73.5 IN AD ME Note: AD stands for Adequate, EF for Effective, IN for Ineffective and ME for Moderately Effective. Total

0.25 100%

Finally, Ratings of the Agencies as regards different performance categories may be put on government or MOF website for information to the general MOF, Govt. of Mongolia

32

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

public and all stakeholders. Actual performance scores (given in columns 3, 4 and 5) might not be published as these scores could suggest a false degree of precision, only the ratings for each category (given in columns 6, 7, 8 in the above table may be made available to the public. However, the detailed methodology and parameters may be published for general information. 5.6 Major Findings and Recommendations Major Performance findings can be drawn on strategic and operational issues relating to the following questions: Strategic

1. Policy Appropriateness:

Did policy address current and emerging needs?

2. Service Effectiveness:

Did agency services meet stated goals?

Operational 3. Cost Efficiency: 4. Operational Prudence:

Were the agency’s unit service costs reasonable? Did the agency control its finances and risks?

On the basis of these findings, recommendations can be made to improve performance in relation to the following broad areas: Strategic issues; and Operational issues. 6. Budget Performance Evaluation Management The Department of Fiscal Policy and Coordination (DFP&C) of the Ministry of Finance may be made nodal department for conducting Budget Performance Evaluation in association with all line Ministries and budgetary agencies. The management process and system along with time schedules are described in Table-18 and Flow Chart-4. To start with DFP&C of the MOF will prepare a detailed guidelines and manual for the Budget Performance Evaluation and would circulate the document among all agencies and line ministries after approval by the Cabinet Committee.

MOF, Govt. of Mongolia

33

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das Table-19: Deadlines for various steps in BPE Activities 1. Department of Fiscal Policy and Co-ordination (DFP&C) of the MOF prepares detailed guidelines and manual on Budget Performance Evaluation (BPE), and circulates the Document (along with Annexes) among line ministries/ Agencies after approval by the Cabinet Committee

DeadlineNo later than 30th April

2. Pilot Line Ministries/ Agencies are selected by MOF for Budget Performance Evaluation (BPE)

5th May

3. Pilot Line Ministries/ Agencies prepare replies and assign marks to the Templates given in Annex-1 to annex-3 and submits the draft reply to the DFP&C

1st July

4. MOF gives comments to the draft Budget Performance Evaluation Report

15th July

5. Pilot Line Ministries / Agencies finalize the reply and resubmit the Budget Performance Evaluation Report to MOF

1st August

6-A. MOF finalizes the Budget Performance Evaluation Report (BPER) and sends copies to the Line Ministry Portfolio Minister and to the the State Central Administrative Body responsible for Finance and Budget.

15th August

6-B. Portfolio Minister also submits the Strategic Business Plan to the State Central Administrative Body responsible for Finance and Budget. 7. The State Central Administrative Body responsible for Finance and Budget shall compile the Budget Performance Evaluation Reports (BPERs) of pilot Line Ministries and submit these along with the consolidated Portfolio Appropriation Estimates to the Government. 8. The State Great Hural shall debate and approve the State Budget. 9. Budget Performance Evaluation Report (BPER) published, publicly released and put on government website. 10. MOF and line ministries/ Agencies conduct multi-stakeholders feedback seminars on the Report. 11. DFP&C of the MOF revise the Manual and Guidelines for Budget Performance Evaluation.

15th August

MOF, Govt. of Mongolia

34

15th September

1st December 15th December 31st January 31st March

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

2. Pilot Line Ministries/ Agencies are selected by MOF for BPE by the 5th May.

1. Department of Fiscal Policy and Coordination (DFP&C) of the MOF prepares detailed guidelines and manual on Budget Performance Evaluation (BPE), and circulates the Document (along with Annexes) among line ministries/ Agencies after approval by the Cabinet Committee no later than the 30th April.

3. Pilot Line Ministries/ Agencies prepare replies and assign marks to the Templates given in Annex-1 to annex-3 and submits the draft reply to the DFP&C (by 1st July 2008).

5. Pilot Line Ministries / Agencies finalize the reply and resubmit the Budget Performance Evaluation Report to MOF no latter than 1st August.

4. MOF gives comments to the draft Budget Performance Evaluation Report no later than the 15th July.

6. MOF finalizes the Budget Performance Evaluation Report (BPER) and sends copies to the Line Ministry Portfolio Minister and to the State Central Administrative Body responsible for Finance and Budget no latter than 15 August. Portfolio Minister also submits the Strategic Business Plan to the State Central Administrative Body responsible for Finance and Budget no later than the 15th of August.

7. The State Central Administrative Body responsible for Finance and Budget shall compile the Budget Performance Evaluation Reports of pilot Line Ministries and submit these along with the consolidated Portfolio Appropriation Estimates to the Government no later than the 15th of September.

9. Budget Performance Evaluation Report (BPER) published, publicly released and put on government website no latter than 15 December.

8. The State Great Hural shall debate and

approve the State Budget no later than the 1st of December.

10. MOF and line ministries/ Agencies conduct multi-stakeholders feedback seminars on the Report by 31 January.

11. DFP&C of the MOF revise the Manual and Guidelines for Budget Performance Evaluation no latter than the 31st March.

Flow Chart-4: Performance Evaluation System and Monitoring

MOF, Govt. of Mongolia

35

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das Selected References Bergin, Jeffrey (2004) Performance Based Budgeting, Performance Management Institute. Canadian Comprehensive Auditing Foundation (2002) Reporting Principles: Taking Public Performance Reporting to a New Leve, Ottawa, 2002. Das, Tarun and E. Sandagdorj (2007a) “Preparation of Strategic Business PlansGeneral Guidelines, Suggestions for Improvement, and Summary of Recommendations”, Final Report, pp.1-74, 30 Sept 2007. Das, Tarun and E. Sandagdorj (2007b) Output Costing and Output BudgetingBasic Concepts and Methodology, pp.1-51, October 2007. Das, Tarun and E. Sandagdorj (2007c) Accrual Accounting and Accrual Budgeting- Basic Concepts and Methodology, pp.1-43, November 2007. Das, Tarun and E. Sandagdorj (2007d) Transition from Cash to Accrual Accounting, pp.1-26, November 2007. Das, Tarun and E. Sandagdorj (2007e) Benchmarks Setting and Best Practices for Output Costing and Output Budgeting- Part-1: Basic Concepts, pp.1-31, Dec 2007. Das, Tarun and E. Sandagdorj (2007f) Benchmarks Setting and Best Practices for Output Costing and Output Budgeting- Part-2: Practical Applications for Mongolia, pp.1-36, Dec 2007. Das, Tarun and E. Sandagdorj (2007g) Terminal Report: Part-1, Major Conclusions and Recommendations, pp.1-70 and Part-2 on Strategic Business Plans, Output costing and Output Budgeting, Accrual Accounting and Accrual Budgeting, and Benchmarks Setting, pp.71-157, ADB Capacity Building Project on Governance Reforms, prepared by Tarun Das for detailed guidelines on Output Costing. Government of Australia, Council on the Cost and Quality of Government (2001) Annual Report 2001, November 2001. Government of Mongolia (2002) Public Sector Management and Finance Act (PSMFA, 27 June 2002). Government of USA (2005) Performance Management Process: Strategic Planning, Budget and Performance Management cycle, General Services Administration (GSA), Office of the Chief financial Officer, 31 January 2005. Hatry, Harry P. (1977) How Effective are your Community Services?, The Urban Institute, Washington, D.C. Kaplan, Robert S. and David P. Norton (1996) The Balanced Scorecard: Translating Strategy into Action, Harvard Business School Press.

MOF, Govt. of Mongolia

36

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das Mercer, John: See Website www.governmentperformance.info

on

GPRA

and

Performance

Management:

Mercer, John (2003) CASCADE Performance Budgeting- A Guide to an Effective System of Integrating Budget and Performance Information and for Linking LongTerm Goals for Day-to-Day Activities, USA, May 2003, www.governmentperformance.info Meyers, Roy T. (1996) Is There a Key to the Normative Budgeting Lock, The World Bank, Washington, D.C. Schick, Allen (1995) Federal Budget: Politics, Policy and Process, Brookings Institution. Steiner, George; Simon and Schuster (1997) Strategic Planning: What Every Manager Must Know. Thomas, Paul G. (2004) Performance Measurement, Reporting and Accountability: Recent Trends and Future Directions, Saskatchewan Institute of Public Policy Paper No 23, February 2004; http://www.uregina.ca/sipp/ Thomas, Paul G. (2005) Performance Management and Management in the Public Sector, Optimum Online — The Journal of Public Sector Management, Vol 35, Issue 2, July 2005. http://www.optimumonline.ca/ Thomas, Paul G. (2006) Performance Measurement, Reporting, Obstacles and Accountability -Recent Trends and Future Directions, Research School of Social Sciences, The Australian National University, Canberra ACT 0200. USA (1993) Government Performance and Results Act (GPRA) of 1993, Office of Management and Budget (OMB). United States of America, Office of Management and Budget (OMB) Homepage: http://www.whitehouse.gov/omb/gils/gil-home.html

MOF, Govt. of Mongolia

37

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Annex-1 Template for Review of Strategic Business Plan (SBP) And Baseline Profile of the Agency Name of the Line Ministry

________________________________

Name of the Budgetary Agency _____________________________ Location _______________________________________________ Part-1: Strategic Business Plan (SBP) (Total Marks=50) 1. Do you have a Strategic Business Plan for your Agency? If Yes, answer the following questions. Answers should be brief and to the point. It may be noted that each question carries 5 marks and answers will be evaluated by the Ministry of Finance. Q1. Which is the period of your latest SBP? Does the SBP contain all the components of a standard SBP as recommended by the MOF viz. Minister’s Foreword, GM’s Statement? Vision, Mission, Values, Priorities, Clients, Strategic Goals and Objectives, SWOT Analysis, Strategic Outcomes. Outputs, Processes and Activities, Medium Term Budget estimate, Organisational Structure, System for Monitoring and Evaluation? If some components are missing, indicate those components and give reasons for not developing those components. Q2. Does any of the Millennium Development Goals (MDGs) concern your Agency? If yes, indicate the MDGs which relate to your Agency. Have you considered these MDGs and suggested measures to achieve those goals in your Strategic Business Plan? If yes, give details. Q3. Does your Agency have a Master Plan? If yes, for which period? Have you linked SBP to the Master Plan? Give a brief account of these linkages. Q4. Indicate briefly Vision, Mission, Values, Priorities, Strategic Goals and Objectives. Q5. Reproduce the SWOT Table from the Agency’s SBP, if you have one. Q6. Who are your clients? Is the Client base growing, shrinking or stable? Do they need differentiated products, services, locations and quality? Are there significant groups of potential clients who are not currently being reached? If yes, why? How do you add value for clients and how can you serve them better? Q7. Provide a current organizational chart of your agency? Q8. Provide a list of the broad output groups indicated in your SBP. If the total number exceeds 25, list the most important 25 outputs or output groups.

MOF, Govt. of Mongolia

38

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das Q9. How well do your current information and communications technology (ICT) systems assist in meeting your mission and long term goals? How is your Agency developing and strengthening the ICT and the skills and capabilities of the workforce to meet these needs? Are there any critical management issues which also need to be addressed? Q10. List and rank the highest priority strategic initiatives and operational priorities required for the success of your Agency. Identify any new proposals, anticipated outputs and link them to your strategic plans. Part-2: Systems Evaluation (Total Marks=25) As in Part-1 each question carries 5 marks and replies will be evaluated by the MOF. Q11. Does your Agency have a system for continual monitoring and annual evaluation of performance parameters recognized in your SBP and Annual Budget? Are the annual Performance Evaluation Reports used consistently to justify funding requests, management actions and legislative proposals? If yes, describe the evaluation system and illustrate your reply on the basis of latest budget. Q12. Do your senior agency managers meet at least quarterly to examine reports on financial and performance information for all outputs/ programs of the Agency? (Evaluation: 5 marks if meetings held at least once in a quarter, 3 marks if meetings held at least once in six months, 2 marks if meeting held at least once in a year, 0 marks if no such review meeting is held in a year). Q13. Does your Agency have well designed plans to improve program performance and efficiency each year? If yes, provide evidence. Q14. Does the latest Annual Budget and performance documents incorporate performance measures identified in the SBP and Master Plan? If yes, provide evidence. Q15. Does your Agency have a proper costing methodology? Does the methodology report the full cost of all outputs accurately in the budget and performance documents? If yes, describe the methodology. Part-3: Human Resource Development (Total Marks=25) As in Part-1 and Part-2, each question carries 5 marks and replies will be evaluated by the MOF. Q16. Do you have Performance Evaluation Reports at least for the middle level and senior level officers? Are these Evaluation Reports used for placements, promotions and training for officers? Are these Reports also used to direct program improvements and hold managers accountable for those improvements? Provide evidence for your answer.

MOF, Govt. of Mongolia

39

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das Q17. Has your Agency implemented a comprehensive Human Capital Development and Utilization Plan that is fully integrated with the agency’s overall strategic plan and annual performance goals? If yes, provide a brief account of the plan. Q18. Has your Agency analyzed the existing organizational structure from service and cost perspectives and is implementing a plan to effectively develop, train, deploy, restructure, recruit and retain employees? If yes, provide a brief account of the plan. Q19. Does your Agency used competitive sourcing, E-Gov solutions, as necessary; and has processes to address future changes in business needs? Q20. Does your Agency have succession strategies, including structured leadership and talent pool development programs, and is able to close leadership competency gaps? Summary of total marks and grading: Questions

Marks for Part-1 M11 M12 M13 M14 M15 M16 M17 M18 M19 M110 Sum of Marks (M1) P1 = 100 * M1/ 50

Questions

Marks for Part-2 M21 M22 M23 M24 M25

Q1 Q11 Q2 Q12 Q3 Q13 Q4 Q14 Q5 Q15 Q6 Q7 Q8 Q9 Q10 Total Marks obtained (M) Express M as % of maximum marks Grade your G1 agency10 Overall Marks for Parts 1, 2 and 3 Express M as % of maximum marks Grade your agency

10

Questions Q16 Q17 Q18 Q19 Q20

Marks for Part-3 M31 M32 M33 M34 M35

Sum of Marks (M2) P2 = 100 * M2/ 25

Sum of Marks (M3) P3 = 100 * M3/ 25

G2

G3 M = M1 + M2 + M3 P = 100 * M / 100 = M G

Grades are determined as per the following Table: Rating of AgencyRange of Performance Scores (in percentage)Effective (EF)85 – 100Moderately Effective (ME)70 – 84Adequate (AD)50 – 69Ineffective (IN)0 - 49

MOF, Govt. of Mongolia

40

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das Annex-2 Template for Strategic and Operational Review On the basis of factual information, rank the following activities of your Agency on a scale of 0 to 5. The rankings must be based on the following score card. 0

Approach had not been considered or attempted or does not exist.

1

Some evidence of individual initiative and unsystematic efforts.

2

Approach is planned and introduced in some areas in a limited way.

3

Systematic approach has been implemented in some areas and results are under examination.

4

Approach has been implemented in some areas and results/outcomes have been used to improve the planning and budgeting.

5

Approach used in most of the areas and results/outcomes have been used to improve the planning and budgeting.

A. Strategic Review Areas- Best practices 1. Strategies and reviews

2. Environment

3. Clients

4. Other stakeholders

5. Regulation

6. Policy regime

MOF, Govt. of Mongolia

Ranking (Marks)

1.1 Do you have a Strategic Business Plan? 1.2 Do you have a Master Plan of your Agency? 2.1 Have you conducted recently any socioeconomic and political environment analysis for your Agency? 2.2 Does your Strategic Business Plan contain a SWOT analysis? 3.1 Have you conducted any clients’ needs assessment and satisfaction surveys? 3.2 Do you have a permanent and independent unit to deal with grievances and complaints of your clients? 4.1 Do you conduct multi stake holders' consultations/ workshops regularly? 4.2 Have your Agency formed focus groups for the services of your Agency? 5.1 Has there been any regulatory review of the activities/ functions/ programs of your Agency? 5.2 Has there been any review of the activities/ functions/ programs of your Agency by any Committee of the Parliament? 6.1 Is there any Ministerial Statement on the vision, mission, objectives and scope of functions of your agency? 6.2 Have the donors made any review of functions of your Agency?

41

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

7. Service delivery

7.1 Have you published any Service Charter indicating rights of your clients? 7.2 Have you conducted any benchmarking analysis for the cost and quality of the goods and services supplied by your Agency? 8. Review plan 8.1 Have you signed performance agreements with the MOF? 8.2 Have the programs been audited by to audit authority? Grading of Agency for Strategic Performance Evaluation Total Marks obtained (M1) Express M1 as % of maximum possible marks (i.e. 80) Grade your agency11

Sum of Marks = 100 * M1/ 80 G1

B. Operational Review Areas- Best practices 9. Work culture

10. Communication

11. Organization structure

12. Reporting lines

13. Human resources

Ranking (Marks)

9.1 Is their written code of conduct for the staff? 9.2 Do the senior officers hold review meetings regularly with subordinate staff on works plan? 10.1 Does the Agency produce an Annual Report every year? 10.2 Does the Agency have a Website for public information? 11.1 Does the Agency have an updated organization chart? 11.2 Does the Agency have written work charts and allocation of works for all staff members? 12.1 Does the Agency have written delegation of powers? 12.2 Does the Agency follow the principle of Chinese Wall12? 13.1 Does the Agency have an updated Manual on human resource development and utilization? 13.2 Does the Agency have permanent training program for skill development?

11

Grades are determined as per the following Table: Rating of AgencyRange of Performance Scores (in percentage)Effective (EF)85 – 100Moderately Effective (ME)70 – 84Adequate (AD)50 – 69Ineffective (IN)0 - 49

12

MOF, Govt. of Mongolia

42

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

14. Process and systems

14.1 Does the Agency have an updated Manual on general rules and procedures for the Agency? 14.2 Does the Agency have a definite plan for ICT upgradation? 15. Controls and 15.1 Does the Agency have computerized metrics government finance information system for accounting and auditing? 15.2 Does the Agency have general management information system (MIS)? 16. Expenditure and 16.1 Does the Agency have a definite plan asset management and program for asset management? 16.2 Does the Agency conduct regular analysis for management of agency overheads? Grading of Agency for Operational Performance Evaluation Total Marks obtained (M2) Express M2 as % of maximum possible marks (i.e. 80) Grade your agency (as given in footnote 11)

Sum of Marks = 100 * M2/ 80 G2

Overall grading: Total Marks obtained (M) Express M as % of maximum possible marks (i.e. 160) Grade your agency (as given in footnote 11)

M= M1 + M2 = 100 * M/ 160 G2

17. If the grade is ineffective, analyze reasons for that and suggest measures for improvement?

MOF, Govt. of Mongolia

43

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das Annex-3 Template for Compliance and Efficiency Evaluation Here we present an example from the Program Budget for the Ministry of Education, Culture and Science for the budget year 2008. However, it may be noted here that MOECS Program budget has confused between outputs and outcomes13. Immediate results of the Programs are called “outcomes”. But, these are, in fact, “outputs” of the program, and not “outcomes”. To reiterate the exact definitions, outputs are the immediate or end results of a project, whereas outcomes are the long term impact of the project on the society after the completion of the project. For example, literacy rate is an outcome, number of students is an output and number of teachers is an input. So, I have replaced outcomes by outputs in this Template. While preparing the next Program Budget, the line ministries and the World Bank ECTAC Team may please keep this distinction in mind. Program/ Output

Budgeted

Actual Performance

Pre-school education

Total budget: 69.6 billion. MNT

1.Services for end-users

Total budget: 64.3 billion. MNT Budget share of education: 92.4%

Output 1.1 Kindergarten service

Output 1.2 24-hour kindergarten services

Output 1.3 Preschool education alternative services

2. Policy renovation activities (strengthening institutions )

Marks

Pre-school

Total budget: 61.2 billion MNT % within the section: 95.0% • Number of children to be covered : 96901 Total budget: 1.0 billion MNT % within the section: 1.0% • Number of children to be covered: 1239 Total budget: 2.2 billion MNT % within the section: 3.0% •

Number of children to be covered: 37682 Total budget: 5.13 billion. MNT Budget share of Pre-school education: 7.4%

For definitions of inputs, outputs and outcomes, consult Tarun Das and E. Sandagdorj (2007a). 13

MOF, Govt. of Mongolia

44

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Program/ Output

Budgeted

Output 2.1 Preschool human recourse capacity development

Actual Performance

Total budget : 30 million MNT % withinthe section: 0.3%

• Output 2.2 Improve coverage and learning environment 3. Administration and Planning

Output 3.1 Preschool education policy, planning and coordination Output 3.2 Provide professional methodological guidence

Number of teachers to be covered by refresh training: 223

Total budget: 5.1 billion MNT % within the section: 99% What is performance parameter? Total budget: 156.6 million MNT Budget share of Pre-school education: 0.2% Total budget : 81.4 mil.MNT. % within the section: 47.0% Total budget: 10.6 million MNT % within the section: 11.8%

Output 3.3 Information, monitoring and evaluation

Total budget :64.6 mil MNT % within the section: 41.2%

a.

Kindergarten service

Total budget : 61.2 billion MNT

b.

24-hour kindergarten

Total budget: 1.0 billion MNT

c.

Alternative preschool education services

Total budget:

d.

Expand kindergarten capacity (increase no. of bed/chairs)

Total budget: 5.1 billion MNT

2.2 billion MNT

a. Indicators for Kindergarten Service Quantity

Year 2008

(i) No. Of kindergartens

737

(ii) No. Of classes

3311

(iii)

No. children (iv) Expenditure

Number

Of

96901 BE (Bln. MNT)

• Wage expenditures

36.5

• Social insurance fee

7.9

• Other var.and fixed exp

19.6

• Food expenditure

8.2

MOF, Govt. of Mongolia

Marks

45

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

• One time allowance Total

MOF, Govt. of Mongolia

0.4 72.6

46

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Program/ Output

Budgeted

b. Indicators for 24-hour kindergarten Service (i) No. of kindergartens

Quantitaves in number / expence in million MNT 6

(ii) Number of classes (iii)

Number of

Actual Performance

Marks

45 1239

children

(iv) Wage expenses

674.8

(v) Social insurance fee

146.1

(vi) Other variable costs

38.9

(vii)

97.9

Food expense

Total

957.7

c. Alternative Per-School Services c-1. Indicators for Shift-classes

Quantitaves in number / expence in million MNT 14676

(i)

Number of children

(ii)

Number classroom days

(iii)

Children/day planning

616.3

(iv)

Wage expenses

436.1

(v)

Social insurance fee

115.1

(vi)

Other variable costs

53.7

42

(vii) Food expense

256.3

(viii) Total expense

861.3

c-2. Indicators for Mobile classes (i) Number of children

Quantitaves in number expence in million MNT 23006 24

(ii)

Number classroom days

(iii)

Children/day planning

483.1

(iv)

Wage expenses

1375.9

(v)

Social insurance fee

297.8

(vi)

Other variable costs

95.8

(vii) Food expense

457.2

(viii) Total expense

2226.7

MOF, Govt. of Mongolia

/

47

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

General Education /GE / 1. Services for end-users

Output 1.1 Primary, basic and high school education services Output 1.2 Dormitory services

Total budget : 235.7 billion MNT Variable exp: 208.9 billion MNT Investment: 26.8 billion MNT Total budget : 189.3 billion MNT % of GE budget : 80.3% Total budget : 172.1 billion MNT % within the section: 91.0% • Number of children to be covered : 548005 Total budget : 11.7 billion MNT % within the section: 6.1%

• Program/ Output Output 1.3 Night and out-of-class education services

Output 1.4 Primary and basic education services for children with disabilities

Output 1.5 Support services for students from vulnerable groups

Number of children to be covered: 46864

Budgeted

Actual Performance

Marks

Total budget : 0.5 billion MNT % within the section: 0.3% •

Number of students to enroll in night classes : 1553

• Number of students to enroll in outside classroom program: 1936 Total budget : 1.6 billion MNT % within the section: 0.8%

•

Number of students to enroll in specialized schools : 1431

Total budget : 1.1 billion MNT % within the section: 0.6% • 8750 children be provided by 4 textbooks • 66590 children be provided by school supplies

Output 1.6 Informal Education Serices

2. Policy renovation activities

(strengthening institutions )

Total budget : 2.3 billion MNT % within the section: 1.2% •

11690 students be covered by consistent programs

•

Literacy education services be delivered to 342 soums

•

Education channel will conduct training for all

Total budget : 29.9 billion MNT % of GE budget: 12.7%

Output 2.1 Strengthening human resource capacity of GE

Total budget : 203.2 million MNT

Output 2.2 GE 12-year school system create learning environment

Total budget : 3 billion MNT % within the section: 10.1%

MOF, Govt. of Mongolia

% within the section: 0.7%

•

Number of teachers to be covered by refresh training: 1682

48

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Output 2.3 Improve GE learning environment

Total budget : 4.6 billion MNT % within the section: 15.4%

Capital repair

Text boo supply services

Total budget : 19.2 billion MNT % within the section: 64.2% Total budget : 0.96 billion MNT % within the section: 3.2%

Output 2.6 English language advanced training

Total budget : 1.9 billion MNT % within the section: 6,4%

3. Admiinistration and Planning

Total budget : 16.7 billion MNT % of GE budget: 7.0%

Output 2.4 Expand GE school coverage Output 2.5

Program/ Output

Budgeted

Actual Performance

Output 3.1 GE Policy, Planning and Coordination Output 3.2 Provide GE institutions with methodological guidence Output 3.3 GE information, monitoring and evaluation Output 3.4

Total budget : 0.44 billion MNT

Marks

% within the section: 2.6%

Total budget : 3.1 billion MNT % within the section: 18.6% Total budget : 0.76 billion MNT % within the section: 4.6% Total budget : billion MNT % within the section: 74.2%

School lunch

I Primary, basic and general education services :

Indicators (i) Number of schools

(ii)

Number of children

Expenditure

Schools under local administration 746 548000 Million MNT

(i)

Wage costs

113,721.8

(ii)

Social insurance fee

24,620.7

(iii)

Other variable expenses

7,720.6

Classroom and practice expenses to be charged from city’s schools’ tuition fee for their advanced training programs (v) Fixed costs

1,121.9

(iv)

(vi)

Olympics participation

20,829.4 423.0

funds

(vii) One time allowance

MOF, Govt. of Mongolia

1,856.6

49

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Total budget

172,117.4

Dormitory services : Indicators

(i)

Quantities 500

Number of dormitories

46864

(ii)

Number of children in dormitories Expenditure

Million MNT 4965.0

(i)

Other expenses for goods and services (ii) Food

6746.1

Total

11711.1

Program/ Output Higher education

Budgeted

Actual Performance

Marks

Total budget : 63.1 billion MNT Variable cost: 60.6 billion MNT investment: 2.5 billion MNT

1. Services for end-users

Output 1.1 Higher education services (bachelor, master, doctorate degree) Output 1.2 Student scholarship services

Total budget: 60.3 billion MNT % within higher education section : 96.0 % % within the GBD’s total budget : 13.7% Total budget: 39.5 billion MNT % within the section : 65.5% Total budget : 20.8 billion MNT % within the section: 34.5 %

• 2. Policy renovation activities (strengthening institutions ) Output 2.1 Strengthen human resource capacity Output 2.2 Strengtheining higher education learning environment and material basis 3. Administration and Planning

Output 3.1 Higher education policy and planning

MOF, Govt. of Mongolia

6200 students will be covered

Total budget: 2.53 billion MNT % within higher education section: 4.0% Total budget: 30 million MNT % within the section: 1.2% Total budget : 2.5 billion MNT % within the section: 98.8% Total budget: 248.9 million MNT % within higher education section: 0.4% Total budget: 144.3 million MNT % within the section: 57.7%

50

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Output 3.2 Higher education policy and coordination Output 3.3 Higher education information, monitoring and evaluation

Art and Culture 1. Services for end-users Output 1.1 Historic and art immovable heritages and museum services Output 1.2 Services of stage and screen art Output 1.3 Library services Output 1.4 Soum Art center services

MOF, Govt. of Mongolia

Total budget : 64.3 million MNT % within the section: 25.4% Total budget: 40.4 million MNT % within the section: 16.9%

Total budget : 33.8 billion MNT Variable costs : 23.8 billion MNT Investment: 10,0 billion MNT Total budget: 23.4 billion MNT % within the art and culturral section budget share: 68,9% Total budget: 3.6 billion MNT % within the section: 15.4% Total budget: 8.9 billion MNT % within the section: 38.2% Total budget: 2.7 billion MNT % within the section: 11.6% Total budget: 7.8 billion MNT % within the section: 33.5%

51

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Program/ Output From this : Output 1.5 Art and cultural events 2Policy renovation activities Output 2.1 Strengthen human resource capacity of Art and culture institutions Output 2.2 Improve Art and culture institutions’s condition Output 2.3 Strengthen material basis of Art and culture institutions Output 2.4 Expand the Art and culture service coverage

Budgeted

Actual Performance

Total budget : 0.3 billion MNT % within the section õóâü: 1.3% Total budget: 10.13 billion MNT % within the art and culturral section budget share: 29.9% Total budget: 0.13 billion MNT % within the section: 1.5% Total budget: 1,6 billion MNT Á% within the section: 25.8% Total budget: 1.4 billion MNT % within the section: 16.4% Total budget: 7.0 billion MNT % within the section: 56.3%

3. Administration and Plannng

Total budget: 0.42 billion MNT % within the art and culturral section budget share: 1.2%

Output 3.1 Art and Culture development policy and planning Output 3.2 Art and culture policy implementation coordinaation Output 3.3 Art and culture section information, monitoring, evaluation

Total budget: 0.18 billion MNT % within the section: 33.3%

Organization

(i)

Ensenmble of National Folk and dance

Total budget: 0.23 billion MNT % within the section: 63.8% Total budget: 0.01 billion MNT % within the section: 2.9%

Number of new vacancies/ job postions 40

(ii)

State philharmonics

18

(iii)

State cartoon theatre

8

Total Aimag theatre and ensembles

66 Number of increased position

(i)

Bulgan

3

(ii)

Umnugobi

2

(iii)

Sukhbaatar

4

(iv)

Darkhan

4

(v)

Mandal soum theatre, Selenge aimag /newly established

MOF, Govt. of Mongolia

Marks

42

52

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

/ Total

55

Annex-4: Methodology for calculation of composite score and grading the Line Ministry/ Budgetary Agency For overall assessment, a weight of 30 per cent will be given for strategic plan and baseline evaluation (Annex-1), a weight of 20 percent will be given for strategic and operational performance evaluation (Annex-2) and 50 per cent will be given for budget compliance and effectiveness evaluation.

Type of Evaluation 1-A 1-B 1-C 2-A 2-B 3-A 3-B

Strategic Plan Evaluation Systems Development Human Resource Development Strategic Performance Evaluation Operational Performance Evaluation Program Budget Compliance Program Budget Effectiveness Total

Weights 10%

Marks obtained M1

% of Marks Weighted Grades in total percentage marks in the category 10% * P1 G1 P1

10%

M2

P2

10% * P2

G2

10%

M3

P3

10% * P3

G3

10%

M4

P4

10% * P4

G4

10%

M5

P5

10% * P5

G5

M6

P6

25% * P6

G6

M7

P7

25% * P7

G7

M

P

P

G

25% 25% 100%

Translating Performance Scores into Ratings: Finally, the overall performance scores will be converted into qualitative ratings using the scoring bands given in the following table: Range of Performance Scores (in percentage)

Rating of Agency (i) Effective (EF)

85 – 100

(j) Moderately Effective (ME)

70 – 84

(k) Adequate (AD)

50 – 69

(l) Ineffective (IN)

0 - 49

MOF, Govt. of Mongolia

53

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das There will be another category called “Results Not Demonstrated” when an Agency does not have performance measures that have been agreed by MOF either for baselines or for the assessment year.

Glossary Definition of Terms Activities are the tasks undertaken by the staff to transform inputs into outputs. Annual Action Plan describes year-wise tactical actions to carry out a Strategy to achieve a Long Term Outcome Goal. Annual Performance Report summarizes an Agency’s performance progress relative to its stated goals for a given year. Annual Target sets a target level of performance for a specified fiscal year that is expressed as a tangible and measurable objective, or quantifiable standard, value or rate against which actual achievement can be compared. Benchmark Analysis answers the question “how is X performing in compared to Y. A benchmark is a quantifiable standard or norm against which actual performance can be compared. Budget Year is the Fiscal Year for which budget estimates are provided. Business Action Plan is a document that summarizes the tactical actions required for the implementation of the Strategic Business Plan. Core Values are the ideals that guide behavior for all interactions internal and external to the organization. Current Year is the Fiscal Year immediately preceding the Budget Year. Goals are long term wide-spread results of public programs and policies, like the achievement of the Millennium Development Goals by 2015 Inputs are resources (Personnel, financial, goods and services) are the basic requirements for any output and strategic planning. Indicators are broad-based metrics which show whether outcomes are trending in the desired direction. Long Term is a period of 5 years or more.

MOF, Govt. of Mongolia

54

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Measures help determine the impact or influence of Department activities on outcomes. They have targets associated with them and can be quantitative or qualitative in nature. Medium Term is a duration of 2-4 years or the Budget Year + 3 years. Mission is the stated purpose of the organization. Outcomes are the medium term impact of public programs and policies on the economy and user groups. Outputs are deliverables (products and services produced) from inputs. Outputs are the immediate or end results of activities. Performance Goal sets a target level of performance for a multiple year period that is expressed as a tangible, measurable objective, against which actual achievement can be compared. It is expressed as a quantitative standard, value or rate. These goals must be included in the Program Budget. Performance Measure is a basis or standard of comparison used to monitor the success in achieving the Performance Goal. A Performance measure is frequently referred to as an Indicator. PERT Chart stands for Program Evaluation Review Technique. It is project management tool used to plan, schedule and co-ordinate plan. Sometimes, it is referred to as the Critical Path Method (CPM). Policy is a general statement designed to guide associates actions in recurring situations. Previous Year/ Prior Year is the Fiscal Year immediately preceding the Current year or the last completed Fiscal Year. Procedure is a sequence of steps describing how to carry out an activity. Program Assessment Rating Tool (PART) is an assessment tool that rates agency programs on effectiveness and ability to achieve desired results. Short Term is a period of less than one year. Strategy is a broad course of action or methodology designed to achieve a long term outcome goals. Strategic Assessment is the process of gathering and analyzing data that impacts an organization’s success in achieving its long-term goals. Strategic Goals are long-term outcomes that help the organization achieve its mission. Strategic Planning is the process of developing and analyzing an organization’s mission, long term goals, performance goals, business actions and allocating resources. MOF, Govt. of Mongolia

55

Glocoms Inc. (USA)

Budget Performance Evaluation Methodology by Tarun Das

Strategic Objectives are broad based outcome statements for a group of value chains. SWOT Analysis is a review of strength-weakness-opportunities-threats impacting an organization. It is critical to the creation of a strategic plan in that it assesses the internal an external environment, existing performance data and forecasts relative to the thresholds an organization has established for itself. Tactical Action Plan is the process of making detailed decisions about what to do, who will do it, when it will be accomplished and how. Tactical Plan is a document stating how an organization will implement its strategic business plan. Value Chains are programs grouped by a common purpose. These help employees understand their shared objectives by identifying activities that cross organizational boundaries in the achievement of the Department’s strategic goals. Value Chain Outcomes describe the intended result from carrying out activities for a group of programs with a common purpose. Vision is a compelling picture of the organization that all can strive for in the future. Although it may never be achieved, it is intrinsically motivating and drives the organization toward a common purpose.

MOF, Govt. of Mongolia

56

Glocoms Inc. (USA)