Handout All

Definition and Purpose ..................A technique that can be used to evaluate government projects and programs. It encompasses an appraisal of a policy based on the costs and benefits of the project. A CBA should be performed for each investment alternative to enable the evaluation and comparison of alternatives. Measurement in the Cost Benefit Analysis 1. Net Present Value The net present value of a project is the sum of the present value of each cost and other effects of a project. A positive NPV indicates that benefits outweigh the costs of a project. 2. Benefit Cost Ratio The ratio obtained by dividing of the sum of the present value of the benefits by the sum of the present value of the costs. A BCR greater than 1 indicates that the benefits outweigh the cost of a project. 3. Net Present Value per unit of Investment the net present value divided by the present value of a project’s capital cost. This ratio provides an indication of the effectiveness of a given level of investment. Process 1. Determine/Define Objectives The CBA should include a problem definition, pertinent background information and a list of investment objectives that identify how the system will improve the work process and support the mission. 2. Document Current Process The current process should be thoroughly documented and address the areas of customer satisfaction and system architecture. 3. Estimate Future Requirements Two items to consider are lifecycle time and lifecycle demands. 4. Collect Cost Data Data can be collected, from several sources, to estimate the costs of each investment alternative. 5. Choose at Least Three Alternatives A CBA should present at least three viable alternatives. “Do nothing” or “Continue current operations” should not be considered as an alternative. Each viable technical approach should be included as an alternative. 6. Document CBA Assumptions It is important to document all assumptions and, if possible, justify them on the basis of prior experiences or actual data. 7. Estimate Costs Many factors should be considered during the process of estimating costs for alternatives. Full lifecycle costs for each competing alternative should be included, and certain factors should be addressed. 8. Estimate Benefits The steps to estimate benefits include: defining benefits, identifying benefits, establishing measurement criteria, classifying benefits, quantifying intangible benefits, and estimating tangible benefits.

9. Discount Costs and Benefits After costs and benefits for each system lifecycle year have been identified, they are converted into a common measurement unit by discounting future dollar values and transforming future benefits and costs to their “present value.” 10. Evaluate Alternatives Many benefits cannot be quantified in dollar terms. As a result, evaluating alternatives cannot always be done using present values, but valid evaluations can be made using a combination of dollar values and quantified relative values (values that are numeric, but do not represent dollar values). 11. Perform Sensitivity Analysis Sensitivity analysis tests the sensitivity of input parameters and the reliability of the CBA result. Sensitivity analysis should assure reviewers the CBA provides a sound basis for decisions. 12. Compare Investments Even if the CBA shows that benefits will outweigh costs, using Payback Period and Return on Investment (ROI) analysis help demonstrate an investment is a better utilization of funds than other proposed investments.

Cost Effectiveness Analysis (CEA) Cost-effectiveness analysis refers to the consideration of decision alternatives in which both their costs and consequences are taken into account in a systematic way. It is a decision oriented tool, in that it is designed to ascertain which means of attaining particular educational goals are most efficient. For example, there are many alternative approaches for pursuing such goals as raising reading or mathematics achievements. Developed in the military, CEA was first applied to health care in the mid-1960s and was introduced with enthusiasm to clinicians by Weinstein and Stason in 1977.

Basic of Cost Effectiveness Analysis Cost Effectiveness Analysis is a technique for comparing the relative value of various clinical strategies. In its most common form, a new strategy is compared with current practice (the "low-cost alternative") in the calculation of the cost-effectiveness ratio. Cost-effectiveness Analysis, or CEA, is a comparison tool to help evaluate choices. It will not always indicate a clear choice, but it will evaluate options quantitatively based on a defined model. For managers, CEA provides peer-reviewed evidence for decision support. Cost-effectiveness analysis is closely related to cost-benefit analysis in that both represent economic evaluations of alternative resource use and measure costs in the same way. However, cost-benefit analysis is used to address only those types of alternatives where the outcomes can be measured in terms of their monetary values. For example, educational alternatives that are designed to raise productivity and- income, such as vocational education, have outcomes that can be assessed in monetary terms and can be evaluated according to cost-benefit analysis. However, most educational alternatives are

dedicated to improving achievement or some other educational outcome that cannot be easily converted into monetary terms. In these cases, one must limit the comparison of alternatives to those that have similar goals by comparing them through costeffectiveness analysis.

Measuring Cost Effectiveness The basic technique has been to derive results for educational effectiveness of each alternative by using standard evaluation procedures or studies (Rossi and Freeman 1985) and to combine such information with cost data that are derived from the ingredients approach. The ingredients approach was developed to provide a systematic way for evaluators to estimate the costs of social interventions (Levin 1983). It has been applied not only to cost-effectiveness problems, but also to determining the costs of different educational programs for state and local planning (Hartman 1981).

Assessing Effectiveness Before starting the cost analysis, it is necessary to know what the decision problem is, how to measure effectiveness, which alternatives are being considered and what their effects are. If a problem has risen on the policy agenda that requires a response, a careful understanding of the problem is crucial to addressing its solution Once the problem has been formulated, it will be necessary to consider how to assess the effectiveness of alternatives. For this purpose, clear dimensions and measures of effectiveness will be needed. Table I shows examples of effectiveness measures that respond to particular program objectives Program Objective Program Completions Reducing Dropouts Employment of graduates

Measure of Effectiveness Number of Students complete program Number of potential dropouts who graduate Number of graduates placed in appropriate Jobs Test scores in appropriate domains utilizing appropriate test instruments

Student learning .

Table 1.

Given the problem and criteria for assessing the effectiveness of proposed solutions, it is necessary to formulate alternative programs or interventions, The search for such interventions should be as wide-ranging and creative as possible. This procedure sets the stage for the evaluation of effectiveness of the alternatives, a process which is akin to the standard use of evaluation methods. Estimates of effectiveness can be derived from previous evaluations or from tailored evaluations for the present purpose. It is important to emphasize that the evaluation of effectiveness is separable from the evaluation of

costs. Most standard evaluation designs for assessing the effectiveness of an intervention are also suitable for incorporation into cost-effectiveness studies. Cost Estimation The costs of an intervention are defined as the value of the resources that are given up by Society to effect the intervention. These are referred to as the ingredients of the intervention, and it is the social value of those ingredients that constitute its overall cost. At a later stage the distribution of these costs among the decision-making agency and other entities can be assessed. Accordingly, the method sets out systematically to identify and ascertain the value of the ingredients that are required for each alternative that is under consideration. The ingredients approach to cost estimation entails three distinct phases: (a) identification of ingredients; (b) determination of the value or cost of the ingredients and the overall costs of an intervention; and (c) an analysis of the costs in an appropriate decision-oriented framework. The first step is to ascertain which ingredients are required for an intervention . Most educational interventions are labor-intensive, so an initial concern is to account for the number and characteristics of personnel. It is important to stipulate whether personnel are part-time or full-time and the types of skills or qualifications that they need. Beyond this it is necessary to identify the facilities, equipment, materials, and other ingredients or resources which are required for the intervention. Identification of ingredients requires a level of detail that is adequate to ensure that all resources are included and are described adequately to place cost values on them. For this reason, the search for ingredients must be systematic rather than casual. The primary sources for such data are written reports, observations, and interviews. Written reports, usually contain at least a brief history and description of the intervention. Other sources of information must be used to corroborate and supplement data on ingredients from evaluations and descriptive reports. If the intervention is present at a nearby site, it may be possible to visit and gather additional data on ingredients through observation. A third valuable source is that of interviews, where present or former personnel are asked to identify resources from among a number of different classifications. The three principal types of information reports, observations, and interviews-can be used to assure the accuracy of the data by comparing the findings from each source and reconciling differences, the process of triangulation. Once the ingredients have been identified and stipulated, it is necessary to ascertain their costs. In doing this, all ingredients are assumed to have a cost, including donated or volunteer' resources. That is, they have a cost to someone, even if the sponsoring agency did not pay for them in a particular situation. At a later stage the costs will be distributed among the constituencies who paid them, but at this stage the need is to ascertain the total costs of the intervention. Ingredients can be divided into those that are purchased in reasonably competitive markets, and those that are obtained through other types of transactions. In general, the value of an ingredient for costing purposes is its market value. In the case of personnel, market value may be ascertained by determining what the

costs would be for hiring a particular type .of person. Such costs must include not only salary, but also fringe benefits ' and other employment costs that are paid by the employer. Many of the other inputs can also be cost by using their market prices. These include the costs of equipment, materials, utilities, and so on. Clearly the cost of leased facilities can also be ascertained in this way. Although the market prices of some ingredients such as personnel can often be obtained from accounting data for educational enterprises, such data are not reliable sources for ascertaining overall program costs. The accounting systems that are used by schools were designed for ensuring consistent reporting to state agencies rather than for providing accurate and consistent cost data on educational interventions. For example, they omit completely or understate the cost of volunteers and other donated resources. Capital improvements are charged to such budgets and accounts during the year of their purchase, even when the improvements have a life of 20-30 years. Normal cost accounting practices would ascertain the annual costs of such improvements by spreading them over their useful lives through an appropriate method. Thus, data from accounting and budgetary reports must be used selectively and appropriately and cannot be relied upon for all ingredients. There exist a variety of techniques for ascertaining the value of ingredients that are not purchased in competitive markets. For example, the method for ascertaining the value of volunteers and other contributed ingredients is to determine the mark-et value of such resources if they had to be purchased. The value of facilities can be determined by estimating their lease value. The annual value of facilities and equipment can be estimated through a relatively simple approach that takes account of depreciation and interest foregone by the remaining capital investment.

Common Application Evaluating Program Options In the case of health screening, it is often difficult to determine the most cost-effective frequency. Too frequent screening has high cost and possibly limited health benefits, while too infrequent screening has low cost, but poor health outcomes. Determining appropriate screening frequencies is a useful application of cost-effectiveness analysis. The following table taken from an analysis on cervical cancer screening shows that life years are saved at a relatively low cost in the first comparison (screening versus no screening), but at a very high cost in the second comparison (the marginal cost and benefit of decreasing the interval between screenings). Typically, an intervention that costs less than $30,000/life year gained is considered cost-effective medicine. Based on this analysis, cervical cancer screening every four years is a relatively cost-effective benefit to cover. It is certainly more cost-effective than screening every three years.

Life expectancy increase, days

Screen every four years vs No Screening 93.8

Screen every three years vs Screen every four years 1.6

Life expectancy increase, days (discounted 5%) Cost Increases, dollars Cost per Life years gained

9.5

0.2

$ 264 $ 10,101

$ 91 $ 184,528

Table 2. Example Data from an Analysis of Cervical Cancer Screening Frequency

Justifying Program Implementation When building a case to justify the use of funds, strong data is often compelling evidence. Cost-effectiveness analyses can be used to support qualitative arguments for health interventions. The following table examines a depression treatment improvement program. Treatment facilities in the study were offered training for practice leaders and nurses, enhanced educational and assessment resources, and trained psychotherapists for patient follow-ups. Not only was the intervention relatively cost-effective, but it also increased attendance in workers suffering from depression.

Quality-adjusted life increase, years Cost Increase Cost per quality-adjusted life Days of Employment Increase

Quality Improvement Program vs Usual Care 0.0226 $ 485 $ 21,460 20,9

Table 3. Example Data from an Analysis of a Depression Treatment Improvement Program

Utility Analysis Utility analysis is a quantitative method that estimates the dollar value of benefits generated by an intervention based on the improvement it produces in worker productivity. Utility analysis provides managers information they can use to evaluate the financial impact of an intervention, including computing a return on their investment in implementing it. The concept of utility was originally introduced by Brogden (1949) and Brogden and Taylor (1950) and further developed by Cronbach & Gleser (1965). The concept has been researched and extended by Cascio (1982); Schmidt, Hunter, and Pearlman (1982); and Reilly and Smither (1983), among others. It was introduced as a method for evaluating the

organizational benefits of using systematic procedures (e.g., proficiency tests) to improve the selection of personnel but extends naturally to evaluating any intervention that attempts to improve human performance. In completing the analysis, the performer needs to generate the following: • A method for measuring role productivity, • A way to assign monetary value to role productivity, • The distribution of productivity among performers of the role, • The dollar value of a one standard deviation difference in role productivity (SD$), and • A method to measure the intervention's impact on role productivity.

Method 1. Determine the productivity of performers. Tip. Execute our plan for measuring the productivity of current role incumbents. 2. Determine the dollar value of a one standard deviation difference in role productivity (SD$). Tip. Distribute the productivity scores we gather. Confirm the distribution is essentially normal and compute its mean and standard deviation. If the distribution is not normal, use a transformation method (e.g., z-transformation) to normalize it. Apply our method for valuing role productivity. Derive the dollar value of productivity achieved by average performers and the dollar value of a one standard deviation difference in productivity (SD$).

3. Compute the effects on performer productivity associated with the performer's participation in the intervention being evaluated. Tip. Apply our method for measuring the affect of the intervention on productivity. Determine how many standard deviations of change in worker productivity the intervention produces (SD). 4. Compute the dollar value of productivity improvements generated by the intervention. Tip. The dollar value of productivity improvements generated by the intervention is the Intervention's utility (U$). To compute utility, multiply the number of standard deviations of change the intervention produces in worker productivity (SD) and the dollar value of a one standard deviation difference in productivity (SD$) (SD x SD$ = U$).

BALANCED SCORECARD A new approach to strategic management was developed in the early 1990's by Drs. Robert Kaplan (Harvard Business School) and David Norton. They named this system the 'balanced scorecard'. Recognizing some of the weaknesses and vagueness of previous management approaches, the balanced scorecard approach provides a clear prescription as to what companies should measure in order to 'balance' the financial perspective. Kaplan and Norton found that companies are using the scorecard to: •

Clarify and update strategy

•

Communicate strategy throughout the company

•

Align unit and individual goals with strategy

•

Link strategic objectives to long term targets and annual budgets

•

Identify and align strategic initiatives

•

Conduct periodic performance reviews to learn about and improve strategy

Although by the end of 2001 about 36% of global companies are working with the balanced scorecard (according to Bain), much of the information in the commercial sector is proprietary, because it relates to the strategies of specific companies. Public-sector (government) organizations are usually not concerned with proprietary information, but also they do not usually have a mandate (or much funding) to post their management information on web sites. Four fundamental perspectives

Financial perspective How do we perform according to our shareholders? Customer perspective

Vision and Strategy

How do our customer see us?

Business proses perspective What must we excell at?

Learning and growth perspective Can we continue to improve & create value?

Comparison of traditional strategic planning and Balanced scorecard : No. 1 2 3 4

TRADITIONAL STRATEGIC PLANNING Typically 5 years Innovation and major change is not likely to occur Contains a large number of goals, objectives,… Managers tend to focus on short term financial goals

5

No performance reporting mechanism

6

Not transparent

BALANCED SCORECARD Shorter than 3 years Innovation emerges from across functional teams Focus on a 3 or 4 strategic themes Managers have a balanced view of the major perspectives of performance Performance data is widely reported via a distributed software system Transparent

Advantages of applying balanced scorecard : •

A framework to focus on key perspectives

•

Helps align key performance measures with strategy at all levels of organization

•

Provides management with a comprehensive picture of business goals and strategies at all

levels of an organization •

Gives a new way to executives of a company to assess how well their organization is functioning, how to predict future performance, how to align the organization toward new strategies to achieve breakthrough performance

•

Transforms the strategic plan from an attractive but passive document into the marching orders for the organization on a daily basis

•

Enables executives to truly execute their strategies

Disadvantages and problems of using balanced scorecard : •

Lack of time for the decision makers to focus on strategy

•

Confusion between operational efficiency and strategy

•

Difficult in creating well defined metrics and connecting them to deliverables

•

Cascading the objectives down to the staff that can deliver the results

•

Difficult and time consuming to implement a comprehensive balanced scorecard system in a large organization

•

Require sustained top level support and commitment to ramp up and put the system in place

Nine steps to implement balanced scorecard : •

Organizational assessment

•

Identify strategic themes

•

Define perspectives and strategic objectives

•

Develop a strategy map

•

Derive performance metrics

•

Craft and prioritize strategic initiatives

•

Automate and communicate

•

Cascade the balanced scorecard through the organization collect data, evaluate, and revise