Cost Of Quality

IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 8, NO 2, FEBRUARY 1990

315

Software Costs of Quality WILLIAM A. MANDEVILLE,

Abstracf-Our hardware manufacturing cousins have taken advantage of the costs of quality for manufacturing by using it as a lever to get needed prevention resources such as new tools, capital, and training to improve their productivity, quality, and to strategically reduce costs. It is time for the software development community to recognize this powerful quality improvement tool and to take full advantage of it.

INTRODUCTION HERE are very few costs to reduce in the software world. We cannot reduce inventories or the costs of materials, and we have little influence over the cost o f platforms, and labor costs continue to rise. The significant savings lie in the costs of quality and our ability to reduce them (by reducing defects). The biggest cost reductions can be gotten by determining what we are spending now for the costs associated with defects, finding and fixing the root causes of these defects, and constantly measuring and monitoring to determine new cost of quality opportunities to reduce. Everyone agrees with this philosophy (preventing defects reduces costs); the problem is how to identify the problems, get management’s attention, and develop the projected return on investments so that prevention resources can be applied to the roots of the problem. This paper defines the costs of quality for software, presents a methodology for the collection of costs, and shows you how to use the costs of quality to point the way toward quality improvement and significant cost reductions. There is a big difference between quality control or assurance and what this paper decribes as quality improvement. The quality function is usually given the charter to provide assurance that for the company’s product meets the requirements of its users.This is different from quality improvement. Quality improvement is the deliberate, planned approach to preventing defects from occurring. Quality improvement responsibility can reside with the Quality Organization or it can be part of the development organization’s tools or methods group. This paper concentrates on using the costs of quality as part of the improvement process.

T

What are the Costs of Quality for Sofirware? Quality costs are categorized into the costs of conformance to specifications (or requirements, if you prefer) Manuscript received May 4, 1989; revised September 25, 1989. The author is with the Carman Group, P.O. Box 867689, Plano, TX 75086. IEEE Log Number 8932 100.

MEMBER, IEEE

and the costs of noncompliance. The costs of compliance are the costs which are spent to ensure that software is developed and maintained in accordance with its requirements. These costs include the costs to prevent defects from occurring in the first place (prevention costs) and the costs o f finding defects (appraisal). A list of typical prevention costs for software follows: the cost of training the cost of tools the cost of methods the cost of standards, policies, and procedures the cost of consulting the cost of quality planning the cost of team prevention meetings the cost of fast prototyping the cost of quality data gathering, analysis, and use the cost of root cause analysis. As you can, see the costs of prevention are the costs to prevent defects from occurring (The costs of putting yourself into a position of doing things right the first time.) The other cost of compliance is appraisal cost. These costs are associated with the review of the product to ensure that it meets requirements. Appraisal costs are usually associated with the defect removal process. They are insurance costs and include the following activities: the review of requirement specifications the review of design specifications the review of component and module specifications the review, walk through, or inspection of code the testing process, including unit test, product test, system test, and independent verification and validation test Beta test, field trial, or parallel test all product audits (process audits could be considered prevention because they are a method to find potential problems and remove them, before they cause defects). The costs of noncompliance are failure costs. Included are the costs to fix errors in software. Failure costs also include the cost of any rework. Rework can be any review, audit, inspection, or test which is conducted more than once. The first review is considered appraisal cost, any re-review (rework) is considered a failure cost (cost of noncompliance). A partial list of failure costs follows: the cost to fix an error the cost to re-review a document the cost to retest a procedure, case, or script the cost to re-review a module of code

0733-8716/90/0200-0315$01.OO 0 1990 IEEE

Authorized licensed use limited to: Ajay Kumar Garg College of Engineering. Downloaded on January 20, 2009 at 13:07 from IEEE Xplore. Restrictions apply.

316

IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 8. NO. 2. FEBRUARY 1990

0

the cost of computer usage for tests (except the

first) the cost of lab allocations for other than the first tests all expenses associated with fixing problems with released systems the cost of expediting fixes (FAX, overnight mail, E-Mail, overtime, etc.). Failure costs are further categorized into the costs of internal failure and the costs of external failure. The internal failures are those failures which occur under the developer’s control (on the company’s premise). These failures occur in requirements specifications, designs, code, and the early phases of test. External failures are those which fail in front of customers or users and are embarrassing. These usually cause reputation damage and can cause bad will and loss of sales due to poor quality.

Why Measure the Costs of Quality? Management (which is responsible for allocating the resources) understands dollars, return on investment, and profit and loss (P & L). No matter how we educate and train them, they have difficulty understanding the relationship of defects per thousand lines of code (or on a function point base) to real dollar investments in prevention activities. For better or worse, the investment decision makers were brought up in a business school environment. This level of management, however, will understand that you are spending up to 50% of your operating expenses finding, fixing, and explaining away software defects. You have great a opportunity to show them the enormous costs of quality and what can be done to gain improvements. To get funding for improvement projects (prevention), it is essential to show the current cost of defects, the cost of the solution, and the resulting return on investment (ROI). The only way to continually gain prevention resources, and thus attack the problems at the source, is to show the problem cost in dollars, the solution cost in dollars, and ROI. The providers of prevention resources understand ROI. COST COLLECTION METHODOLOGY Where Costs of Quality Must be Measured The costs of quality must be measured at an activity level. Activities are defined as a group of tasks relating to a milestone event. System test would be an activity. Unit test would be an activity. So would coding, requirements review, and design review. Errors occur in each activity. They may either be caused by the activity itself or may be the result of an error from a previous activity. If the error is found and fixed in the same activity, say coding, it is much less expensive than if it originated in function design and did not manifest itself until system test. This is because the cost of correcting a code error in the coding activity is a matter of changing the code, recompiling, and other administrative activities such as reconfiguration and possible re-review,

while the cost of fixing a design error found in system test includes the cost of correcting the design documents, changing the code, and retesting (unit, functional, and system). Determining which activity is contributing the most to quality costs is important in root cause analysis. It will give you a leg up on directing prevention resources to the responsible activity. These can be termed liability costs, since the originator of the error is responsible for the entire cost to fix the error.

What Should be Measured? It is important to measure the relationship among the four costs of quality and the relationship to some base that top level managers understand. The importance of measuring the costs to each other is that it will dramatically show how much is spent on failure costs (fixing defects) and how dramatically little is spent on prevention costs. Appraisal costs should also be shown so that this relationship can be analyzed. Fig. 1 presents a typical cost of quality chart. Notice that as prevention resources are added, total cost of quality reduces. This is why quality is considered free. The relationship of the costs should also be shown in ways that management can understand and clearly see that the total cost of quality is reducing. This can be accomplished by measuring the costs of quality to a known base (operating expenses or budget). Fig. 2 is a typical chart of the total costs of quality to budget. Notice that there is cost improvement in the relationship. If there is improvement in the relationship of cost to budget, then the money that would have been spent on failures now drops through to the bottom line of the Income statement. A Costing Method There are two methods for collecting the costs of quality: through a defect collection and costing system and through the accounting system. The accounting system requires that a labor distribution system be in place and that the time spent on reviews, tests, and audits be coded as appraisal, that the time spent on training, methods, and tools be recorded as prevention, and that the time spent on fixing defects be recorded as failure costs. The advantage to this method is that it is an Accounting Department function and the numbers are usually not suspect if the right audits and controls are in place. There are also several disadvantages. 1) The Accounting Department has rigid rules and they sometimes do not lend themselves to incorporating a cost of quality system or changes of any kind. 2) The Development Department is not in control of the costs of quality, the Accounting Department is. Needed changes are sometimes difficult to get because they are not the Accounting Department’s first priority. 3) It is difficult for developers to accurately record time spent on different activities. 4) The costs of quality can be shown, but the trail to

Authorized licensed use limited to: Ajay Kumar Garg College of Engineering. Downloaded on January 20, 2009 at 13:07 from IEEE Xplore. Restrictions apply.

317

MANDEVILLE: SOFTWARE COSTS OF QUALITY

MONTHLY INPUT 4

140 120 100 80 60 40 20

TPUT

0 1989

I MONTHLY TOTAL

I APPRAISAL

EKITRNAL

SET UP

INTERNAL

CALCULATIONS

Fig. 3. System layout.

Fig. I . Total cost of quality. Summary by cost type for project.

I

4

0 PREVENTION

1989

+ MONTHLY TOTAL * DLTERNAL -K- APPRAISAL + PREVENTION

I

1990

I

8 INTERNAL

Fig. 2 . Percent of quality cost to budget.

which defects are causing the largest costs is dead ended. This is an important disadvantage. You need a trail to the defects so that root cause analysis can be performed. A defect collection and costing system uses actual defects and their causes, the labor rates of the personnel fixing the defects, and the average times to fix defects. These defect costs are used in the calculations for internal and external failure costs. The number of reviews, tests, and inspections are used for appraisal costs. Again, there are both advantages and disadvantages to a defect costing system. The advantages are that the development activity is in control of its cost of quality system, the rules of accounting are relaxed, and the system will have the capability to point the way toward the most costly defects. The disadvantage is that the numbers are likely to be less accurate than an accounting system. But, the reason you are using a cost of quality system is to strategically reduce the development costs by getting prevention resources. Accounting accuracy finishes in second place behind root cause analysis and quality improvement. The costs of quality using defects and events (such as reviews and tests) is not well publicized and deserves exploration. To start with, a defect collection system can be a good system for root cause analysis and cost reduction. By investigating the top ten defects occurring during an activity and then performing root cause analysis on the top offenders, the roots of the problems can be determined. However, to get proper prevention funding, you must know the cost of the problem, the cost of the solution, and the return. For example, it would be difficult to obtain an investment of $200 000 in prevention resources if the costs of the problem were not known. And it would be unwise to invest the funds if the return was less than adequate. A cost of quality system for software using defects and events as a base looks like the diagram in Fig. 3.

The set-up database contains the rates of personnel working on particular tasks such as functional design reviews or fixing bugs in a requirements document. These can be established once and reviewed periodically. The data entry database is monthly input, where the number of reviews and tests and the resultant defects are input. The data in the set-up database and the data entry database must be kept at an activity level (i.e., code, function design, system test, etc.) so that the costs can be traced to the activities. This aids in root cause analysis, by getting to the activity with the highest cost. On a monthly basis, the calculations should be performed. Monthly is preferred to quarterly because it is more timely or weekly because the numbers will have more impact. Besides, managers are used to looking at costs and their impact on a monthly basis. Once the costs of failure are determined (failure cost is always the largest), we need to look at the defects which are occurring the most frequently Fig. 4 shows the top ten defects by type. This chart is easily obtained from a defect tracking system. However, if we are interested in strategic cost reductions, we must look beyond the frequency of defects and look at the overall cost of the defects. There is a relationship to the frequency of defects and their costs, however, costs have a stronger relationship to where a defect was found in the life cycle, versus where it was inserted. A review of Fig. 5 shows this. Notice that defect DO0056 on the defect-by-type chart (Fig. 4) is the most frequent, while defect DO0051 (Fig. 5) is the most costly. To strategically reduce costs, the most costly defects should be tackled first. This defect strategy is different than trying to get a release out. In that case, the most severe defects must be fixed first. But, fixing software errors to get a release out does not improve the quality of the software development process. It does affect the quality of the particular release, but since there is no root cause analysis and corrective action, the same errors will appear in release after release. The root causes of defects are rarely paid any attention when you are under release pressure. The improvement in quality and the resultant reduction in costs of quality come only when we determine the root causes of defects and take the appropriate corrective action to eliminate the defect, forever. Using a cost of quality system not only points us toward the most costly defects, but it will also provide liability costing. Liability costs are those costs assigned to the

318

IEEE J O U R N A L ON S E L E C T E D A R E A S IN C O M M U N I C A T I O N S . VOL. 8. N O 2. F E B R U A R Y 1990 ~

~~

TOTAL COST BY DEFECT (Top Ten) March/1990

Fig. 4. Total defects by type. Top ten, July 1989

Unique Project Number Goes Here. Proj No.

Fig. 5 . Total cost by defect. Top ten, July 1989.

originators of the bugs in software. For example, the cause of a bug inserted by the systems requirements function, and found in test, is long forgotten when we are trying to make a release schedule. However, if the costs to fix the problem are assigned to the systems requirements function, a whole new list of management concerns and questions are likely to be raised. Such as, “How did this problem arise?” “How did it get so far without being noticed?” and “What are we going to do about preventing it from recurring?” These are excellent questions and the type that quality professionals and designers should ask themselves. But without the added impact of showing the defects in costs, the requirements and design errors are long forgotten. Fig. 6 is a typical liability chart showing the costs of defects assigned to the functions responsible for them. This chart brings the right emphasis on problem solving. It also provides the first step in root cause analysis and corrective action. We try to solve most problems by getting faster test methods (automated testers, faster test platforms, etc.). This is needed, but the most costly defects are caused by the early life cycle activities (testers do not cause defects, designers do). Root Cause Anulysis and Corrective Action Using Sofrwure Costs of Quality Once you have determined a starting place for your root cause analysis (the most costly defects), you need to determine the cause. After you have determined the root cause, you need to analyze the solutions. The solutions then need to be costed. With the cost of the problem in hand and the cost of the solution, you need to present the return on investment. This is not a trivial task and should not be taken lightly. There are many activities competing for investment funds. You must explain the costs of quality, what poor quality is costing the company, the costs of the solution. and the return that can be provided. Monitoring the cost of quality improvements is a continual task. It is the only way to ensure that your quality

improvement process is on an even keel and headed in the right direction. Opportunities for improvement are always there, and the cost improvements to the bottom line follow along hand in glove with the quality improvement process. The three most important things in improving quality are Management Commitment, Management Commitment, and Management Commitment. The way to get and maintain management commitment is to provide them a solid return on investment, just as they must provide a return to their management and stockholders. Cost of quality systems show you what you are spending on defects, what the most expensive defects are, and where they originate.

CONCLUSION The time has come for software quality professionals to show their contributions to the company’s well being. We have always known that our activities have positively contributed, now we can show it. Establishing a software cost of quality system is an easy, cost-effective method to start you down the road of long-term cost reductions and real quality improvement.

William A. Mandeville (M‘88) received the B . S . degree from the University of New Haven. New Haven, CT. and the M.S. degree from the University of Southern California. He is a nationally recognized expert in the ficld of software quality assurance and software quality costing. His experience spans 20 years in software engineering and quality management with major companies including General Dynamics. ITT. ITEC. and Ericsson. He has prehentcd keveral innovative papers and has led national discussion panels o n new methodologies in the area of commercial and military software quality costing and defect collection and reporting. He is currently the Vice President of Development with the Carman Group. Dallas. T X . The Carman Group provides software quality aswrance tools. training. and consulting services.

Authorized licensed use limited to: Ajay Kumar Garg College of Engineering. Downloaded on January 20, 2009 at 13:07 from IEEE Xplore. Restrictions apply.