Work Term Report

The Cost of Defects Joseph Cheng APSC 210 606

Ashley Whitehead Atimi Software Submitted on May 8, 2009

Letter of Transmittal To whom it may concern: This report was prepared for Ashley Whitehead, a Quality Assurance lead at Atimi Software. It details the concept of defect fixing, a crucial element in any software development cycle. Ashley Whitehead has reviewed this report, and all the contained information is confirmed to be non-confidential and within the public domain. This report focuses on the economic perspective of defects, detailing how certain factors can improve development efficiency and reduce total project costs. This work was done from January to May 2009. Sincerely,

Joseph Cheng

The Cost of Defects Joseph Cheng 14157077 January – May 2009

Submitted to UBC Science Co-op Engineering Physics

I certify that this report is non-confidential and may be submitted to UBC

Abstract

This report compares different factors that could influence the cost and efficiency of fixing defects, errors that prevent software from behaving properly. Defect fixing consumes most of a developer’s time; a Capers Jones study in 1998 found that 74 percent of a developer’s time is spent fixing defects rather than developing. Customers care most about getting what they want, in this case a functional product. However, it is sometimes too time consuming and inefficient to fix certain defects. By weighing the consequences and risks of defects, companies will be able to increase profit and better satisfy customers.

iii

Summary

Defect fixing is a very time consuming task for developers. Dealing with this problem effectively will significantly reduce the costs spent on a project and greater success in the market. The key to this problem is time, because the longer detects go undetected in a project, the harder they will be to fix. To do this, quality assurance staff is needed to weed out design errors before coding takes place. During the development phase, the software should be continually tested starting from the basics. Also during the development phase, teams should ideally be openly discussing issues, and resolve them in an orderly fashion. Lastly, both developers and customers need to realize that almost all software is defective. At some point, defect fixing must stop because changing the code any further risks introducing worse problems.

iv

Table of Contents

Abstract………………………………………………………………………………… ………………. ii Summary………………………………………………………………………………… ……………... iii Table of Contents……………………………………………………………………… ……………. iv Introduction…………………………………………………………………………… ……………… 1 Quality Assurance…………………………………………………………………… ……………... 2 Quality Control………………………………………………………………………… …………….. 3 Company Culture……………………………………………………………………… ……………. 4 Variable…………………………………………………………………………… ……………. 5 Routine…………………………………………………………………………… …………….. 5 Steering…………………………………………………………………………… ……………. 6 Risky Changes………………………………………………………………………… v

……………… 7 Conclusion……………………………………………………………………………… ……………… 8 Works Cited……………………………………………………………………………… ……….…… 9 Bibliography…………………………………………………………………………… ……………… 10

vi

Introduction

All software companies face the unique challenge of fixing defects. Defects fixing requires a lot of time and effort from developers, meaning more costs; therefore dealing with them efficiently is vital. The solution to this problem is not always clear, and usually comes through experience because of the number of factors involved. Ultimately the results of this challenge will determine whether the project will profit the company or not.

This report aims to establish how a company’s approach to defect fixing will affect the costs of a project. In particular, the effects of having competitive testing staff, good management, and experienced developers and how they affect the project’s costs will be explored.

Quality Assurance

Software quality assurance involves monitoring and improving the software development process to ensure certain procedures and standards are followed. It is a process that attempts to prevent defects. (Hower) Defect prevention begins with the requirement analysis – understanding what the customer wants and transforming it into product specifications without introducing errors. This is important because errors in the software requirements and design documents are more frequent than in the source code itself. (Soni)

An integral part of quality assurance is to analyze defects, find the root cause, determine a quick solution and take preventive action. These preventive measures, after approval from team members are embedded into the company as a baseline for upcoming projects. This methodology provides companies a long-term solution and experience from learning from their mistakes. Preventing defects that originate from the design phase could save as much as 64 percent of the total defect cost. (Soni)

viii

Quality Control

Software quality control involves operating a system or application in a controlled environment, and assessing the results. Testing conditions and procedures, commonly called a “test case” should cover both normal and abnormal behaviour. (Hower) Ideally, quality control staff should start testing as early as possible to maximize efficiency, because detecting defects early on dramatically affect the costs of fixing them. (Soni)

Compared to finding a defect in the early stages, defects found late in the development process can cost almost six times more. Defects found during programming cost around a quarter as much as defects found after coding is finished. Overall, fixing problems that crop up late in the development process contributes to 85 percent of the defect fixing effort. (van Megan and Meyerhoff)

ix

Company Culture

The approach companies take towards defect fixing depend largely on the management. The attitude the manager takes towards quality will significantly affect how the project will be dealt with. For example, compare the mind-set of the two following engineering managers:

"We'll fix that when we have time. In the meantime, just keep developing! How can you possibly tell how much it will cost to fix later?" — An Engineering manager, eager to continue software development

"Let's find all the defects before system test. The customer will wait for the product." — An Engineering manager, concerned that shipping a product with defects would prevent customers from buying the product

x

Management often has to make the choice of when to fix defects, and their beliefs will determine the development process within a company. Weinberg identifies three types of company cultures, with each culture prioritizing quality and defect fixing differently. (Rothman)

Variable Culture

The engineering manager eager to continue software development represents a company with a variable culture. Management does not recognize that quality is a management issue, and problems are fixed through personal effort. The company becomes inefficient at fixing defects, often finding more than they could fix. After shipping their product, many issues arise as customers become disappointed about the product. The company must go through the testing cycle again, which incurs additional costs from developing release notes, extensive testing on different hardware and software platforms, testing areas around the fix, and then merging the fix back into the build. (Rothman)

Routine Culture xi

Within a routine culture, management agrees quality is valuable, but no time or money is allocated to ensure quality is there. However, management does understand the consequences of releasing a product that is not up to standards. Teams usually fix problems only after a severe defect has been found. Typically, the company culture also helps find all the easy defects before testing occurs. This type of management is most effective with small-scale projects, fixing almost all defects during the development phase. Conversely, fixes are rarely made once the product has been shipped, but costs will significantly increase if they need to. (Rothman)

Steering Culture

The second engineering manager fits the criteria of a steering culture. Management in a steering culture understands the importance of quality, and that quality is a systemic issue. In this environment, emphasis is on preventing defects rather than testing and problems are discussed and resolved in an orderly manner. Steering culture management deals with defects the best, finding and dealing with them faster than other types of company. Only minor problems are

xii

ever seen after release, and the costs to fix them are minimal. (Rothman)

Risky Changes

At some point, developers have to stop fixing defects because it risks introducing worse ones. Four factors need to be considered about defects: Severity, Frequency, Cost, and Risk. Defects with high severity and frequency are given the most priority and should be fixed. Defects xiii

with low severity and frequency should not be fixed. Cost and risk are tradeoffs involved with fixing the defect; if the costs and/or risk are high, the relevant defect’s priority should go down. It is critical that the expectations of the customer and market are understood, because defects are sometimes acceptable. Experienced developers should be able to make these decisions quickly, so that time and effort is not wasted on risky problems. (Sink)

Conclusion

xiv

In 2002, a Research Triangle Institute study estimated that software defects cost the U.S. $59.5 billion annually. The report further stated that improving testing infrastructure would enable defects to be dealt with more effectively and potentially eliminate $22.2 billion of the costs. Costs can be lowered by 64 percent from having QA staff, and a further 85 percent by having QC staff. Openly discussing problems is also critical, especially in larger projects as it makes it much easier to release fixes after release. Additionally, discussing problems will allow managers and even customers to understand why some problems occur so they can decide whether it’s worth the time and effort to fix it. Being able to avoid or reduce these costs while still satisfying customers is key to success in the market.

xv

Works Cited

Bank of Montreal, Toronto, Ontario | ispw.com. 11 Jan. 1998. ComputerWire, Inc. 11 Apr. 2009 . Hower, Rick. Software QA and Testing Resource Center. 17 Apr. 2009. Digital Media Group, Inc. 8 Apr. 2009 . Rakitin, Steve. Food for Thought: All Software is Defective. Dec. 2005. Software Quality Consulting, Inc. 16 Apr. 2009 . Rothman, Johanna. What Does It Cost You To Fix A Defect? And Why Should You Care? Oct. 2000. Rothman Consulting Group, Inc. 12 Apr. 2009 . Sink, Eric. Inside IT: Why we all sell code with bugs. 25 May 2006. The Guardian

xvi

Newspaper. 15 4 2009 . Soni, Mukesh. Defect Prevention: Reducing Costs and Enhancing Quality. 19 July 2006. iSixSigma.com. 14 Apr. 2009 .

Tan, Gang. A Collection of Well-Known Software Failures. 22 Dec. 2008. Lehigh University. 15 Apr. 2009 < http://www.cse.lehigh.edu/~gtan/bug/softwarebug.html>. van Megan, Rudolf and Dirk Meyerhoff. "Costs and benefits of early defect detection: experiences from developing client server and host applications." Software Quality Journal 4 Apr. 1995: 247-56.

xvii

Bibliography

Software Quality Assurance. N.d. 14 April 2009 . “QA – Quality Assurance and Software Testing” Ask Numbers. N.d. 13 April 2009 . “Popular testing phrases: Testing Should Start Early.” Compendium Developments. 1 April 2002. 14 April 2009 . “Cost of fixing software defects ‘runs into millions’” SC Magazine. 18 July 2008. 16 April 2009 . “Testing vs. Quality Assurance” CM Crossroads. 30 May 2006. 11 April 2009 . xviii

“The Economics of Testing” Rice Consulting. N.d. 14 April 2009 .

xix