Defect Prevention Training

Defect Prevention Training Induction – Sep 2007

Protection notice / Copyright notice Version 2.0

Introduction Defect Prevention is a process of improving quality and productivity by preventing the injection of defects into a software work product. Definition: “…an activity of continuous institutionalized learning during which common causes of errors in work products are systematically identified and process changes eliminating those causes are made.” [Eickelmann] SEI has identified ‘Causal Analysis and Resolution’ as Level 5 PA of CMMI

Page 2

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Objectives Course Objective To enable participants understand and apply defect prevention concepts

Defect Prevention Objectives  Identify and analyze the causes of defects& Reduction in number of defect categories  Reduction in the extent of defect escape between phases  Reduction in frequency of common defects  Improvement in PCB values

Page 3

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Contents Defects and Bugs (Examples) Origin of Defects Classification of Defects Defect Management Defect Detection Defect Prevention Cycle

Page 4

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Objectives of Defect Prevention • Establish practice of Root Cause Analysis within projects for Analysis of Identified Defects • Identify critical processes as part of root cause analysis • Set goals for improving critical process (shift mean and narrow variation) • Reduce most frequent type of defects such as “ not following coding guidelines” • Analyze opportunities for improvement by conducting escape analysis. • Use defect distribution data to drive process improvement activities • Spread lessons learnt - Team Meetings, SEPG, Process Database

Page 5

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Responsibility Project team is responsible for the DP activities pertaining to the project life cycle activities & Project Manager (at project level) Project Quality Manager (at project level) P&Q (at Org level) SEPG (at Org level)

Page 6

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Defects and Bugs

Page 7

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Defects and Bugs - Example The Atlas-Agena spacecraft, destined for Venus, had to be blown up during launch because it became unstable about 90 miles up. (Malfunctioning rockets have to be destroyed to avoid crashes in populated areas). A missing hyphen in the flight plan resulted in the loss of the $18.5 million US spacecraft.

Page 8

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Defects and Bugs - Example On January 15, 1990, 114 switching computers in the AT&T telephone network crashed because of a software flaw. 65 million subscribers were unable to use their phones. The problem arose when a switching computer in New York crashed, sending out a digital “out of service” message to nearby computers. Normally, other switches would route traffic around the disabled computer. However, a misplaced “break” in a C statement caused the nearby computers to go down as well. For the next 9 hours, the switches went down, rebooted themselves, and came back up, only to go back down immediately.

Page 9

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Effect of Software Error •unreasonable added cost •lost time and effort •inconvenience and annoyance •death

Page 10

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Could these incidences of software errors been prevented?

YES!

Page 11

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Product and Process Defects

Definition

Types of Defects

PRODUCT Artifacts created during the life cycle of the project.

PROCESS Complete set of activities needed to transform user requirements to a product.

Product Defect are related to requirements : Functional and Non-Functional

Process Defects are related to tasks/activities: Non-adherence to standards

Strategy for Handling Defect Removal/ Elimination Defect

Poor Documentation Schedule overrun Defect Prevention Training related

Product defect is always a result of Process Defect Process defect is like a potential carrier of disease Page 12

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Stages of a Software Cycle

Proposal Requirements

Design

Coding

Testing

Maintenance

Reviews White Box Black Box Stress/Load Problem Reports/CRs

Defect Prevention - Feedback and Process adjustments Defect Analysis and Process Improvement

Page 13

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Origin of Defects

Page 14

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Human Errors Types of Errors Omission

More than 80% of software errors are human

Ignorance Commission Typography Knowledge Information External Page 15

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Translation Errors

NEEDED

URS

TOLD

DESIGN

BUILD Detected Bugs

TRANSLATION ERRORS

Hidden Bugs

A requirement is often stated in terms of a solution Focus on solution may hide the real requirement The mismatch between the solution desired and the real requirement leads to translation errors Page 16

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Design Errors Types of Errors Mitigation of design errors

Errors that affect data integrity Errors that alter correctly stored data Incorrect algorithm used to compute a value

Checklist

Some examples: •Does each module in the system design exist in detailed design? •Are all assumptions explicitly stated? Are they acceptable? •Have the exceptional conditions been handled? •Are all data formats consistent with the system design? •Are the loop termination conditions properly specified? Page 17

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Coding Errors Exception handling Incorrect Algorithm Missing Functionality Language pitfalls Memory release Omitted program sections

Page 18

Sep-07

P&Q

A programming error alters a program’s ability, in a negative sense, to completely and effectively meet the user’s requirement.

Protection notice / Copyright notice For Internal Use Only

Testing Errors Failure to notice a problem Misreading the screen Failure to execute a planned test Failure to use the most ‘promising’ test cases. Ignoring programmers suggestions Corrupt data file used Incorrect test cases Concentration on trivial Failure to report Page 19

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Objectives of Defect Prevention 140 120 100 80

Ideal Acceptable Costly Disaster

60

rt R ep o

g Pr ob

le m

Te st in

R ev e

C od

R ev ie w

w ev ie

D es ig n

R

FS

R S

R ev ie w

40 20 0

Defect Detection as early as possible ❑ High proportion defect reported by customer - Unacceptable ❑ More defect detected in testing – Quality at High cost ❑ Maximize defect detection during reviews- Quality at right Protection notice / Copyright notice Page 20

Sep-07 cost

P&Q

For Internal Use Only

So where should the focus be? On Proactive Defect Prevention On Early Defect Detection On Usage of Past Experience

Page 21

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Defect Prevention

Page 22

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Defect Management Rules Barry Boehm Victor Basili

•Fixing downstream is costly… •Rework eats away resources... •Pareto Rule - About 80% of the available rework comes from 20% of the defects

•Another Pareto Rule - About 80% of the defects come from 20% of the modules (and about half the modules are defect free) •Peer Reviews catch 60% of the defects •Perspective-based reviews catch 35% more defects than nondirected reviews (use of checklists) •Disciplined personal practices can reduce defect introduction rates by up to 75% •About 40-50% of user programs enter with trivial defects

Page 23

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Activities Performed during DP – Organization Level • Defect Prevention Plan is prepared at SBU-level (by TC/Business Partner) – contains lessons learnt and improvement actions • SPI (Software Process Improvement) Plan is an overall Plan for the Organization and bears reference to DP Plan as well • DP Plan is shared with SEPG members and PQMs • Lesson’s Learnt from SBU is disseminated through SEPG • Improvement Action are tracked and DP Plan is updated accordingly Page 24

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Activities Performed during DP – Project Level •Kick-off/Start PES meeting – •Goal setting, • Identification of critical processes • Incorporation of Lessons Learnt from previous projects as preventive measures • Phase-wise Defect Distribution goal setting • Defect reporting – Reviews & Testing • Root Cause Analysis • Action Implementation • Information Dissemination Page 25

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Root Cause Analysis • • • •

• • • • • •

Causal Analysis should be conducted as early as possible after a defect is found Evaluate all errors (team effort) - from LOD and test results. All team members learn from the errors of other team members Prepare Pareto Chart for common causes and identify special causes  Special causes of defects are focused on defects, which cause high impact to the project.  Common causes of defects are focused on high frequency defects that occur often in the project or across projects. Perform Fishbone Analysis & escape analysis Create an action list against root causes Compare the results of the tasks performed with the goals set in the kickoff meeting Record Corrective and Preventive Actions (in PPR/PQR, RCA report) Disseminate information to team - Project Meetings Disseminate information across projects - SEPG

Page 26

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Causal Analysis Cycle Reviews/ Testing

Test Logs LODs

Classify Defects (Type, Injected & Detected Phase) Identify Top 80% Defects for RCA

Application

And select all high impact defects

Preventive Feedback

Prj RCA Report

Page 27

Sep-07

Arrive at Root Cause And Action List

P&Q

Perform Fishbone Analysis Using Potential Causes

Protection notice / Copyright notice For Internal Use Only

Pareto Chart What is a Pareto Chart Bar chart arranged in descending order Bars on the left are more important than those on the right Separates the “vital few” from the “trivial many” Uses of Pareto Chart Breaks a big problem into smaller pieces Identifies most significant factors (80-20 rule) Shows where to focus efforts Allows better use of limited resources

Page 28

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Pareto Chart - Exercise Participants to discuss possible Code Review Defects Classify the defects under different categories/types and assign a number of defects against each Prepare a Pareto Chart using Excel to focus on the most significant defects (80-20 rule)

Page 29

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Page 30 Sep-07 P&Q Naming convention not followed

Wrong parameters passed

Header incomplete

Code readability

Functionality missing

Implementation error

Comments not exhaustive

Remove debugging comments

Redundant code

Logical error

Number of Defects 8

7

6

5

4

3

2 40

1 20

0

% of Defects

Pareto Chart (contd.) Pareto Chart - Code Review 120

100

80

60

0

No. of defects

Percentage

Protection notice / Copyright notice For Internal Use Only

Cause & Effect Diagram - Fishbone What is a Cause & Effect Diagram? A graphic tool that helps identify, sort and display possible causes of a problem or quality characteristic Benefits of CED Determination of root causes Encourages group participation Indicates possible causes of variation Uses a orderly, easy-to-read format Identifies areas for collecting data Page 31

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Cause & Effect Diagram • Decide the “Effect” to examine • Identify the main categories • Identify as many causes or factors as possible and attach them as sub branches of the major branches • Identify increasingly more detailed levels of causes by asking a series of why questions • Look for causes that appear repeatedly. These may be root causes • Identify and circle the causes that we can take action on

Page 32

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Cause & Effect Diagram Method

Machinery Wrong gears used

Driven too fast

Carburetor needs adjustment

Under inflated tires

Poor Maintenance

Wrong Octane gas used Poor Driving habits

Sep-07

Improper Lubrication

Materials

Man Page 33

Poor Mileage

P&Q

Protection notice / Copyright notice For Internal Use Only

Exercise on RCA

Make Groups Assign a PM to each group Brainstorm and prepare a cause and effect/fishbone analysis Present the result (20 mins)

Page 34

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Defect Estimation

Phases Requirement Design Coding & UT IT/ST AT

AT 3%

IT/ST 27%

Requirement 7%

Coding & UT 49%

Page 35

Proposed Goal 9% 16% 44% 29% 2%

Defect Distribution 7% 14% 49% 27% 3%

Sep-07

P&Q

Design 14%

- Use Historical Data - Focus on Business Objectives and Process Improvement - Set more Challenging Goals Protection notice / Copyright notice For Internal Use Only

Applicability of RCA •Defects •Customer Feedback •Non-conformance (NC) •Process Capability Baselines •Major Issues (that impact cost, quality, schedule)

Page 36

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Example DEFECTS AT END OF CODING & UT Process/ Standards

People Not Trained/ Inadequate resources Guidelines not followed

Guidelines not updated

Measurement Assets not available

Data not adequate

Coding & UT Not aware

Hardware/Software/ Tools

Support/ Guidance Page 37

Sep-07

Not adequate Tool not inspected

Not Communicated Long Overdue Not Available

P&Q

Protection notice / Copyright notice For Internal Use Only

Common Defect Types Defect Types

Examples

Function/Class/Object error is one that affects significant capability, end-user interfaces, product interfaces, interface with hardware architecture, or global data structure(s) and should require a formal design change. Assignment error indicates a few lines of code, such as the initialization of control blocks or data structure. Interface/Messages corresponds to errors in interacting with other components, modules or device drivers via macros, call statements, control blocks or parameter lists. Checking addresses program logic, which has failed to properly validate data and values before they are used. Timing/serialization errors are those, which are corrected by improved management of shared and real-time resources. Build/package/merge describe errors that occur due to mistakes in library systems, management of changes, or version control. Page 38

Sep-07

P&Q

Database design/modeling error, functionality not working, etc

Oversight during coding, initialization of parameters/variables, incorrect setting of variables, java script validation, etc

Incorrect validation, missing validation, error handling, return value not checked

Incorrect packaging, Setup problem, etc

Protection notice / Copyright notice For Internal Use Only

Common Defect Types (contd.) Defect Types

Examples

Documentation errors can affect both publications and maintenance notes. Algorithm errors include efficiency or correctness problems that affect the task and can be fixed by (re)implementing an algorithm or local data-structure without the need for requesting a design change. External Environment errors that occur due to factors that are outside the application scope. Performance errors affect the performance of the system. Database errors are related to errors in database or scripts. Trivial/Minor

Unclear specifications, standards not followed, redundant code, GUI errors, incorrect description, ambiguous description, etc

Hard coded values used, data type mismatch, etc Test data, test drivers, other tool defects, support system, concurrent work, inherited from previous release, third party software dependency, etc. Memory not released, Web session timeout not handled properly, browser cache related problems, etc Integrity constraint violated, SQL statements not tuned, Error in SQL statement, etc Typo/minor errors in documentation, rephrasing, extra information in document, etc

Page 39

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Common Root Causes Major Defect Categories (from Fishbone)

Root Cause

Support/Guidance (e.g. Management Support, Training, etc) Handover (Change Coordination) Inadequate training (QMS, Defect Prevention, technical) Process/Standards Guidelines/Standards/Procedures not updated Inadequate Process for Handling Requirements/Design Change in Requirements/Design People Breakdown of communications Lack of knowledge (domain/system/tool) Oversight Hardware/Software/Tools Configuration related problem Inadequate tools Measurements Incorrect analysis of data

Page 40

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Escape Analysis (Requirements defects getting slipped to next phases of Life cycle) Phase Detected Requirements Design Phase Injected Requirements 10 9 Design 16 Code review UT IT ST AT Total 10 25

Defects Found during Code Review Defects slipped from Coding (not considering those injected in IT & ST) Defects slipped into Coding Phase TOTAL (Coding + subsequent Phases) Page 41

Sep-07

Code Review

UT 1 3 24

IT 1 1 20

ST 2 1 16

AT

Total

3 2 6

2 3 28

28

52 4 80 P&Q

22

21

Slippage (from Requirements to Design = 9/19) DRE of Requirements Phase (10/19) Slippage (from Coding Phase to subsequent phases = 52/80) DRE of Code Review = 28/80)

14

0

26 23 66 0 2 3 0 120

47% 53%

65% 35%

Protection notice / Copyright notice For Internal Use Only

Root Cause & Action Planning Root Causes of Defects for

<Month_1>,

<Month_n>,

Special Causes: (Root causes of high impact defects) Common Causes: (Root causes of high occurrence defects) Implementation Action Plan Technique

Team to be given training on Domain Knowledge

Workshop every Friday by each team member in turn

Page 42

Sep-07

Monitoring technique

Expected Actual Associated Impact Person Priority Date of Status Date of Risk on PCB * Responsible Closure Closure

PM to ensure that training High is held

P&Q

Defects in software

Schedule 20th June PM Slippage 2004

Open

Protection notice / Copyright notice For Internal Use Only

Some Lessons Learnt

1

Integration Testing should be scheduled so that the core modules are initially tested.

2 Client should be given overview for SISL's P&Q processes 3 4 5 6 7

Client responsibilities should be clearly communicated in the beginning of the project. Design documents should contain all the necessary validations to avoid validation error Checklist should be used to avoid GUI errors Rigorous unit testing to be done to avoid logical errors. Basic level review and testing should be done at developer's level before handing over the code to the testers and reviewers.

8 Test cases should be formed with test data.

Page 43

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Recap… Defects and Bugs (Examples) Origin of Defects Classification of Defects Defect Management Defect Detection Defect Prevention Cycle

Page 44

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Any Questions?

Page 45

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only

Thank You

Page 46

Sep-07

P&Q

Protection notice / Copyright notice For Internal Use Only