Quality Assurance Institute’s (QAI) Effective Methods of Software Testing A Perspective Seattle Area Software Quality Assurance Group John Napier, CSQE, CSTE BEST Consulting 10/14/08
Purpose
Value of Professional Associations Value of Software Certification programs QAI’s “Effective Methods of Software Testing”
Disclaimers
The opinions expressed in this presentation are my own, and not those of: ◆ ◆ ◆ ◆
BEST Consulting The Boeing Company Quality Assurance Institute SASQAG
What its like in the trenches...
No common language “In my last job…” No individual professional context No organizational professional context Can’t see the forest...
What’s a person to do?
Establish a personal professional context Get to know “the forest” Help create a common language Help bring best practices into your organization
The Quality Assurance Institute What’s QAI?
10/14/08
The Quality Assurance Institute
An international association of 1000 organizations banded together to share best practices, benchmark data and experiences Software quality focus Certification Programs ◆ Certified Quality Analyst ◆ Certified Software Quality Engineer
QAI Certified Software Test Engineer (CSTE)
Optional Skills Mandatory Skills ◆ Test Planning ◆ Risk Analysis ◆ Use of Test Tools ◆ Measurement ◆ Test Performance ◆ Test Case Design ◆ Defect Management ◆ Establishing and Maintaining a Test Environment
ASQC* Certified Quality Software Engineer (CQSE)
Experience Professional Association Software Engineering Body of Knowledge ◆ Exam Many other ASQC Certification Programs
*American Society for Quality Control
Effective Methods of Software Testing Carolyn Fairbanks Q-Star Consultants 10-14 March 1997 Seattle 10/14/08
Effective Methods of Software Testing There are really only two issues in software testing:
Requirements Organizational Culture 10/14/08
QAI Test Materials Organization
Test Strategy Test Plan Supporting Materials ◆ Metrics ◆ Test Toolbox - Techniques ◆ Test Conditions - Techniques ◆ Test Case Design - Techniques
Test Strategy Definition
1. Type of Project 2. Type of Software 3. When Testing will Occur 4. Critical Success Factors 5. Who will Conduct the Testing 6. Tradeoffs
1. Types of Project
Traditional Client/Server Prototyping Object Oriented Purchased / Contracted
2. Types of Software Batch Process Control Graphic User Interface Object Oriented Procedure Control Expert System Multi-Media Advanced Math Models
Diagnostic Software Sensor and Signal Simulation DB Management Data Presentation Decision Analysis Pattern & Image Computer Operations Software Development Tools
3. When will Testing Occur?
Requirements Phase Activities Design Phase Activities Build Phase Activities Test Phase Activities Installation Phase Activities Maintenance Phase Activities
4. Critical Success Factors Correctness Authorization Data Integrity Audit Trail Continuity of Processing Service Levels Access Control
Methodology Reliability Ease of Use Maintainable Portable Coupling Performance Ease of Operations
Testing Concerns Matrix Critical Success Factors
Correctness
Life Cycle Stages Requirements
Design
Complete, Clear, Correct,
Etc
Testable, Feasible
7. Tradeoffs (What can be given up?)
Schedule Cost / Resources Scope Technology Quality / Critical Success Factors
Some QAI Philosophy Test Life Cycle, Test Workbench
10/14/08
Testing Relationship Definitions Testing Lifecycle Concept Acceptance Test
Operational or Business Need
System Test
Define Requirements
Integration Test
Design System Build System
Unit Test
Testing Relationship Definitions Continuous Life Cycle Testing Validate
Verify
Acceptance Test
Operational or Business Need
Validate
Verify
System Test
Define Requirements
Validate
Verify
Integration Test
Design System
Verify Build System
Validate Unit Test
Testing Relationship Definitions Test Support Activities TEST OBJECTIVES Validate
Verify
Acceptance Test
Operational or Business Need
TEST PLANNING
Validate
Verify
System Test
Define Requirements
Validate
Verify
Integration Test
Design System
TEST EXECUTION (STATIC)
Verify
Build System
Validate
Unit Test
TEST EXECUTION (DYNAMIC) & TEST REPORTING
QAI Definitions “Test Workbench” Concept Problems Products for Test
Test Processes
Test Quality Control
Test Toolbox
Test Measures
Validated Test Results
The QAI Test Process 1. 2. 3. 4.
Set Test Objectives Develop Test Plan Execute Tests Summarize and Report Results
Step 1 - Set Test Objectives
Task 1 - Organize Test Team Task 2 - Perform Risk Assessment Task 3 - Set Test Objectives Task 4 - Quality Control
Set Test Objectives
Task 1 - Organize Test Team
Define Test Responsibilities / Tasks Determine People to Perform Tasks
Worksheets 4,5
Set Test Objectives
2. Perform Risk Assessment
Dimensions of Risk ◆ Criticality / Mission Impact ◆ System Size, Scale and Complexity ◆ System Environment and Stability ◆ Reliability and Integrity ◆ Technology Integration Simple or Complex Scoring Approach
Impact of Failure Likelihood of Failure
Set Test Objectives
3. Set Test Objectives
Using: Prioritize Each◆ System Objectives Low ◆ Test Strategy Medium High ◆ Risk Score Analysis ◆ Team Assignments Worksheet 6 Set Objectives ◆ Brainstorming ◆ Relate Objectives to Function / Output ◆ Relate Objectives to Events / System Input
Set Test Objectives
Quality Control
Objectives QC Checklist Objectives Incident Report (Change Request)
…lots of checklists!
Develop Test Plan
1. Define Business Processes 2. Define Structural Attributes 3. Define Tests 4. Create Test / Function Matrix 5. Define Admin Requirements 6. Formulate System Test Plan 7. QC System Test Plan 8. Develop Unit Test Plan 9. QC Unit Test Plan
Develop Test Plan
1. Define Business Functions
Describe what the system should do ◆ …stands alone as an activity ◆ …is controllable ◆ …can be a transition ◆ …can be described by a process
Business Function Matrix Function
Description
Evaluation Criteria
Worksheet
8
Develop Test Plan
2. Define Structural Attributes
Reliability Efficiency Integrity Usability Maintainability
Testability Flexibility Portability Reusability Interoperability
Worksheet 9
Structural Attribute Matrix Attribute
Priority
Description
Evaluation Criteria
Develop Test Plan
3. Define Tests
Using ◆ Test Objectives Worksheet ◆ Business Functions Worksheet ◆ Structural Attributes Worksheet Define Tests ◆ Associate test factors, techniques and test types ◆ Associate evaluation criteria with techniques and types of tests ◆ Determine type of test which satisfies each objective
Type of Test Techniques
Functional ◆ Requirements ◆ Regression ◆ Error Handling ◆ Manual Support ◆ Intersystems ◆ Control ◆ Parallel
Structural ◆ Stress ◆ Execution ◆ Recovery ◆ Operations ◆ Compliance (to process) ◆ Security
Test Data Sources
Test Data / Test Desks Production Data Simulation / Prototyping Review / Confirmation Instrumentation / Tracing Test Script
Test Definition
Each Test is defined in terms of: ◆ Controls ◆ Inputs ◆ Outputs ◆ Procedures ...At a high level
Develop Test Plan
4. Create Function / Test Matrix Function
Tests
Test 1 Test 2
Function #1
25
17,30
Function #2
5,7
45
Ect.
Develop Test Plan
5. Define Admin Test Requirements
Schedule Budget Resources Testing Materials Test Training Tests to be Conducted Defect Estimation Worksheets 13,14
Develop Test Plan
6. Formalize Test Plan
Section I - General Information Section 2 - Plan Section 3 - Specifications and Evaluation Section 4 - Test Description
See Test Plan Example
Develop Test Plan
7. Test Plan Quality Control
Quality Control Checklist Test Plan Incident Report (Change Request)
…even more checklists!
Develop Test Plan
8. Develop Unit Test Plan
Using: ◆ System Test Plan ◆ Program Initiation Document ◆ Program Design / Specification Documents Develop ◆ Test Plan Administration ◆ Subfunction / Test Matrix ◆ Interface / Test Matrix Worksheets 20,21,22
Develop Test Plan
9. Unit Test Plan Quality Control
Checklists
Step 3 - Execute Tests
Task 1 - Select Test Tools Task 2 - Develop Test Cases Task 3 - Execute Tests Task 4 - Quality Control
QAI Tool Definition
What’s a Test Tool? ◆ …any vehicle which assists in performing a test. ◆ Examples Decision
Tables Automated capture / playback Checklists Program Analyzer
Execute Tests
Step 1 - Select Test Tools
Using: ◆ System Test Plan ◆ Unit Test Plan ◆ Tool Selection Methodology
Execute Tests
Step 2 - Develop Test Cases Test Condition
Expected Result
Test Procedures
Worksheet 23
Execute Tests
Step 3 - Execute Tests
Test Activities Log Regression Testing ◆ Full ◆ Partial
Worksheet 24
Step 4 - Summarize and Report Test Results
Task 1 - Record Defects Task 2 - Perform Data Reduction Task 3 - Develop Findings and Recommendations Task 4 - Finalize Test Report Task 5 - Test Report Quality Control ◆ Test Defect Report ◆ Test Defect List ◆ Test Defect Log Worksheets 25,26,27,28
Supporting Material
Metrics Toolbox Techniques Test Condition Analysis Test Design Analysis 10/14/08
Test Metrics Method Goal - Question - Metric
QAI Recommend Test Metrics
Defect Reports Requirements Implemented Backlog of Defects (Not Corrected) Status of ◆ Project, Tests, Schedule, Budget Selected Test Metrics
QAI Test Metrics Selection Types 1 4 3 20 2
General Subject Involvement in Testing Extent of Testing Resources Consumed in Testing Effectiveness of Testing Assessment of Testing
The QAI Test Toolbox
Data Modeling Boundary Value Analysis Capture/Playback Cause-Effect Graphing Change Control Trackers Checklists Checkpoint Review Code Comparison Compiler-based Analysis Complexity-based Analysis
Compliance Checkers Control Flow Analysis Correctness Proofs Coverage Based Analysis Data Dictionary Decision Tables Defect Trackers Desk Checking Equivalence Partitioning Error Guessing
The QAI Test Toolbox
File Comparison Flowcharts Help Desk Trackers Inspections Instrumentation Integrated Test Facility Mapping Model Animation Model Balancing Network Analyzers
Performance / Timing Analyzers Population Analysis Risk Matrix Scoring Snapshots Symbolic Execution System Logs Test Case Generator Test Data Generator Test Scripts
The QAI Test Toolbox
Test Stubs and Harnesses Test Suite Managers Tracing Walkthroughs
The QAI Test Toolbox Test Tools
Types of Functional and Structural Tests X
Boundary Analysis Population Analysis Checklists
X X
X
X
Test Condition Analysis
Sources ◆ 1. Specification Decomposition ◆ 2. Population Analysis ◆ 3. Business Process Analysis ◆ 4. Structural Analysis ◆ 5. Test Transactions
Test Condition - Expected Result - Test Process
Identify Test Conditions
1. System Specifications or Production Environment 2. Population Analysis ◆ Files and Tables ◆ Screens ◆ Field/Data Element
Test Condition - Expected Result - Test Process Worksheet 4
Identify Test Conditions 3. Test Transaction Types
Field Record File Relationships Error Use (Outputs) Search
Match / Merge Stress Control Attributes States Procedures Control Flow
Test Condition - Expected Result - Test Process
Identify Test Conditions
4. Business Cases ◆ …business scenarios… 5. Structural Analysis ◆ …decision points...
Test Condition - Expected Result - Test Process
Design Test Cases
Design Test Cases Develop Scripts / Sets
Expand Test Conditions
Design Test Cases
Expand Conditions into Test Cases using ◆ Boundary Value Analysis ◆ Equivalence Partitioning ◆ Decision Tables ◆ Code Coverage Statement
Coverage Condition Coverage Expand Test Conditions
Design Test Cases Test Scripts Actor/Scene …Sam scowls at the report...
Dialog “Where did this come from?”
Detailed Test Script Descriptions Script Reuse Expand Test Conditions
QAI Summary - Ten Key Test Concepts
Use strategic test questions to set direction Develop Test Objectives Use a cross-functional “test team” Prepare a test plan, then execute it Utilize both verification and validation methods Collect and analyze defects Prepare test management reports
QAI Summary - Ten Key Test Concepts
Make test efforts reusable Feed the continuous improvement process with information from testing Continuously improve the process
Incorporating the Process Models within Models …Make them your own
10/14/08
Questio ns?