Test Case Design
This document was uploaded by user and they confirmed that they have the permission to share it. If you are author or own the copyright of this book, please report to us by using this DMCA report form. Report DMCA
Overview
Download & View Test Case Design as PDF for free.
More details
- Words: 1,553
- Pages: 14
Research and Technology
TACOS‘04, Barcelona, March 2004
Test Case Design by means of CTM and CTE Joachim Wegener DaimlerChrysler AG, Research and Technology [email protected] Introduction and Motivation State of the Art Classification-Tree Method Classification-Tree Editor CTE Conclusion and Future Work
Wegener, Software Technology Research Dept., Slide 1
Introduction and Motivation Research and Technology
Software Development Requirements at DC Increasing amount of software in products in almost all areas
Cost-effective development and time-to-market crucial for competitiveness
High consequential costs due to faults
Software as differentiating factor for competitive advantage Increasing complexity of systems
Increasing significance of software in safety relevant areas up to 60 communicating embedded controllers up to 1,000,000 lines of software code High requirements for software various buses, hundreds based systems dueoftomessages norms, standards and legal regulations > 2 km wire
Wegener, Software Technology Research Dept., Slide 2
Introduction and Motivation Research and Technology
Applications at DC
Wegener, Software Technology Research Dept., Slide 3
Introduction and Motivation Research and Technology
Applications at DC
Wegener, Software Technology Research Dept., Slide 4
Introduction and Motivation Research and Technology
System Development at DC DaimlerChrysler
Acceptance Test SPICE Assessments
Pre-Development
Vehicle Tests Vehicle Integration
Prototyping
Code Quality Assessments
System Test
Specification
Integration Test
Component Development
Integration
Implementation
Unit Test
Suppliers
Wegener, Software Technology Research Dept., Slide 5
Introduction and Motivation Research and Technology
Testing in Practice • testing is an immense cost factor in the course of system development average distribution of costs for embedded systems system and acceptance testing
20 50
module and integration testing
other development activities
30
• high cost • testing is too unsystematic • low error detection rate
Test automation Test methods
Wegener, Software Technology Research Dept., Slide 6
Introduction and Motivation Research and Technology
Objectives of Testing Through executing a system with selected test data the test is aiming at z detecting errors in the system under test and z gaining confidence in the correct functioning of the test object.
Strong Features taking into consideration the real environment (e.g. target computer, compiler) and z testing the dynamic system behavior (e.g. run time behavior, memory space requirement) z
Weak Features z
an exhaustive test is usually impossible
test data selection by certain criteria
Wegener, Software Technology Research Dept., Slide 7
State of the Art Research and Technology
Test Activities Test Planning Specification
Program
Test Execution
Monitoring
Test Documentation
Test Organization
Test Case Design
Test Evaluation • most important for test quality • methods and tools needed Wegener, Software Technology Research Dept., Slide 8
State of the Art Research and Technology
Testing Methods Functional test: • test case design performed on the basis of the specification • most important test approach • common methods show insufficient systematics when partitioning the input domain, difficult procedure, or no efficient tool support • test size is hard to quantify (coverage of requirements, function points, ...) • difficult to automate • examples: equivalence partitioning, boundary value test, category-partition method Structural test: • test case design performed on the basis of the program structure • common test approach (included in many standards) • not possible to check whether all requirements have been implemented • difficult to automate (limits of symbolic execution) • examples: statement test, branch test, path test
Wegener, Software Technology Research Dept., Slide 9
State of the Art Research and Technology
Testing Methods Statistical Test: • test data generation in accordance with operational profile • operational profile hard to determine, especially for new systems • extensive test evaluation, if no test oracle available Random Test: • generation of random test data • specific test-relevant value combinations are produced only with little probability • extensive test evaluation, if no test oracle available Evolutionary Test: • generation of test data by means of search techniques • test objectives have to be transformed into an optimization problem Mutation Test: • test cases are build in order to kill mutants of original program • no procedure how to define the test cases
Wegener, Software Technology Research Dept., Slide 10
Classification-Tree Method Research and Technology
Functional Testing • lack of functional testing methods which are • efficient • easy to use in practice • suited for automation • lack of testing tools for black-box test case design
Classification-Tree Method efficient functional test method CTM: • systematic stepwise procedure which is easy to understand • graphical notation with compact representation of the over-all test • good test documentation • tool support with the CTE XL • proven in practical use
Wegener, Software Technology Research Dept., Slide 11
Classification-Tree Method CTM Research and Technology
Classification-Tree Method Test object: computer vision system recognizing the size of different building blocks large
Aspects Size
Color
Size
Shape
small large
Color red
green blue
Shape circle
triangle
square
Shape of triangle 1
equilateral isosceles scalene
2 Input Domain
3
Wegener, Software Technology Research Dept., Slide 12
Classification-Tree Method CTM Research and Technology
Classification-Tree Method
Size small
large
Color red
green
Shape blue
circle
triangle
square
Shape of triangle equilateral
isosceles
scalene
Wegener, Software Technology Research Dept., Slide 13
Classification-Tree Method CTM Research and Technology
Classification-Tree Method
Size small
large
Color red
green
Shape blue
circle
triangle
square
Shape of triangle 1
equilateral
isosceles
scalene
2 3
Wegener, Software Technology Research Dept., Slide 14
Classification-Tree Method CTM Research and Technology
Classification-Tree Method
Size small
large
Color red
green
Shape blue
circle
triangle
square
Shape of triangle equilateral
1
isosceles
scalene
2 3
Wegener, Software Technology Research Dept., Slide 15
Classification-Tree Method CTM Research and Technology
Classification-Tree Method Test object: computer vision system recognizing the size of different building blocks classification
large
class
Aspects Size
Color
Size
Shape
small large
Color red
green blue
Shape circle
triangle
square
Shape of triangle 1
equilateral isosceles scalene
2 Input Domain
3
Wegener, Software Technology Research Dept., Slide 16
Classification-Tree Method CTM Research and Technology
Classification-Tree Method Test object: computer vision system recognizing the size of different building blocks Minimality criterion: 5 testcases five disjoint classes for large shape
Aspects Size
Color
Size
Shape
small large
Color red
green blue
Shape circle
triangle
square
Shape of triangle equilateral isosceles scalene
1 2 3
Input Domain
Wegener, Software Technology Research Dept., Slide 17
Classification-Tree Method CTM Research and Technology
Classification-Tree Method Test object: computer vision system recognizing the size of different building blocks Maximality criterion: 30 testcases 2*3*5 large
Aspects Size
Color
Size
Shape
small large
Color red
green blue
Shape circle
triangle
square
Shape of triangle 1
equilateral isosceles scalene
2 Input Domain
3
Wegener, Software Technology Research Dept., Slide 18
Classification-Tree Editor CTE Research and Technology
Classification-Tree Editor CTE • syntax directed • context sensitive • graphical editor to support the systematic test case design by means of the classification-tree method Supporting • hierarchies to structure large classification trees and • hierarchies to structure sets of test cases • description of logical dependencies between classes of the classification-tree • combination rules for the generation of test cases according to logical dependencies and combination rules • sorting of test cases • comprehensive export functionalities • open API and architecture supporting the implementation of tool integrations such as with • DOORS • Test Director • Tessy • Time Partition Testing • Win Runner
Wegener, Software Technology Research Dept., Slide 19
Hierarchies for large trees Research and Technology
2. Test cases can be hierachically grouped: 1. Subtrees can be folded/unfolded at will:
folded subtree
groups of test cases
Wegener, Software Technology Research Dept., Slide 20
Implicit (Syntactical) Rules Research and Technology 1. Classes belonging to the same classification are disjoint (i.e. exclusive). e.g., red, green, and blue
2. If a class is marked all parent tree nodes are marked, too. e.g., equilateral implies triangle
‘triangle’ is implicitly marked disjoint classes
Wegener, Software Technology Research Dept., Slide 21
Usage of Logical Dependencies Research and Technology
1. Specification of impossible or required class combinations e.g., circle => NOT red
2. Verification of dependencies e.g., circle => NOT red
verification of logical dependencies
specification of logical dependencies red
circle
Wegener, Software Technology Research Dept., Slide 22
Combination Rules Research and Technology • express the desired combinatorics between classes and classifications • usage of combinatorial expressions
specification of combination rules
Wegener, Software Technology Research Dept., Slide 23
Automated Test Case Generation Research and Technology • based on combination rules and logical dependencies • algorithm based on heuristic combinatorial search
Wegener, Software Technology Research Dept., Slide 24
Classification-Tree Editor CTE Research and Technology
Real-World Example: Positioning
Wegener, Software Technology Research Dept., Slide 25
API Supporting Tool Integrations Research and Technology
Examples Tool Integration
Acceptance Test Vehicle Tests Vehicle Integration Specification
DOORS
TPT
System Test
Win Runner
MSC
TTCN3
dSPACE Automation Desk
System Test Integration Test Integration MTest
Unit Test
Tessy
Wegener, Software Technology Research Dept., Slide 26
Conclusion and Future Work Research and Technology
Summary • Classification-Tree Method enables systematic test case design for functional testing • successfully applied in practice to nearly all application domains • Efficient tool support by means of the CTE XL • available on www.systematic-testing.com • you can start using it • Many tool integrations available. Further integrations under development.
Future Work • Extensions for test management and product line approaches • Description of expected behavior • Generation rules for sub-trees • Metrics to assess test case design quality
Wegener, Software Technology Research Dept., Slide 27
TACOS‘04, Barcelona, March 2004
Test Case Design by means of CTM and CTE Joachim Wegener DaimlerChrysler AG, Research and Technology [email protected] Introduction and Motivation State of the Art Classification-Tree Method Classification-Tree Editor CTE Conclusion and Future Work
Wegener, Software Technology Research Dept., Slide 1
Introduction and Motivation Research and Technology
Software Development Requirements at DC Increasing amount of software in products in almost all areas
Cost-effective development and time-to-market crucial for competitiveness
High consequential costs due to faults
Software as differentiating factor for competitive advantage Increasing complexity of systems
Increasing significance of software in safety relevant areas up to 60 communicating embedded controllers up to 1,000,000 lines of software code High requirements for software various buses, hundreds based systems dueoftomessages norms, standards and legal regulations > 2 km wire
Wegener, Software Technology Research Dept., Slide 2
Introduction and Motivation Research and Technology
Applications at DC
Wegener, Software Technology Research Dept., Slide 3
Introduction and Motivation Research and Technology
Applications at DC
Wegener, Software Technology Research Dept., Slide 4
Introduction and Motivation Research and Technology
System Development at DC DaimlerChrysler
Acceptance Test SPICE Assessments
Pre-Development
Vehicle Tests Vehicle Integration
Prototyping
Code Quality Assessments
System Test
Specification
Integration Test
Component Development
Integration
Implementation
Unit Test
Suppliers
Wegener, Software Technology Research Dept., Slide 5
Introduction and Motivation Research and Technology
Testing in Practice • testing is an immense cost factor in the course of system development average distribution of costs for embedded systems system and acceptance testing
20 50
module and integration testing
other development activities
30
• high cost • testing is too unsystematic • low error detection rate
Test automation Test methods
Wegener, Software Technology Research Dept., Slide 6
Introduction and Motivation Research and Technology
Objectives of Testing Through executing a system with selected test data the test is aiming at z detecting errors in the system under test and z gaining confidence in the correct functioning of the test object.
Strong Features taking into consideration the real environment (e.g. target computer, compiler) and z testing the dynamic system behavior (e.g. run time behavior, memory space requirement) z
Weak Features z
an exhaustive test is usually impossible
test data selection by certain criteria
Wegener, Software Technology Research Dept., Slide 7
State of the Art Research and Technology
Test Activities Test Planning Specification
Program
Test Execution
Monitoring
Test Documentation
Test Organization
Test Case Design
Test Evaluation • most important for test quality • methods and tools needed Wegener, Software Technology Research Dept., Slide 8
State of the Art Research and Technology
Testing Methods Functional test: • test case design performed on the basis of the specification • most important test approach • common methods show insufficient systematics when partitioning the input domain, difficult procedure, or no efficient tool support • test size is hard to quantify (coverage of requirements, function points, ...) • difficult to automate • examples: equivalence partitioning, boundary value test, category-partition method Structural test: • test case design performed on the basis of the program structure • common test approach (included in many standards) • not possible to check whether all requirements have been implemented • difficult to automate (limits of symbolic execution) • examples: statement test, branch test, path test
Wegener, Software Technology Research Dept., Slide 9
State of the Art Research and Technology
Testing Methods Statistical Test: • test data generation in accordance with operational profile • operational profile hard to determine, especially for new systems • extensive test evaluation, if no test oracle available Random Test: • generation of random test data • specific test-relevant value combinations are produced only with little probability • extensive test evaluation, if no test oracle available Evolutionary Test: • generation of test data by means of search techniques • test objectives have to be transformed into an optimization problem Mutation Test: • test cases are build in order to kill mutants of original program • no procedure how to define the test cases
Wegener, Software Technology Research Dept., Slide 10
Classification-Tree Method Research and Technology
Functional Testing • lack of functional testing methods which are • efficient • easy to use in practice • suited for automation • lack of testing tools for black-box test case design
Classification-Tree Method efficient functional test method CTM: • systematic stepwise procedure which is easy to understand • graphical notation with compact representation of the over-all test • good test documentation • tool support with the CTE XL • proven in practical use
Wegener, Software Technology Research Dept., Slide 11
Classification-Tree Method CTM Research and Technology
Classification-Tree Method Test object: computer vision system recognizing the size of different building blocks large
Aspects Size
Color
Size
Shape
small large
Color red
green blue
Shape circle
triangle
square
Shape of triangle 1
equilateral isosceles scalene
2 Input Domain
3
Wegener, Software Technology Research Dept., Slide 12
Classification-Tree Method CTM Research and Technology
Classification-Tree Method
Size small
large
Color red
green
Shape blue
circle
triangle
square
Shape of triangle equilateral
isosceles
scalene
Wegener, Software Technology Research Dept., Slide 13
Classification-Tree Method CTM Research and Technology
Classification-Tree Method
Size small
large
Color red
green
Shape blue
circle
triangle
square
Shape of triangle 1
equilateral
isosceles
scalene
2 3
Wegener, Software Technology Research Dept., Slide 14
Classification-Tree Method CTM Research and Technology
Classification-Tree Method
Size small
large
Color red
green
Shape blue
circle
triangle
square
Shape of triangle equilateral
1
isosceles
scalene
2 3
Wegener, Software Technology Research Dept., Slide 15
Classification-Tree Method CTM Research and Technology
Classification-Tree Method Test object: computer vision system recognizing the size of different building blocks classification
large
class
Aspects Size
Color
Size
Shape
small large
Color red
green blue
Shape circle
triangle
square
Shape of triangle 1
equilateral isosceles scalene
2 Input Domain
3
Wegener, Software Technology Research Dept., Slide 16
Classification-Tree Method CTM Research and Technology
Classification-Tree Method Test object: computer vision system recognizing the size of different building blocks Minimality criterion: 5 testcases five disjoint classes for large shape
Aspects Size
Color
Size
Shape
small large
Color red
green blue
Shape circle
triangle
square
Shape of triangle equilateral isosceles scalene
1 2 3
Input Domain
Wegener, Software Technology Research Dept., Slide 17
Classification-Tree Method CTM Research and Technology
Classification-Tree Method Test object: computer vision system recognizing the size of different building blocks Maximality criterion: 30 testcases 2*3*5 large
Aspects Size
Color
Size
Shape
small large
Color red
green blue
Shape circle
triangle
square
Shape of triangle 1
equilateral isosceles scalene
2 Input Domain
3
Wegener, Software Technology Research Dept., Slide 18
Classification-Tree Editor CTE Research and Technology
Classification-Tree Editor CTE • syntax directed • context sensitive • graphical editor to support the systematic test case design by means of the classification-tree method Supporting • hierarchies to structure large classification trees and • hierarchies to structure sets of test cases • description of logical dependencies between classes of the classification-tree • combination rules for the generation of test cases according to logical dependencies and combination rules • sorting of test cases • comprehensive export functionalities • open API and architecture supporting the implementation of tool integrations such as with • DOORS • Test Director • Tessy • Time Partition Testing • Win Runner
Wegener, Software Technology Research Dept., Slide 19
Hierarchies for large trees Research and Technology
2. Test cases can be hierachically grouped: 1. Subtrees can be folded/unfolded at will:
folded subtree
groups of test cases
Wegener, Software Technology Research Dept., Slide 20
Implicit (Syntactical) Rules Research and Technology 1. Classes belonging to the same classification are disjoint (i.e. exclusive). e.g., red, green, and blue
2. If a class is marked all parent tree nodes are marked, too. e.g., equilateral implies triangle
‘triangle’ is implicitly marked disjoint classes
Wegener, Software Technology Research Dept., Slide 21
Usage of Logical Dependencies Research and Technology
1. Specification of impossible or required class combinations e.g., circle => NOT red
2. Verification of dependencies e.g., circle => NOT red
verification of logical dependencies
specification of logical dependencies red
circle
Wegener, Software Technology Research Dept., Slide 22
Combination Rules Research and Technology • express the desired combinatorics between classes and classifications • usage of combinatorial expressions
specification of combination rules
Wegener, Software Technology Research Dept., Slide 23
Automated Test Case Generation Research and Technology • based on combination rules and logical dependencies • algorithm based on heuristic combinatorial search
Wegener, Software Technology Research Dept., Slide 24
Classification-Tree Editor CTE Research and Technology
Real-World Example: Positioning
Wegener, Software Technology Research Dept., Slide 25
API Supporting Tool Integrations Research and Technology
Examples Tool Integration
Acceptance Test Vehicle Tests Vehicle Integration Specification
DOORS
TPT
System Test
Win Runner
MSC
TTCN3
dSPACE Automation Desk
System Test Integration Test Integration MTest
Unit Test
Tessy
Wegener, Software Technology Research Dept., Slide 26
Conclusion and Future Work Research and Technology
Summary • Classification-Tree Method enables systematic test case design for functional testing • successfully applied in practice to nearly all application domains • Efficient tool support by means of the CTE XL • available on www.systematic-testing.com • you can start using it • Many tool integrations available. Further integrations under development.
Future Work • Extensions for test management and product line approaches • Description of expected behavior • Generation rules for sub-trees • Metrics to assess test case design quality
Wegener, Software Technology Research Dept., Slide 27
Related Documents
Test Case Design Methods
November 2019 16
Test Case Design
November 2019 8
Test Case Design
July 2020 13
Guidelines Of Test Case Design
November 2019 14
Test Design
November 2019 18