Rup
This document was uploaded by user and they confirmed that they have the permission to share it. If you are author or own the copyright of this book, please report to us by using this DMCA report form. Report DMCA
Overview
Download & View Rup as PDF for free.
More details
- Words: 3,880
- Pages: 9
Vol. 4 Issue 3
-1-
www.testinginstitute.com
RUP: Implementing an Iterative Test Design Approach Steve Boycan, Managing Director, SIAC Yuri Chernak, Consultant, Valley Forge Consulting, Inc.
According to RUP, test design should iteratively evolve along with system requirements and design. Here’s how it can be implemented by combining features of the requirements-based and exploratory testing schools. When SIAC* projects began implementation of the Rational Unified Process (RUP), the main challenge for testers was to adjust the rigid requirements-based testing style to the iterative nature of the RUP framework. According to the RUP, software requirements and design iteratively evolve throughout the project lifecycle, and so it is expected that the test design would also do so. Hence, testers have to implement a test design approach that should be effective in the context of evolving software requirements. Although the RUP methodology is a comprehensive source of activities and guidelines that define the overall testing workflow, still it lacks some details about test design. In particular, we needed more details about a) how the test design steps and artifacts relate to other RUP disciplines, and b) how to implement the test design activities based on evolving requirements. To fill this gap, we developed an approach that we called – Iterative Test Design (ITD). In this article, we describe our approach that combines features of the requirements-based and exploratory testing schools. The main benefit of the ITD is that it provides a framework for test design that is effective in the context of incomplete or evolving requirements.
Two Schools of Testing When we refer to requirements-based testing, we usually mean a traditional school of formal testing that has been practiced for the last 20-30 years. Some sources that present this methodology could be the books by G. Myers1, B. Hetzel2, E. Kit3, just to name a few. The main concept of this methodology is that test design and test cases are derived from external functional requirements that are assumed to be complete and frozen before testers start using them for test design. And, the commonly cited issue with this approach is that the requirements-based testing is only as good as the requirements that are often incomplete, late, or not available at all. On projects that follow the Unified Process, testers also have an issue with this approach as requirements iteratively evolve and change during the course of a project. As an alternative to the requirements-based testing, the exploratory testing school emerged as a solution for projects that lack software requirements. Cem Kaner initially coined the term “exploratory testing” in his book4 published over ten years ago. Another *
SIAC – Securities Industry Automation Corporation
Vol. 4 Issue 3 -2www.testinginstitute.com testing expert James Bach has been evolving and teaching the exploratory testing as a systematic risk-based approach to testing5. The main concept of this methodology is that test design and test execution happen at the same time; and testers develop test ideas driven by exploration of the product’s quality risks. As we will discuss in this article, the features of both testing schools can be combined to help us effectively deal with incomplete or evolving requirements.
Iterative Test Design Approach Our approach to test design has the following main features: Iterative Test Design – test design iteratively evolves in a top-down fashion throughout the Plan-Design-Execute phases of the testing lifecycle. Use-Case-Driven Test Design – this means that use cases, defined for the system, are the primary source used for the high-level test design. Quality-Risk-Based Test Design – the approach is based on exploration of product quality risks existing in the context of use cases. The identified quality risks become testing objectives that guide testers in designing, prioritizing, and executing tests. Project-Context-Driven Test Design – this means that the amount and details of test documentation depend on a project’s context and management objectives. The ITD workflow is defined as the following four steps: Step 1 - Identify Generic Product Quality Risks (and develop a high-level approach to application testing) Step 2 - Identify Specific Quality Risks Existing in a Use Case Context (and develop ideas about what to test) Step 3 - Refine a Use Case Test Approach (and develop ideas about what and how to test) Step 4 - Manually Execute Tests to Explore the Product (and search for new test ideas) There is an important difference between the last step and the first three steps. In particular, while performing our test design activities during the first three steps, we explore product requirements; whereas at Step 4, we manually execute tests to explore the product itself. Test execution is sometimes viewed as a simple procedure that can be completely automated. Therefore, someone may have a question, “Why is test execution a part of test design?” According to the exploratory testing concept6, software testing is a challenging intellectual process that focuses on software product exploration. That, in turn, requires human observation and analysis that can never be replaced by automated scripts. As a result, in the course of manual test execution, testers can develop new test ideas and enhance their existing test design.
STEP 1 – Identify Generic Product Quality Risks ITD begins early in the process with defining an approach to application functional testing. The word approach implies that at this point we capture just high-level ideas about what needs to be tested and define preliminary steps for test execution. At Step 1 of our workflow, we do not know many details about the system implementation. However, based on our prior experience and having such sources as user requirements and business
Vol. 4 Issue 3 -3www.testinginstitute.com use cases, that are usually available early in the process, we can start developing ideas about what in general can go wrong with the system and what kind of failures it can experience in production. That is why we entitled this step – Identify Generic Product Quality Risks. By our definition, generic quality risks are particular product quality concerns that have a recurring context across the functional areas of a system under test. Some examples could be GUI features, data-entry field validation, record deletion rules, concurrency control, etc. Once we have identified generic quality risks, we then develop ideas about how to execute tests for the identified risks. Forms to capture the test logic could be as simple as a checklist of test conditions, or it could be a much more elaborate form, such as test design patterns. In general, patterns in software development are used to capture a solution to the issue that has a recurring context7. Hence, the test logic for generic quality risks can be presented by test design patterns. The deliverable of this step is a defined approach to application functional testing. The forms to present the test approach could be either a list of test ideas we developed at this step, or a section in a test plan document, if it is a required deliverable that follows the IEEE Std.829-1998 “IEEE Standard for Software Test Documentation”.
STEP 2 – Identify Specific Quality Risks Existing in a Use Case Context ITD is a use-case-driven approach. We can proceed to the next step as soon as drafts of use cases become available, which is, again, early in the process. At this step, we should explore software requirements to identify specific quality risks existing in the context of use cases. Using the word context we would like to emphasize that testers, developing test ideas, should look beyond what is written in the use-case description and analyze the whole situation presented in a given use case to understand all possible ways the system can fail. We begin our activities at this step with analyzing the use case description and decomposing it into a few control flows. For this purpose, we can use a use-case activity diagram, if it is available. Then we analyze each control flow to identify specific quality risks and develop ideas about what to test, i.e., what can go wrong if the user (or actor) performs all possible valid and/or invalid actions. While exploring requirements described in a use case, we should use the list of generic quality risks (our deliverable from Step 1) as a reference source that can suggest some test ideas that are not obvious from the use case description. Analyzing use cases, we can find that a particular kind of quality risks not initially included in the generic list is identified across a number of use cases; hence, it can also be qualified as a generic risk. In this case we should go back to Step 1 and update the list of generic quality risks. Finally, we define test objectives for the use case based on the identified specific quality risks. An important deliverable of this step is a draft of high-level test design specifications that capture our test ideas and test objectives for the use cases.
STEP 3 – Refine a Use Case Test Approach As the project progresses, more sources may become available to testers that provide additional details about the system implementation. For example, on our projects at SIAC
Vol. 4 Issue 3 -4www.testinginstitute.com these could be various Rational Rose diagrams, a user interface specification, storyboards that illustrate use cases, detailed Software Requirements Specification (SRS), and so on. Based on the additional specifications, we can refine the test approach for each use case and define a rationale for selecting test cases for already identified test objectives. This is the point where conventional black-box test design techniques, e.g. decision table, boundary-value analysis, equivalence partitioning, and so on can be used to identify positive and negative test cases. Once we have identified test cases, we can then develop ideas about how to execute these tests. As we learn more details about the system implementation, our understanding of quality risks can further evolve and we can identify additional test objectives for the use cases. In this case, we should go back to Step 2 and update objectives in our test designs. When we feel our test designs are fairly complete, we should review them with business experts and, based on their feedback, revise the documents, if necessary. Another good practice is to validate the test designs using a system prototype to make sure our documents captured correct test ideas. Deliverables of this step are high-level test design specifications (completed and reviewed at this point) and test case specifications, still in draft form. According to our approach, the high-level test design specifications are testers’ main deliverables. Even though this document type has been known for many years, for some reason testers do not use it as frequently as test case specifications. As we found in the context of complex projects with iterative development, this document type provides the following benefits. First, test design specifications are easy to review and verify as they present high-level ideas about “what” to test as opposed to test case specifications that are intended to capture detailed information about “how” to test. Second, being high-level documents, test design specifications are typically less affected by changing and evolving system requirements and design specifications as opposed to detailed test case specifications. Hence, their maintenance cost can be significantly lower. Third, from test design specifications, management can see the number of test cases to be developed and executed. These data, being available early in the process, can help management better allocate resources and schedule testing activities in the project plan. STEP 4 – Manually Execute Tests to Explore the Product At some point in the project lifecycle, developers feel that the build is completed and stable enough to be given to testers to do their part. At Step 4, which is the last step of our test design workflow, testers manually execute tests to further explore the product’s quality risks. When testers manually execute tests, their understanding of quality risks can significantly evolve. Hence, they can develop new test ideas and add test cases to the existing test designs. The number of additional or “exploratory” test cases will primarily depend on a) incompleteness of requirements used for the initial test design, and b) testers’ experience with exploratory testing. Deliverables of this step are high-level test design specifications, enhanced and validated, and detailed test case specifications, completed and verified. There is one more important activity of this step - analysis of software defects reported during test execution - that will be discussed in detail next. Defect Data Analysis During Test Execution Before test execution, our test design was based primarily on assumptions about product quality risks. However, during the test execution step, additional data become available to
Vol. 4 Issue 3 -5www.testinginstitute.com testers that reflect the areas of product instability. Indeed, software defects are a clear manifestation of the product’s implementation flaws. Therefore, by analyzing reported defects we can see how our initial assumptions about quality risks are actually realized in the course of test execution. By building a distribution of defects, for example, by generic quality risks or by functional areas of the application under test we can better see the areas of product instability and make a better decision about where the focus of test execution should be. When testers perform system testing as a team, another effective practice is analyzing each software defect reported by other testers. Seeing how other testers find defects and analyzing their test logic can help us develop additional test ideas and perform in-process improvement of our own test design. Static View: RUP Workflows and ITD Steps The RUP methodology is represented by a number of disciplines8, where each discipline groups related process activities and presents them as a logical workflow. Figure 1 shows four of the RUP disciplines – Business Modeling, Requirements, Analysis and Design, and Implementation – whose artifacts are primary sources for our iterative test design approach. Understanding how the ITD steps relate to other RUP disciplines is important because it can help us better plan and schedule test design activities. Also, as the software process iteratively evolves, deliverables of the four disciplines may change and affect the test design deliverables. Hence, understanding how the test design steps depend on other RUP workflows can help us perform impact analysis and keep the evolving project artifacts consistent.
RUP Disciplines
ITD Steps
ITD Deliverables
Business Modeling
Step 1
Approach to testing
Requirements
Step 2
High-level Test Designs (draft)
Analysis and Design
Step 3
• High-level Test Designs (completed / reviewed) • Test Case specs (draft)
Implementation
Step 4
• High-level Test Designs (enhanced / validated) • Test Case specifications (completed / verified)
Figure1. RUP Workflows and ITD Steps Dynamic View: Testing Lifecycle and ITD Activities Figure 2 shows an activity diagram that presents as a logical workflow of all activities necessary to implement our iterative test design approach. In particular, on this diagram we can see that from each consecutive step we always come back and revise or enhance test ideas that we developed at the previous step. Also, this diagram illustrates how the steps of the proposed approach correspond to the test process lifecycle defined by the RUP. As we can see, our activities are performed throughout the Plan-Design-Execute phases of the testing lifecycle, which illustrates a top-down nature of our approach.
Vol. 4 Issue 3 -6www.testinginstitute.com According to the RUP, project team members have particular roles for performing the software process activities. The test process roles are defined as Test Manager, Test Analyst, Test Designer, and Tester9. Different roles do not necessarily mean different people, but rather define various responsibilities that can be assigned either to the same person or to different team members. Figure 2 illustrates how these roles are involved in our approach implementation. By definition, Test Analyst is primarily responsible for generating test ideas. As the diagram shows, this role is involved at all steps of the approach workflow. This means, that the process of developing test ideas continues throughout the testing lifecycle. Another interesting point is that during test execution (Step 4 of ITD workflow) all three roles are involved in the test design activities. As we discussed, Test Analyst continues to explore the product’s quality risks and analyze the reported software defects in order to find new test ideas. On the other hand, Test Designer validates the test design specifications and updates them if necessary, while executing tests based on the existing test design. Also, if Test Analyst found a new test idea, Test Designer should enhance the existing test design based on the new test idea. Finally, Tester completes and verifies detailed test case specifications while executing tests to make sure they correctly reflect the test execution details. This is especially important when these documents are used for test automation or considered a product that we deliver with the system.
Vol. 4 Issue 3 Phases of testing lifecycle
Project team roles
-7Plan Test
www.testinginstitute.com
Design Test
Execute Test
Test Analyst Test Designer Tester
Figure 2. Testing Lifecycle and ITD Activities
ITD is a Project-Context-Driven Approach On critical projects the ITD approach can be compliant with the IEEE Std.829 by delivering two types of test design related documents – test design specification and test case specification. High-level test design specifications are a tester’s key deliverables that capture ideas about “what” to test. And detailed test case specifications capture the “how” details about test execution. If establishing traceability is one of the management objectives on a project, it can be implemented at different levels, for example, from a use case to a test design specification (this will be a case of 1:1 relationship), or from a test design specification to the corresponding test case specifications (this will be a case of 1: M relationship). Our test design approach is project-context-driven. This means that our decisions about whether we need detailed test case specifications and what level of details is appropriate for test design specifications should depend on a project’s context and management objectives. In general, before making a decision about how much to invest in designing and maintaining test documentation, we should clarify requirements for the test documentation by answering such questions as “What is the purpose of test documentation?” and “Who and how will be using test documentation?” The same type of test documentation can have different details for different purposes. For example, it could be a product to be delivered with a system that should comply with contractual requirements; or it could be a tool for testers that should help them execute testing in a structured way; or test documentation could be a tool for management that should help them better plan and control the project. Therefore, as any other deliverable on a project, test documentation should have defined requirements and design guidelines tailored to a given project context. The book6 “Lessons Learned in Software Testing” has a detailed discussion of this topic.
Vol. 4 Issue 3
-8-
www.testinginstitute.com
ITD Implementation Challenges While implementing iterative test design on our projects, we faced a few challenges that apparently may be common to any project that follows this approach. The first issue is concerned with a paradigm shift. Transition to the exploratory way of testing could be challenging for testers who are mostly experienced with the requirements-based testing. To resolve this issue, management should plan training for testers and test design reviews. The second issue can be best described by a question - “What to Test Next?” Under time pressure during test execution, if a tester finds a new test idea, it could be a challenge for the tester to decide what to test next – either develop and execute new test cases or keep going with the original plan. This issue relates to the next one - planning a test schedule. Planning the test schedule, a test manager should take into account the additional number of test cases that testers will identify during test execution. This can also resolve the previous issue and leave testers enough time to explore new test ideas that can be found during test execution. However, we do not know in advance how many new test cases will be found. The test manager should make an assessment based on a given project context and the experience of previous projects. The last issue we would like to address relates to test documentation management. Commercial test management tools may not directly support a two-level hierarchy of documents - high-level test designs and detailed test case specifications. This issue can be resolved in a number of ways. For example, we can use a commercial test management tool only as a repository of test case specifications, and the test design specifications can be created as external Word documents that are associated with the test cases in the repository. The alternative approach could be to develop and use a customized test management tool that can better fit particular project needs as opposed to commercial tools.
Iterative Test Design Benefits Concluding this article we would like to highlight the main benefits that the approach can provide for a project team: ITD provides testers with a framework for product exploration and effective test design when requirements are incomplete or evolving. ITD is defined based on a top-down workflow that helps testers better cope with the functional complexity of a system under test. High-level test design specifications, the main deliverable of the ITD approach, are easy to review and maintain. By delivering test designs early in the process, the approach provides management with better visibility into the expected cost and effectiveness of functional testing.
Biography of Yuri Chernak Yuri Chernak is the president and principal consultant of Valley Forge Consulting, Inc. As a consultant, Yuri has worked for a number of major financial firms in New York helping senior management improve their software test process. Yuri has a Ph.D. in
Vol. 4 Issue 3 -9www.testinginstitute.com computer science and he is a member of the IEEE Computer Society. He has been a speaker at several international conferences and has published papers on software testing in the IEEE journals. Contact him by email: [email protected].
Biography of Steve Boycan Steve Boycan has been leading software engineering and process improvement efforts in the military, telecommunications, and financial sectors for 12 years. He currently manages several testing related functions for critical NYSE systems. He also manages groups that provide management support tools and automated test infrastructure development. Process improvement and software project measurement are key areas of interest. Acknowledgement This work was supported by the Application Scripting Group (ASG) at SIAC, and the authors are grateful to the ASG testers for their feedback. References 1 2 3 4 5 6
7
8 9
G. Myers "The Art of Software Testing", John Wiley & Sons, New York, 1979 B. Hetzel “The Complete Guide to Software Testing”, John Wiley & Sons, 1988 E. Kit "Software Testing in the Real World", Addison-Wesley, 1995 C. Kaner “Testing Computer Software”, International Thomson Computer Press, 1988 J. Bach “Risk-Based Testing”, STQE Magazine, November 1999, pp.22-29 C. Kaner, J. Bach, B. Pettichord “Lessons Learned in Software Testing”, John Wiley & Sons, 2002 E. Gamma, et al. “Design Patterns. Elements of Reusable Object-Oriented Software”, Addison-Wesley, 1995 P. Kruchten “The Rational Unified Process. An Introduction”, Addison-Wesley, 2000 P. Kroll, P. Kruchten “The rational Unified Process Made Easy. A Practitioner’s Guide the RUP”, Addison-Wesley, 2003
-1-
www.testinginstitute.com
RUP: Implementing an Iterative Test Design Approach Steve Boycan, Managing Director, SIAC Yuri Chernak, Consultant, Valley Forge Consulting, Inc.
According to RUP, test design should iteratively evolve along with system requirements and design. Here’s how it can be implemented by combining features of the requirements-based and exploratory testing schools. When SIAC* projects began implementation of the Rational Unified Process (RUP), the main challenge for testers was to adjust the rigid requirements-based testing style to the iterative nature of the RUP framework. According to the RUP, software requirements and design iteratively evolve throughout the project lifecycle, and so it is expected that the test design would also do so. Hence, testers have to implement a test design approach that should be effective in the context of evolving software requirements. Although the RUP methodology is a comprehensive source of activities and guidelines that define the overall testing workflow, still it lacks some details about test design. In particular, we needed more details about a) how the test design steps and artifacts relate to other RUP disciplines, and b) how to implement the test design activities based on evolving requirements. To fill this gap, we developed an approach that we called – Iterative Test Design (ITD). In this article, we describe our approach that combines features of the requirements-based and exploratory testing schools. The main benefit of the ITD is that it provides a framework for test design that is effective in the context of incomplete or evolving requirements.
Two Schools of Testing When we refer to requirements-based testing, we usually mean a traditional school of formal testing that has been practiced for the last 20-30 years. Some sources that present this methodology could be the books by G. Myers1, B. Hetzel2, E. Kit3, just to name a few. The main concept of this methodology is that test design and test cases are derived from external functional requirements that are assumed to be complete and frozen before testers start using them for test design. And, the commonly cited issue with this approach is that the requirements-based testing is only as good as the requirements that are often incomplete, late, or not available at all. On projects that follow the Unified Process, testers also have an issue with this approach as requirements iteratively evolve and change during the course of a project. As an alternative to the requirements-based testing, the exploratory testing school emerged as a solution for projects that lack software requirements. Cem Kaner initially coined the term “exploratory testing” in his book4 published over ten years ago. Another *
SIAC – Securities Industry Automation Corporation
Vol. 4 Issue 3 -2www.testinginstitute.com testing expert James Bach has been evolving and teaching the exploratory testing as a systematic risk-based approach to testing5. The main concept of this methodology is that test design and test execution happen at the same time; and testers develop test ideas driven by exploration of the product’s quality risks. As we will discuss in this article, the features of both testing schools can be combined to help us effectively deal with incomplete or evolving requirements.
Iterative Test Design Approach Our approach to test design has the following main features: Iterative Test Design – test design iteratively evolves in a top-down fashion throughout the Plan-Design-Execute phases of the testing lifecycle. Use-Case-Driven Test Design – this means that use cases, defined for the system, are the primary source used for the high-level test design. Quality-Risk-Based Test Design – the approach is based on exploration of product quality risks existing in the context of use cases. The identified quality risks become testing objectives that guide testers in designing, prioritizing, and executing tests. Project-Context-Driven Test Design – this means that the amount and details of test documentation depend on a project’s context and management objectives. The ITD workflow is defined as the following four steps: Step 1 - Identify Generic Product Quality Risks (and develop a high-level approach to application testing) Step 2 - Identify Specific Quality Risks Existing in a Use Case Context (and develop ideas about what to test) Step 3 - Refine a Use Case Test Approach (and develop ideas about what and how to test) Step 4 - Manually Execute Tests to Explore the Product (and search for new test ideas) There is an important difference between the last step and the first three steps. In particular, while performing our test design activities during the first three steps, we explore product requirements; whereas at Step 4, we manually execute tests to explore the product itself. Test execution is sometimes viewed as a simple procedure that can be completely automated. Therefore, someone may have a question, “Why is test execution a part of test design?” According to the exploratory testing concept6, software testing is a challenging intellectual process that focuses on software product exploration. That, in turn, requires human observation and analysis that can never be replaced by automated scripts. As a result, in the course of manual test execution, testers can develop new test ideas and enhance their existing test design.
STEP 1 – Identify Generic Product Quality Risks ITD begins early in the process with defining an approach to application functional testing. The word approach implies that at this point we capture just high-level ideas about what needs to be tested and define preliminary steps for test execution. At Step 1 of our workflow, we do not know many details about the system implementation. However, based on our prior experience and having such sources as user requirements and business
Vol. 4 Issue 3 -3www.testinginstitute.com use cases, that are usually available early in the process, we can start developing ideas about what in general can go wrong with the system and what kind of failures it can experience in production. That is why we entitled this step – Identify Generic Product Quality Risks. By our definition, generic quality risks are particular product quality concerns that have a recurring context across the functional areas of a system under test. Some examples could be GUI features, data-entry field validation, record deletion rules, concurrency control, etc. Once we have identified generic quality risks, we then develop ideas about how to execute tests for the identified risks. Forms to capture the test logic could be as simple as a checklist of test conditions, or it could be a much more elaborate form, such as test design patterns. In general, patterns in software development are used to capture a solution to the issue that has a recurring context7. Hence, the test logic for generic quality risks can be presented by test design patterns. The deliverable of this step is a defined approach to application functional testing. The forms to present the test approach could be either a list of test ideas we developed at this step, or a section in a test plan document, if it is a required deliverable that follows the IEEE Std.829-1998 “IEEE Standard for Software Test Documentation”.
STEP 2 – Identify Specific Quality Risks Existing in a Use Case Context ITD is a use-case-driven approach. We can proceed to the next step as soon as drafts of use cases become available, which is, again, early in the process. At this step, we should explore software requirements to identify specific quality risks existing in the context of use cases. Using the word context we would like to emphasize that testers, developing test ideas, should look beyond what is written in the use-case description and analyze the whole situation presented in a given use case to understand all possible ways the system can fail. We begin our activities at this step with analyzing the use case description and decomposing it into a few control flows. For this purpose, we can use a use-case activity diagram, if it is available. Then we analyze each control flow to identify specific quality risks and develop ideas about what to test, i.e., what can go wrong if the user (or actor) performs all possible valid and/or invalid actions. While exploring requirements described in a use case, we should use the list of generic quality risks (our deliverable from Step 1) as a reference source that can suggest some test ideas that are not obvious from the use case description. Analyzing use cases, we can find that a particular kind of quality risks not initially included in the generic list is identified across a number of use cases; hence, it can also be qualified as a generic risk. In this case we should go back to Step 1 and update the list of generic quality risks. Finally, we define test objectives for the use case based on the identified specific quality risks. An important deliverable of this step is a draft of high-level test design specifications that capture our test ideas and test objectives for the use cases.
STEP 3 – Refine a Use Case Test Approach As the project progresses, more sources may become available to testers that provide additional details about the system implementation. For example, on our projects at SIAC
Vol. 4 Issue 3 -4www.testinginstitute.com these could be various Rational Rose diagrams, a user interface specification, storyboards that illustrate use cases, detailed Software Requirements Specification (SRS), and so on. Based on the additional specifications, we can refine the test approach for each use case and define a rationale for selecting test cases for already identified test objectives. This is the point where conventional black-box test design techniques, e.g. decision table, boundary-value analysis, equivalence partitioning, and so on can be used to identify positive and negative test cases. Once we have identified test cases, we can then develop ideas about how to execute these tests. As we learn more details about the system implementation, our understanding of quality risks can further evolve and we can identify additional test objectives for the use cases. In this case, we should go back to Step 2 and update objectives in our test designs. When we feel our test designs are fairly complete, we should review them with business experts and, based on their feedback, revise the documents, if necessary. Another good practice is to validate the test designs using a system prototype to make sure our documents captured correct test ideas. Deliverables of this step are high-level test design specifications (completed and reviewed at this point) and test case specifications, still in draft form. According to our approach, the high-level test design specifications are testers’ main deliverables. Even though this document type has been known for many years, for some reason testers do not use it as frequently as test case specifications. As we found in the context of complex projects with iterative development, this document type provides the following benefits. First, test design specifications are easy to review and verify as they present high-level ideas about “what” to test as opposed to test case specifications that are intended to capture detailed information about “how” to test. Second, being high-level documents, test design specifications are typically less affected by changing and evolving system requirements and design specifications as opposed to detailed test case specifications. Hence, their maintenance cost can be significantly lower. Third, from test design specifications, management can see the number of test cases to be developed and executed. These data, being available early in the process, can help management better allocate resources and schedule testing activities in the project plan. STEP 4 – Manually Execute Tests to Explore the Product At some point in the project lifecycle, developers feel that the build is completed and stable enough to be given to testers to do their part. At Step 4, which is the last step of our test design workflow, testers manually execute tests to further explore the product’s quality risks. When testers manually execute tests, their understanding of quality risks can significantly evolve. Hence, they can develop new test ideas and add test cases to the existing test designs. The number of additional or “exploratory” test cases will primarily depend on a) incompleteness of requirements used for the initial test design, and b) testers’ experience with exploratory testing. Deliverables of this step are high-level test design specifications, enhanced and validated, and detailed test case specifications, completed and verified. There is one more important activity of this step - analysis of software defects reported during test execution - that will be discussed in detail next. Defect Data Analysis During Test Execution Before test execution, our test design was based primarily on assumptions about product quality risks. However, during the test execution step, additional data become available to
Vol. 4 Issue 3 -5www.testinginstitute.com testers that reflect the areas of product instability. Indeed, software defects are a clear manifestation of the product’s implementation flaws. Therefore, by analyzing reported defects we can see how our initial assumptions about quality risks are actually realized in the course of test execution. By building a distribution of defects, for example, by generic quality risks or by functional areas of the application under test we can better see the areas of product instability and make a better decision about where the focus of test execution should be. When testers perform system testing as a team, another effective practice is analyzing each software defect reported by other testers. Seeing how other testers find defects and analyzing their test logic can help us develop additional test ideas and perform in-process improvement of our own test design. Static View: RUP Workflows and ITD Steps The RUP methodology is represented by a number of disciplines8, where each discipline groups related process activities and presents them as a logical workflow. Figure 1 shows four of the RUP disciplines – Business Modeling, Requirements, Analysis and Design, and Implementation – whose artifacts are primary sources for our iterative test design approach. Understanding how the ITD steps relate to other RUP disciplines is important because it can help us better plan and schedule test design activities. Also, as the software process iteratively evolves, deliverables of the four disciplines may change and affect the test design deliverables. Hence, understanding how the test design steps depend on other RUP workflows can help us perform impact analysis and keep the evolving project artifacts consistent.
RUP Disciplines
ITD Steps
ITD Deliverables
Business Modeling
Step 1
Approach to testing
Requirements
Step 2
High-level Test Designs (draft)
Analysis and Design
Step 3
• High-level Test Designs (completed / reviewed) • Test Case specs (draft)
Implementation
Step 4
• High-level Test Designs (enhanced / validated) • Test Case specifications (completed / verified)
Figure1. RUP Workflows and ITD Steps Dynamic View: Testing Lifecycle and ITD Activities Figure 2 shows an activity diagram that presents as a logical workflow of all activities necessary to implement our iterative test design approach. In particular, on this diagram we can see that from each consecutive step we always come back and revise or enhance test ideas that we developed at the previous step. Also, this diagram illustrates how the steps of the proposed approach correspond to the test process lifecycle defined by the RUP. As we can see, our activities are performed throughout the Plan-Design-Execute phases of the testing lifecycle, which illustrates a top-down nature of our approach.
Vol. 4 Issue 3 -6www.testinginstitute.com According to the RUP, project team members have particular roles for performing the software process activities. The test process roles are defined as Test Manager, Test Analyst, Test Designer, and Tester9. Different roles do not necessarily mean different people, but rather define various responsibilities that can be assigned either to the same person or to different team members. Figure 2 illustrates how these roles are involved in our approach implementation. By definition, Test Analyst is primarily responsible for generating test ideas. As the diagram shows, this role is involved at all steps of the approach workflow. This means, that the process of developing test ideas continues throughout the testing lifecycle. Another interesting point is that during test execution (Step 4 of ITD workflow) all three roles are involved in the test design activities. As we discussed, Test Analyst continues to explore the product’s quality risks and analyze the reported software defects in order to find new test ideas. On the other hand, Test Designer validates the test design specifications and updates them if necessary, while executing tests based on the existing test design. Also, if Test Analyst found a new test idea, Test Designer should enhance the existing test design based on the new test idea. Finally, Tester completes and verifies detailed test case specifications while executing tests to make sure they correctly reflect the test execution details. This is especially important when these documents are used for test automation or considered a product that we deliver with the system.
Vol. 4 Issue 3 Phases of testing lifecycle
Project team roles
-7Plan Test
www.testinginstitute.com
Design Test
Execute Test
Test Analyst Test Designer Tester
Figure 2. Testing Lifecycle and ITD Activities
ITD is a Project-Context-Driven Approach On critical projects the ITD approach can be compliant with the IEEE Std.829 by delivering two types of test design related documents – test design specification and test case specification. High-level test design specifications are a tester’s key deliverables that capture ideas about “what” to test. And detailed test case specifications capture the “how” details about test execution. If establishing traceability is one of the management objectives on a project, it can be implemented at different levels, for example, from a use case to a test design specification (this will be a case of 1:1 relationship), or from a test design specification to the corresponding test case specifications (this will be a case of 1: M relationship). Our test design approach is project-context-driven. This means that our decisions about whether we need detailed test case specifications and what level of details is appropriate for test design specifications should depend on a project’s context and management objectives. In general, before making a decision about how much to invest in designing and maintaining test documentation, we should clarify requirements for the test documentation by answering such questions as “What is the purpose of test documentation?” and “Who and how will be using test documentation?” The same type of test documentation can have different details for different purposes. For example, it could be a product to be delivered with a system that should comply with contractual requirements; or it could be a tool for testers that should help them execute testing in a structured way; or test documentation could be a tool for management that should help them better plan and control the project. Therefore, as any other deliverable on a project, test documentation should have defined requirements and design guidelines tailored to a given project context. The book6 “Lessons Learned in Software Testing” has a detailed discussion of this topic.
Vol. 4 Issue 3
-8-
www.testinginstitute.com
ITD Implementation Challenges While implementing iterative test design on our projects, we faced a few challenges that apparently may be common to any project that follows this approach. The first issue is concerned with a paradigm shift. Transition to the exploratory way of testing could be challenging for testers who are mostly experienced with the requirements-based testing. To resolve this issue, management should plan training for testers and test design reviews. The second issue can be best described by a question - “What to Test Next?” Under time pressure during test execution, if a tester finds a new test idea, it could be a challenge for the tester to decide what to test next – either develop and execute new test cases or keep going with the original plan. This issue relates to the next one - planning a test schedule. Planning the test schedule, a test manager should take into account the additional number of test cases that testers will identify during test execution. This can also resolve the previous issue and leave testers enough time to explore new test ideas that can be found during test execution. However, we do not know in advance how many new test cases will be found. The test manager should make an assessment based on a given project context and the experience of previous projects. The last issue we would like to address relates to test documentation management. Commercial test management tools may not directly support a two-level hierarchy of documents - high-level test designs and detailed test case specifications. This issue can be resolved in a number of ways. For example, we can use a commercial test management tool only as a repository of test case specifications, and the test design specifications can be created as external Word documents that are associated with the test cases in the repository. The alternative approach could be to develop and use a customized test management tool that can better fit particular project needs as opposed to commercial tools.
Iterative Test Design Benefits Concluding this article we would like to highlight the main benefits that the approach can provide for a project team: ITD provides testers with a framework for product exploration and effective test design when requirements are incomplete or evolving. ITD is defined based on a top-down workflow that helps testers better cope with the functional complexity of a system under test. High-level test design specifications, the main deliverable of the ITD approach, are easy to review and maintain. By delivering test designs early in the process, the approach provides management with better visibility into the expected cost and effectiveness of functional testing.
Biography of Yuri Chernak Yuri Chernak is the president and principal consultant of Valley Forge Consulting, Inc. As a consultant, Yuri has worked for a number of major financial firms in New York helping senior management improve their software test process. Yuri has a Ph.D. in
Vol. 4 Issue 3 -9www.testinginstitute.com computer science and he is a member of the IEEE Computer Society. He has been a speaker at several international conferences and has published papers on software testing in the IEEE journals. Contact him by email: [email protected].
Biography of Steve Boycan Steve Boycan has been leading software engineering and process improvement efforts in the military, telecommunications, and financial sectors for 12 years. He currently manages several testing related functions for critical NYSE systems. He also manages groups that provide management support tools and automated test infrastructure development. Process improvement and software project measurement are key areas of interest. Acknowledgement This work was supported by the Application Scripting Group (ASG) at SIAC, and the authors are grateful to the ASG testers for their feedback. References 1 2 3 4 5 6
7
8 9
G. Myers "The Art of Software Testing", John Wiley & Sons, New York, 1979 B. Hetzel “The Complete Guide to Software Testing”, John Wiley & Sons, 1988 E. Kit "Software Testing in the Real World", Addison-Wesley, 1995 C. Kaner “Testing Computer Software”, International Thomson Computer Press, 1988 J. Bach “Risk-Based Testing”, STQE Magazine, November 1999, pp.22-29 C. Kaner, J. Bach, B. Pettichord “Lessons Learned in Software Testing”, John Wiley & Sons, 2002 E. Gamma, et al. “Design Patterns. Elements of Reusable Object-Oriented Software”, Addison-Wesley, 1995 P. Kruchten “The Rational Unified Process. An Introduction”, Addison-Wesley, 2000 P. Kroll, P. Kruchten “The rational Unified Process Made Easy. A Practitioner’s Guide the RUP”, Addison-Wesley, 2003
