Regression Testing

A Strategy for Designing and Executing an Effective Regression Test Program 1. Introduction This paper focuses on the area of regression testing. An industry-wide definition of regression test is “retesting of a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made.” We have adapted this slightly for the type of testing we perform to give the following definition: “a test designed to confirm that an existing feature still performs correctly after any software modifications.” Both these definitions provide the reader with a sense of the importance of regression testing. It is of no benefit providing your valued customer with lots of new and exciting features if you manage to break features they have been using successfully for the last number of years. In fact, some feedback received from a customer several years back indicated that they expect to have problems with new features when they first get a new release of software, but

September/December 2001

that they cannot tolerate existing features not working on the new release. So, regression testing is recognised as being very important, yet it often remains the test phase given the least attention in the planning of test activities. Too often, organizations take the easy way out and regression test phases simply consist of reruns of existing test suites in the hope that these will catch all the problems. This approach has a number of potential flaws:

new features or other code changes are being added. • Although you may build up a library of regression tests over time, and this library can be quite extensive, they can also prove troublesome as they contain both redundant test suites & test cases which themselves contain errors. There can be a huge overhead in maintaining this library of regression suites and test cases.

• If you are dealing with a large complex system, you can end up with thousands upon thousands of regression tests as you continue to add feature test suites from previous releases to the regression program for your current release. This is all well and good if you have a fully automated and stable test environment, which allows you to run all these tests quickly without using a lot of resources.

So, an unplanned approach to regression testing can lead to inefficient use of resources. If you take the time to plan your regression test program, there are a number of questions you need to ask:

• You may not have the required coverage with your existing test suites to find problems being introduced as

The first question on this list is the most important one to answer, as it will help answer the remaining questions. Apart

http://www.testinginstitute.com

• What do I need to test? • How long is it going to take? • What resources are required? • What tools are required? • Are there any other special requirements?

Journal of Software Testing Professionals

1

100% sure that we have full coverage, as the method of selecting the regression tests was purely to try and cover the basic functionality of the MSC.

from being the most important question it is also the most difficult to answer. 2. Understanding Our Problem We are involved in sub-system and system testing of software releases for a Mobile Switching Centre (MSC). We test in both an MSC-only configuration and also in a full cellular system configuration with other network elements. Over the past 3 years the complexity of the features being developed for the MSC has increased tremendously as we move away from MSC-only features to features that span several network elements. Our regression testing is a mixture of automated (30%), semi-automated (30%) and manual (40%). On our current release of MSC software, we decided to spend some time analysing the way we approached our regression testing. In general we had been taking an ad-hoc approach to planning our regression test program for previous releases and our analysis uncovered the following facts: • We are dealing with a legacy system; there has been a lack of consistency in the way features developed over the years have been documented and controlled. This has led to a lack of traceability from existing test cases/suites to old feature/functional requirements. • Consistently on all our releases, our Problem Report (PR) arrival rate showed a straight-line increase with no levelling off towards the end of our cycle as shown in Figure 1. A ‘normal’ PR arrival rate is one where you are finding the majority of your problems early in your cycle, thus

September/December 2001

We have been performing Escaped Defect Analysis & Root Cause Analysis on problems that escape to our customers, and this showed us a number of things in relation to our regression testing: allowing them to be fixed early. This then reduces the risk of impacting quality with fixes having to be made to the release late in the cycle prior to customer deployment.

• We were good at testing new feature and enhancements but problems escaped in other areas that should have been covered by our regression testing.

• We had an increasing number of regression tests per release, as we simply added new tests to cover new features developed on the previous release we had tested. This increasing number of tests had turned our regression cycle into a 12-week interval on average and this was increasing with every release. In fact, over a 6-year period our regression cycle had increase from 6 weeks to the current average of 12 weeks.

• By simply re-running existing regression suites, we could not hope to catch most of the problems that were escaping. Regression suites often covered the basic functionality of a particular feature, so if a code change resulted in an impact outside of this basic functionality, we did not have the required coverage to catch the problem.

• There was a lot of frustration among test engineers, as they were simply running regression tests for no apparent reason except we did it on the last release so must do it again! More often or not, these tests would not even find any problems, so the frustration increased. • Over the years we have built up an extensive library of regression tests through the addition of tests to cover new functionality. Even with this library of tests, we can never be

http://www.testinginstitute.com

• Quite often, the solution to the Escaped Defect Analysis was just to add a test to provide coverage for the problem that escaped, thus increasing the number of regression tests even further. Clearly it would be better to try and detect the problem in the first place rather than adding a test just in case it re-occurred. 3. A Time For Change Once we had done the analysis, it was time for some change. Quite simply we could not continue the way we were going.

Journal of Software Testing Professionals

29

Related impacts can be considered to be of lower priority and tests for these features can be included in the Secondary phase (refer to Section 4.). Where there are no impacts, the cell can be left blank.

Due to the effort involved for older features and functionality, we could not go back and resolve the problem of poorly documented and controlled requirements and the lack of traceability, so we had to work around this in some way. Why did our typical problem arrival rate show a straight line for all our releases? Some of it was due to churn on the release, i.e., features being added late in our cycle causing new problems to be introduced and found late in our cycle or sometimes not at all. However, the major finding was that we simply had no strategy in terms of the way we executed our regression tests—engineers simply took their suites, went into our labs and started executing. We decided that we needed a way of prioritising our tests and then to execute based on the priority. While looking at the issue of the increasing number of tests per release, it was clear that we had no regression selection criteria, except that we were trying to show that existing features and functionality still worked. We decided we needed some way of reducing the amount of tests we were running by identifying the tests that were actually necessary to run. When testing new features you have requirements from which to work from in developing tests and have a methodology that can be followed to create these tests. For regression we had nothing, so there was a need to create some form of requirements, which we could then use to identify the required regression testing. To do this we needed specific information from the development organization on the changes they were implementing in

the software, both for new features/ enhancements and any other changes they were making (problem fixes, reengineering work etc.). We used 2 ways of collecting this information: 3.1 A Regression Impacts Matrix. We created a simple Regression Impacts Matrix that lists all existing features and areas of functionality. Each development team working on a new feature or enhancement was then asked to complete the matrix indicating the features and functionality both directly and indirectly impacted by their code changes. Table 1 shows a sample Regression Impacts Matrix. It is quite simple in format and lists all existing features and functionality along the top row with all new features on the release being listed in the first column. Where a new feature has an explicit impact on an existing feature, an ‘X’ is used to indicate the impact. Where the impact is not explicit, an ‘R’ is used to indicate a related impact.

30 Journal of Software Testing Professionals

http://www.testinginstitute.com

3.2 A Test Impacts Checklist We used an existing Test Impacts Checklist. The development engineers are required to complete this checklist for every submission of code they make to the software release. The checklist asks specific questions that the developer has to answer, and the test engineers can then use this information to identify regression impacts due to code changes not associated with a new feature or enhancement (i.e. fixes, re-engineering). In our case, the checklist was broken down into several areas as follows: Area of Impact: We were able to identify 8 main functional areas (including call-processing, Billing, Fault Management & Statistics), and the developer first has to indicate which area was being impacted by their code change. This will then give the testers an idea of the type of regression test they might require. Description of Impact: They are then asked to describe the impact in their own words. This information helps further define the regression test that might be required. Call Scenarios: Because a lot of our testing involves call processing, developers are asked to identify specific callscenarios that are impacted. Example scenarios include: three-party conferencing, call forwarding, and hand-offs. Knowing the call-scenarios impacted allows the tester to select these scenarios from our existing suites of regres-

September/December 2001

sion tests if they exist, or be in a position to create new regression tests if required. Area of Code Change: Developers are asked whether the code change is in a frequently executed leg of code, i.e., would every call-scenario be affected. This gives testers an idea of the likely complexity of the required regression test, as a code change in a less frequently executed leg of code may require additional effort to create the regression test. Recommended Regression Test: The developer is asked to recommend what regression testing would be required to exercise both the code change itself and also functionality around the code change. Depending on the developer’s experience, this information can again be very useful to the tester. Features Impacted by Code Change: Similar to the Regression Impacts Matrix, the developer must indicate what features are impacted by the code change. This will allow the tester to cross-check this with the Regression Impacts Matrix and they can determine if additional feature-specific regression testing is required. External Message Changes: As we deal with a product that interfaces to a large number of other products, there can be external message changes involved in the code change. If so, this can often add complexity to the regression testing required, as it can either mean that: a) Another product will also need to be modified and we may need that product modification to fully regression test, or b) We will need to make modifications to our test tool in order to update it to include the new external message change.

September/December 2001

By using all the information provided by both the checklist and the matrix, test engineers can then decide on the regression testing required and allowed them to identify any required test environment changes. This gives them a sense of ownership and purpose, and a chance to help design the regression program and make decisions themselves rather than being simply told what regression suites to execute.

will increase the possibility of detecting problems due to those impacts. These suites would also cover overall basic feature functionality thus giving a high degree of confidence in the overall quality of the software release.

4. The Regression Strategy A strategy was needed to pull everything together so that:

• Suites that have repeatedly found problems on previous releases – this information was readily available from our Final Test Reports for previous releases.

a) We were consistent in our approach, b) It was repeatable, and c) It led to focused regression testing taking place. We came up with the following approach based on grouping our suites into 3 main types and then executing our regression tests in 3 phases: 4.1 Primary The Primary group consists of suites and test cases that are selected for execution based on the following criteria: • Providing coverage for areas directly impacted by new features and other known impacted code changes – all the information provided in the Regression Impacts Matrix and Test Impacts Checklist is used to identify either the individual regression tests or suites of regression tests required to test the impacted areas. • A small subset of tests covering basic functionality for all features – here we simply select smaller subsets of suites from our regression library. It is much easier to then maintain this smaller subset. The idea behind identifying Primary test suites is that these will focus on the impacted areas of the software and so

http://www.testinginstitute.com

4.2 Secondary The Secondary group consists of suites and test cases that are selected for execution based on the following criteria:

• Suites that cover customer-sensitive functionality such as billing and statistics. 4.3 Tertiary The Tertiary group consists of suites and test cases that are selected as follows: • Hot-spot suites – these are selected based on problems found by earlier Primary and Secondary testing, i.e. you might wish to increase test coverage due to large number of problems found in specific areas. • Additional testing due to late code additions – once again we would use the information provided in the Test Impacts Checklist to identify the tests required. • Clean-run/Sanity testing prior to deployment of software release to customers. Primary and Secondary phase testing can be identified early in the release cycle, and can therefore be planned for. Tertiary tests are more ad-hoc and unplanned, but we allow time for them in the schedule when Primary and Secondary testing is complete. We can

Journal of Software Testing Professionals

31

then plan and report on their status on a short-term basis. 5. The Benefits This new approach to regression testing has just been used on our current release of software, and we have seen the following benefits: • We have reduced the number of tests in our regression program to 2000 from a figure of approximately 6000 on previous releases. While this is a large decrease in the number of tests being executed, we were confident that there would not be an impact to quality as the strategy allowed us to focus on covering all basic functionality, plus the areas where changes were being made. This was clearly a big improvement on simply rerunning tests from previous releases without any sound reason. • Our problem arrival rate was less linear - Figure 2 shows our Current Release Problem Report Arrival Rate. For this release we had a total regression test interval of 28 weeks, and the graph shows that we opened 30 PRs during the first half of this interval. This is 65% of the total number of PRs we opened during regression test. This compares to approximately 30% of PRs being opened during the first half of our interval on previous releases. This difference can be attributed to testing of impacted areas earlier in our cycle. • Test engineers have increased satisfaction because they see a purpose to what they are doing as they are actively involved in designing the regression program. They also get a chance to learn more about the feature and functionality they are testing, as they are not executing a suite simply because they were told to do it.

• We are focusing on high-risk areas earlier, increasing the chances of finding the more serious problems earlier in the cycle. • The strategy is flexible and there is scope for additional testing if required. • There is an increased level of awareness from the development community with regards to the impacts their code changes can have. We have also increased their interaction with testers in helping to define the regression program. • It had led to improved maintenance of existing regression suites given that impacts and feature interactions are identified earlier and changes/ updates to suites can then be planned during the test development phase of the release. The current release has just been deployed on a customer site for the first time. When we compare the level of escaped defects during this deployment phase to the previous release, we see a reduction in the amount of problems found, from 21 to 8. This reduction cannot however be directly attributed to our new regression strategy as other defect reduction activities have also

32 Journal of Software Testing Professionals

http://www.testinginstitute.com

taken place on this release earlier in the lifecycle. These activities focused on improvements to the formal inspection process including: improved inspection checklists and training, along with regular analysis of inspection data, leading to reinspection of out-of-bound inspections. 6. Issues While we have seen many benefits of this new strategy, there are some issues we have noted: • Depending on the number of test engineers available, it can be difficult to schedule execution of the test suites in each phase so that each phase completes fully before the other begins. It is recommended that test managers aim to complete a certain percentage of the Primary phase before beginning the Secondary phase and similarly with the Secondary phase before moving onto the Tertiary phase. It is suggested that between 80 and 90% of testing in each phase should be completed before starting the next phase. • There is effort involved in planning and executing this regression strategy, and this effort needs to be estimated, resourced and scheduled in the project plan accordingly. It needs

September/December 2001

to be seen as an integral part of the project. • The benefits of this strategy do need to be explained clearly to the development community so that they can see the importance of their contributions. We have been moving to an organizational structure where developers and testers belong to the same team. For those organizations where an independent test team exists, there may be some friendly persuasion required! 7. Summary & Conclusion Regression suites can often contain thousands and thousands of tests as they are built up over releases of software, especially when dealing with a legacy product. When looking to focus regression testing, requirements are needed along with a strategy that is consistent, repeatable and flexible. Development and Test Engineers need to be actively involved in defining the requirements for the regression program and then designing the program that will test these requirements. This strategy does

September/December 2001

not provide for 100% coverage of the system being tested. What it does is identify what regression tests need to be executed based on the ‘requirements’ we have defined using the strategy, i.e., the tests in the Primary, Secondary and Tertiary Phases. Given the complexity of the system we are testing, we don’t know for sure that we have exact coverage but we are sure that we are covering all basic feature functionality, plus focusing on areas where changes are being made. For operational reasons, my organization is moving towards a combined development/test team. As a result of this we are going through some ‘test reinvention’ at the moment. As we have successfully reduced the amount of subsystem & system regression testing we need by using this strategy, increased focus is now being placed on lower level testing such as unit & integration testing, and the idea of regression testing is now being applied to our integration testing, where we will now regression test individual low-level functional areas of the MSC. We will now be using

http://www.testinginstitute.com

this strategy for our low-level functional area regression testing. We have seen several benefits on the first release where we used this strategy to design our regression program and feel it can be adapted to suit different applications and products. About the Author Martin Gilleran graduated with a B.Sc. in Electrical/Electronic Engineering from Trinity College, Dublin in 1986 and has spent the last 13 years working in industry. He spent 4.5 years with Measurex, Inc. as a software development engineer responsible for developing, testing and installing measurement and control systems for the paper industry. Mr. Gilleran has spent 8.5 years working with Motorola in different test roles. He has 4 years experience working as a test project. He then spent 2 years as a Test Manager responsible for testing a range of Motorola's Network Products including Mobile Switching Centre (MSC) and Home Location Register/Visitor Location Register (HLR/VLR).

Journal of Software Testing Professionals

33