Software Structural Test Method

Software Structural Testing Methods By George N. Brower Analex Corporation

❖

S

levels discussed below. tructural testing encompasses ❝Structural Structural testing should be done three critical phases of softat the unit, integration, and system ware development and testing; yet, one or more of these phas- testing assures the levels of testing.2 (As used herein, a “unit” is the smallest separately es is often deliberately bypassed, program’s compilable – or equivalent – eleoverlooked, or performed in a less ment of code, such as a procedure, than rigorous manner because either statements and subroutine, class, method, or datathe technical advantages are not base table.) Structural testing fully considered or, more often, the decisions are assures the program’s statements cost and schedule benefits are not fully exercised and decisions are fully exercised by appreciated. While structural testcode execution. For example, it ing is required by the FDA for by code confirms that program loop conmedical devices of moderate and 1 structs behave as expected at their major level of concern, this testing execution.❞ data boundaries. For configurable should be done for all software, software, the integrity of the data in regardless of the level of concern. Also, it should be noted that there is no fundamental configuration tables are evaluated for their impact difference between structural testing of software on program behavior. At the unit level, structural testing also includes the identification of “dead used in a medical device and that used in a manufacturing process or a manufacturer’s quality system code,” which is code that cannot be reached for exe(or, for that matter, in any other software). An under- cution by any code pathway. Integration structural testing should be performed standing of these considerations and, thus, the after all verification testing of the units involved and importance of performing structural testing, are disbefore system-level structural testing. Figure 1 illuscussed below. trates the general relationship between software verDefinition of Software Structural Testing ification and validation. Software verification confirms that the output of each phase of software Software structural testing is meant to challenge development is true to (i.e., is consistent with) the the decisions made by the program with test cases inputs to that phase. Performance qualification testbased on the structure and logic of the design and ing confirms the final software product, running in source code. Complete structural testing exercises its intended hardware and environment, is consistent with the intended product as defined in the product the program’s data structures (such as configuration specifications and software requirements. tables) and its control and procedural logic at the test Journal of Validation Technology

37

George N. Brower

Figure 1

Figure 2

Software Verification Versus Software Validation Testing

Configuration for a Structural Unit Test Driver Simulating “Compute Y”

Concept

(Unit-UnderTest)

Product Specifications

Get Formatted ABC

Verification Software Requirements Stub Simulating “Format ABC”

Verification Performance Qualification

Design Verification Source Code

Executable Code

Hardware

Verification (Test)

Maintenance

Unit-Level Structural Testing Figure 2 illustrates a typical configuration for a structural unit test. The unit is compiled and linked with a driver and stubs, as needed. The driver is a substitute for any actual unit that will eventually call the unit-under-test, and if the driver passes data to the unit-under-test, it is set up to pass test case variable values such as maximum, minimum, and other nominal and “stress-test” values. The stubs are substitutes for any units called by the unit-under-test. As with the driver, if the stubs return data to the unitunder-test, they also pass “stress test” and nominal data values, as appropriate. The interface of the drivers and stubs, including their names, are the same as the true units’ interfaces, allowing the set of units to be linked without altering the unit-under-test. 38

Journal of Validation Technology

Stub Simulating “Notify Operator of Invalid Data”

Unit-level structural tests can be conducted on the actual target hardware, on an emulator or simulator of the actual hardware, or, if the situation requires, on a totally different processor. The latter case may occur, e.g., if the actual hardware has no provision for determining the results of a unit’s tests, but where the code is written in a higher order language. Thus, the higher order source code (such as C or C++) can be compiled and linked to run on another computer that supports reading the test results where the target computer (for example, an embedded microprocessor) could not support the tests. Structural testing (a.k.a. white-box testing) is performed with the item-under-test, in this case the unit, being viewed internally for purposes of determining how the item should behave – for example, in determining all possible code branches. The primary purpose of unit-level structural testing is to verify the code complies with the design, including logic, algorithms’ correctness, and accuracy (versus parallel code or hand calculations), and the correctness of the data’s engineering units. This requires, for each unit, complete branch tests, complete code tests (including verifying there is no dead code), and stress tests (to detect, e.g., overflow and underflow conditions as well as nominal and maximum loopcontrol data values). The detailed design is used to develop the acceptance criteria.

George N. Brower

The Environment for Both Integration-Level and System-Level Structural Tests It is best to set up the integration and system structural tests using the actual hardware and environment to the extent practical. There are several reasons for this, but the two most significant are (a) the software may have subtle conditions, both good and bad, that will only show up when running on the actual hardware, and (b) the final computerized system, including the intended hardware and software, must be qualified running on that hardware, and the structural tests should advance the software development towards that end. However, there are also good and sufficient reasons to perform structural tests partially or wholly in a simulated environment. In considering establishing simulation capabilities, the two most common configurations are to either emulate the computer and simulate the environment (used most often when the actual computer is an embedded microprocessor and it is difficult to stimulate known inputs and/or read the outputs of a test) or to simulate both the computer and the environment (used, e.g., if an emulator of the target computer is not available). The principal advantages, then, in using a simulation of the environment and, at times, the computer include the following: (a) the ability to set up absolutely known input values, such that the results can be predetermined to establish the acceptance criteria of each test; (b) a simulator makes it easy to establish inputs that are over, under, and at the exact limits of critical data values; (c) it is easy to set up illegal inputs to test all error and failure conditions; and, finally, (d) the results of each test can be readily seen.

Integration-Level Structural Testing Integration structural testing combines functionally cohesive units of verified code (which includes unit-level structurally tested code) by compiling and/or assembling and linking the code, along with any drivers and stubs needed. The structure is then loaded into the actual or simulated environment for execution. This allows the tester to focus on that one functional package to confirm its correct operation, including all internal and external interfaces.

Following completion of each functional package’s test, the next functional package may be either separately tested or added to (i.e., linked with) the previously tested package(s). Regression testing (i.e., running a selected subset of previous, successfully run test cases) must be performed on the previously tested packages to confirm they are not adversely affected by the newly introduced functional package. Figure 3 illustrates a building-block, or incremental, approach to structural testing. While the figure’s illustration is related to a structured design, the same approach is used for all other design methods, including flowchart and object-oriented design. In Figure 3, the first function needs two stubs, “Get Formatted ABC” and “Output Y to Device X,” (where a stub is a simple software dummy needed to link successfully, but is not part of the software being tested). The stubs in the figure have no calls to additional units. The second and third functions require no driver (because they use the actual “Compute Y”) and they require no stubs (because they use the actual “Output Error Messages” unit).

The Incremental Approach The incremental approach to integration-level structural tests is the best for software developers (as opposed to third party, validation testers – see below), especially if the program is large or complex. In this approach, selected small, functionally cohesive portions of the software are compiled, linked, and tested. This approach is used regardless of the software life cycle development method being employed, including any of the following three methods. In the waterfall method, all of the requirements are developed, then the design is completed, and, finally, selected “threads” are coded and structurally tested. In the spiral method, a major element of the software system is discussed and then the requirements, design, and code are developed, and the element’s structural test is performed prior to going on to discuss and develop the next major element. Finally, in the incremental software development method, all of the specifications and requirements may be developed, but the design and implementation are developed one function at a time. In any case, if the operating system was uniquely developed for the system-under-test, that operatJournal of Validation Technology

39

George N. Brower

Figure 3

Incremental Integration Structural Testing

Compute Y

Get Formatted ABC

Initialize Data

Output Y to Device X

Open Files Notify Operator of Invalid Data

Format ABC Notify Operator of Any Errors

2nd Function

Notify Operator if Device X is Off-Line 3rd Function

1st Function

Output Error Messages

ing system should be structurally tested first. This portion of the code itself should be broken down into functionally cohesive packages if it is large and/or complex; otherwise, it can be structurally tested as an entity. The second portion of the code to be tested is normally the unique input/output section. If there are diverse input/output devices, these may be structurally tested separately. But it is often best to select a thread that includes both the ability to input data and to see the resulting output for each structural test. The third step is to select and structurally test a functionally cohesive portion of the application and the utilities needed to support that application. (Remember: The units selected for integration structural testing, and this is especially true for the utilities, should be verification tested first – which includes unit testing – prior to any integration-level structural testing.) Then, select the next functionally cohesive portion of the application software and associated application 40

Journal of Validation Technology

utilities for the next structural test, and so on. All previously tested functions should be regression tested, as appropriate.

The Nonincremental Approach As used herein, a nonincremental approach is one in which the complete software package, as it is intended to be deployed, is totally built and then structurally tested. The nonincremental approach to integration-level structural tests may reasonably be used for small programs and is often the best approach for third party, independent testers. Although in most circumstances integration-level structural testing is the purview of the software developers and not, for example, that of the computer system validation team, at times these third-party testers are asked to support a program in this manner. An immediate problem that most often surfaces, however, is due to budget, schedule, and/or personnel con-

George N. Brower

straints, testers independent of the software development team cannot, in a cost-effective manner, select, compile, and link individual software functions. However, they can build and load the fully integrated program, select and test a function (ignoring the remainder of program while maintaining a structural-tester’s point-of-view), and then select the next function for test. One advantage to this approach is since a majority of the code may be executing during each test, the need for regression testing of the previous functions is diminished. A disadvantage to this nonincremental approach is if a function is found to not work properly, finding the cause of the problem may be more difficult precisely because so much of the code is executing. At the integration level, structural testing uses the architectural high-level design to develop the test cases and to derive the acceptance criteria. (During the integration test phase, functional, black-box testing should also be performed.)

System-Level Structural Testing The system-level structural tests must be run on the fully integrated software package. However, whether the software functions for the integration structural tests discussed above are built incrementally or all at once prior to conducting the integration structural tests, the system-level structural tests must still be run. The primary differences between integration level and system level structural tests are the tester’s frame of reference for developing the test inputs and determining the acceptance criteria of the test outputs. An important secondary difference in integration structural testing is that the code may be altered (e.g., to create test drivers and stubs) or the normal code execution flow may be altered (e.g., running the software in a de-bug compiler mode where the execution can be halted, a data value changed, and the execution restarted, allowing for a variable’s maximum or minimum value, or a failure or other perturbation to be inserted). In system structural testing, the code must not be changed, and the program execution flow must not be altered. Rather, all test inputs must be accomplished by control of the actual or simulated environment. At the system level, structural testing is also best performed along with functional, black-box testing,

both using the product specifications (which in turn may include the user’s manual and other labeling) and software requirements for the test development and the acceptance criteria. Although, at the system level, a knowledge of the design may be necessary to determine how to set up a test for a specific functionality (for example, an output error message that a file cannot be found versus an output error message indicating there is invalid data in a file), that design knowledge should not be used in constructing the acceptance criteria of the tests – such as the boundaries of legitimate data. That information should be derived from the specifications and requirements.

The Importance of Each Testing Phase As with each verification, integration, and qualification activity, each structural testing phase finds unique problems that otherwise may be either very difficult to find at best or, at worst, may be left undetected during pre-release testing, waiting for the error to surface in the field under actual use. Using a step-wise approach to testing finds problems and solutions faster and with a greater guarantee of producing a solid software product than if any phase is skipped or not performed with rigor. In this context, rigor means (a) planning all tests and reviewing the plans for completeness, correctness, and consistency; (b) writing the procedures sufficiently well so they may be reviewed and, when necessary, rerun (e.g., following software updates); and (c) maintaining a written record of the tests performed and the results so those may also be reviewed for confirmation that all tests have been completed. 3 In performing the above three tasks, program management is supported in that all of the activities, from planning through execution and reporting, can be scheduled, budgeted, and, therefore, managed. Figure 4 illustrates, for each structural test phase, the documentation against which the software is tested.1 In that figure, software development implementation is shown on the left side, going from product specifications to code implementation. Software (structural) test is shown on the right side, going from unit test to system test. The documentation for each test phase is illustrated by the horizontal arrows. This Journal of Validation Technology

41

George N. Brower

general approach is followed for any software development and testing life cycle approach, be it waterfall, spiral, or iterative. For example, if the spiral approach is used, a piece of the product specification is determined, the software requirements for that piece are developed, the high-level and detailed design, and then the code for that piece are developed, and then the testing is accomplished starting at the unit level, working through the integration level, and finally working up to the system level. Figure 4

Structural Test Verification Documentation Product Specifications and Software Requirements

System-Level Structural Test

Top-Level Design

Integration-Level Structural Test

Detailed Design

Unit-Level Structural Test

Implementation (Coding)

Again, the preceding discussion, as well as all aspects of this paper, applies to all software applications, whether the software is for a process control system, medical device, software supporting a quality system, or any other software, such as that used for simulators or even business systems.

Summary Structural testing allows a step-by-step, rigorous test approach that easily uncovers problems at each phase that may otherwise be very difficult to detect during other software verification and validation activities. Therefore, structural testing not only adds value from a technical point of view, but it also supports programmatic issues, mitigating the possibility 42

Journal of Validation Technology

of hidden, difficult-to-find software problems in the succeeding test phases that can easily disrupt any schedule and budget. ❏

About the Author George N. Brower has published several articles and has given presentations at numerous confer ences on his areas of expertise: software develop ment and software Independent Verification and Validation (IV&V) testing. For the past 11 years, he has served as Deputy Director of the Denver office of Analex Corporation, providing software engineer ing services for computer systems validation and outsource software development. Analex-Denver is an ISO 9001 4 registered company that is the recipi ent of the 1995 Supplier of the Year Award for IV&V testing and the 1996 recipient of the James S. Cogswell Industrial Achievement Award. In 1998 Analex received a second award for IV&V testing of critical software. Brower can be reached by phone toll free at 1-888-262-5391, by fax at 303-730-2057, or by e-mail at [email protected].

References 1.

Guidance for FDA Reviewers and Industry, Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices, CDRH, May 29, 1998. 2. Guidance for Industry, General Principles of Software Validation (Draft Guidance) Version 1.1, June 9, 1997. 3. FDA Current Good Manufacturing Practice (cGMP) Final Rule; Quality System Regulation, 21 CFR Part 820.30, Design Control, fully effective June 1, 1998. 4. International Organization for Standardization, 1994. (Analex is registered for independent software verification and validation, software development, systems analysis, and hardware prototyping.)