Requirement Testing

How Fast is Fast? Using Non-Functional Requirements to Test Requirement Quality. Abstract By reviewing this paper, the reader should gain additional insight into the following areas: • The differences between functional and non-functional requirements • The importance of non-functional requirements during system analysis and testing • The major categories of non-functional requirements • The primary characteristics of each non-functional requirement type • Suggested methods for integrating non-functional requirements into software testing practices Introduction From an early age, we are all taught the basic questions to ask when gathering information. Whether you are a school child researching a report, a journalist investigating a story, or a witness recounting a crime scene, you immediately ask yourself the so-called “Magic Five Questions.” These, of course, include “Who,” “What,” “Where,” “When” and “How.” And as every good journalist knows instinctively, without all of this information the resulting work seems unfinished and/or raises as many questions as it answers.

September 2002

This paper discusses the difference between functional and non-functional requirements, defining each as they pertain to application software development. Further, it outlines a basic approach for using non-functional requirements to improve requirement quality. Suggested testing methods will also be discussed with an eye toward building validation plans using non-functional requirements for quantitative measurement.

by Joseph D. Schulz Senior Product Champion Starbase Corporation

For some reason, however, we IT professionals seem to forget these basic investigative techniques when we attempt to define the requirements for a software application. As we interview customers for their needs, we focus on only one aspect; that is, the “What.” Nearly all of the questions we ask address what the customer wants to be able to do with the application, and we assume that if we deliver everything the customer asked for then our application will be a success. Unfortunately, this assumption is blatantly inaccurate. It is just as important to ask the remaining four questions as it is to thoroughly document the first. By focusing on only the “What” of the application, we miss much of the underlying context and may ignore critical business constraints that will ultimately determine our effort’s success or failure. And just like a journalist who skips one or more of his “Magic Five,” we often end up building software applications that seem unfinished and/or cause as many problems as they solve. Functional vs. Non-Functional Requirements Unfortunately for IT, software is not like Kevin Costner’s baseball diamond

http://www.testinginstitute.com

in the Hollywood motion picture “Field of Dreams.” In the movie, Costner has a vision which tells him that “If you build it (the baseball diamond), they will come (ghostly players and real-life fans).” In software, on the other hand, just because we build it (the software), it doesn’t mean they will come (the customers). Instead, it is our job to first determine exactly what a customer needs before we begin building the solution. We do this by asking the “What” questions, and the answers to those questions are usually referred to in the IT world as the “functional requirements” for the application. Functional requirements describe what it is that a customer needs to be able to do with our software. They may be documented in the form of rigorously specified process models or use cases, or they may simply be lists of required features and functions. Whatever the form used, functional requirements should always identify the minimum functionality necessary for the software implementation to be considered a success. However, as mentioned earlier, this is not enough. Instead, business analysts

Journal of Software Testing Professionals

1

need to get a complete picture of the customer’s needs by asking the additional four questions from our “Magic Five.” These characteristics are usually referred to as “Non-Functional Requirements.” Unfortunately, the name non-functional requirements is somewhat misleading. From the name alone, it sounds like non-functional requirements are requirements that don’t function (i.e., things that don’t work). Obviously, this is just the opposite. Rather than describe what the customer wants to do, non-functional requirements describe the additional constraints that affect the customer while performing the functional requirements. They address the aspects of “Who,” “Where,” “When” and “How.” Functional Requirements Example For example, a simple functional requirement for an Order Processing System might define the application functionality necessary to create new orders. The definition of the requirement, then, would describe what an end user must be able to do during the order creation. A sample functional requirement definition of this type might look like the one below. The system must be able to enter customer orders for products that are currently in stock. A unique order number must be generated by the system at the time the order is saved. Each order must be able to specify different products and unit sizes, with each combination resulting in a separate order item. The desired unit, quantity, and unit price must be specified for each order item. Figure A: “Enter Orders” Functional Requirement Definition This example probably looks familiar to most readers, because this is the format September 2002

that the majority of requirement specification documents use. However, while the functional requirement definition shown in this example is well-formed, it is by no means complete. It accurately describes what the application end user must be able to do, but it leaves out critical information that will be necessary to design a complete application solution for the problem. For example, the functional specification above does not address any of the quantities involved in the order entry process (e.g., number of orders, order items, products, unit sizes, etc.). Without knowing this information, the database designer cannot determine the most appropriate storage technologies and the user interface designer cannot develop the optimal screen layout. These are just two of the basic non-functional issues that are left to interpretation by the development team. Requirement Ownership Another key difference between the functional requirements and the non-functional requirements rests in the ownership of the information. Specifically, there is a major difference between who is ultimately responsible for managing the functional and non-functional requirements definitions. Functional requirements are usually considered to be “owned by” the customer or end user for the application. Since functional requirements describe the work the customer needs to do, the application end user is the most appropriate source for the data and the most likely to be able to define it correctly. In a perfect world, end users can identify all of the necessary functional requirements based on their current or desired workflow. Non-functional requirements, on the other hand, are usually owned and maintained by the technical staff. This

http://www.testinginstitute.com

is primarily due to the fact that the non-functional requirements describe the context of the work to be done, rather than the work itself. While the customer is thinking about and managing the workflow, it is often up to the technical staff to ask questions about the non-functional characteristics. The answers to those questions, of course, must still come from and be validated by the customer, but the technicians must usually raise many of the relevant issues themselves. Application Verification It is these non-functional requirements, then, that are used to fully “flesh out” the test plan necessary to validate operation of our completed software application. Without knowing how the application is going to be used, we cannot conclusively determine whether the software will actually meet the customers’ needs when it is delivered. That is, we need to test not just functionality, but also usability and applicability to the business problem being solved. These non-functional characteristics often are referred to as “application stress points” because they identify the operational limits the application must work within. To adequately define these stress points, non-functional requirements must specify both the metric to be used and the measurement technique to be employed. Without both of these items, a complete test plan cannot be developed. For example, in the “Enter Orders” requirement above, it would not be enough to say that the application must handle 100 products per order. This would still leave open the question about how to “count” products. Is each unique product on the order counted as one or is each product code and unit size combination? Instead, the formula or method for counting must also be specified in the non-functional requirement definition.

Journal of Software Testing Professionals

13

Non-Functional requirements address a broad spectrum of issues...and as scuh there are several different kinds of information that we are looking for...instead of having one large pool of non-functional requirements, it is often helpful to categorize these requirements by their major subect area. To this end, the Quality Assurance (QA) staff also should be involved in the development of the requirement specifications. While QA Analysts do not define requirements, they usually review the requirement definitions and point out possible deficiencies. They look for both omissions (test characteristics not discussed at all) and ambiguities (test characteristics that are vague or incomplete). At this time, a preliminary test plan also can be developed to establish the basic pattern of testing that will be used to verify functional compliance.

Non-Functional Requirement Types As discussed above, non-functional requirements address a broad spectrum of issues. They address four of the five basic investigative questions, and as such there are several different kinds of information that we are looking for. Therefore, instead of having one large pool of non-functional requirements, it is often helpful to categorize these requirements by their major subject area. The easiest way to do this is to group facts by the questions they answer. Since we have already addressed the “What” questions, we should define non-functional categories for the “Who,” “Where,” “When” and “How” aspects. In addition, there are several layers to the “How” area, and each of these merit their own sub-category. At a 14

minimum, this should include “How Many,” “How Often,” “How Fast,” and “How Easy.” Non-Functional Requirement Categories The table below identifies the non-functional questions that need to be answered by a requirement specification document. For each, a typical name for the corresponding non-functional requirement category is listed. A detailed description of each category, including relevant examples, is included in the following sections of this document. The most common method of identifyNon-Functional Requirement Categories

Who? Where? When? How Many? How Often? How Fast? How Easy?

Requirement Security Topographical Timing Scalabilty Frequency Performance Usability

Figure B: Non-Functional Requirement Categories ing the non-functional requirements is to ask a series of “qualifying questions” for each of the categories listed above. This is done by inspecting each functional requirement and then asking the qualifying questions to the appropriate decision maker(s). The recommended qualifying questions for each category

Journal of Software Testing Professionals

are listed in the appropriate sections below. Once the qualifying questions have been answered for each requirement, commonalities then are extrapolated into global or group-based rules. Each unique rule is defined as an instance of a non-functional requirement, and then these non-functional requirements should be matrixed to the functional requirements (refer to the “Requirements Traceability” section below). In this manner, non-functional requirements are defined as separate, normalized objects in the requirements specification, but the relationships between them and to the functional requirements are documented. Security Requirements The first non-functional requirement type addresses the “Who” question. This type is used to specify which end users are allowed to perform which functions so that the appropriate security scheme can be designed into the specifications. In addition, approvals/ authorizations need to be considered as a part of the functional workflow. Unfortunately, security requirements often are not addressed until the end of the design process. This can lead to a large amount of rework, as the security issues will very often affect the application workflow itself. Without a determined, premeditated effort to address these constraints early, the workflow often becomes convoluted when the

http://www.testinginstitute.com

September 2002

security scheme is “bolted on” at the last moment. The most common qualifying questions used to discover security requirements include: • Who is allowed/denied access to perform this function? • Does the function vary by authorized end user? • Is a separate approval process (third party) required to complete this function? • If an additional approval is required, who is authorized to provide it? • Is the approver allowed to change information entered by the originator? • Is there any formal notification necessary for this function? In our “Enter Orders” example, additional non-functional requirements to address these security issues might be defined as shown below: • All members of the Order Entry department are allowed to enter new orders. • New product orders are in a status of “Pending” until approved. • All members of the Order Entry department may approve orders with a total value of less than or equal to $50,000 USD. • Only Order Entry department supervisors may approve orders with a total value of greater than $50,000 USD. • An order acceptance letter must be generated and sent to the ordering customer at the time the order is approved. Figure C: Sample Security Requirements Topographical Requirements Topographical requirements address the “Where” question. Specifically, this type of non-functional requirements defines any application characteristics that are dependent upon geographic September 2002

location, hardware profiles, and/or network topography. While these requirements are most helpful during the physical system design, they can just as easily affect the logical transaction design in the functional workflow. Of course, there is a real danger in trying to design the physical aspects of the application too early in the process. Therefore, during the functional requirement specification, only enough information about the topography needs to be captured to identify any limiting constraints. That is, topographical requirements should only be defined when the characteristic is fixed or beyond the ability of the application design team to change. The most common qualifying questions to discover topographical requirements include: • What type(s) of interface hardware/ software will this function require? • What type(s) of interface hardware/ software will the end user have available? • Where will the end user(s) for this function be physically located? • Will this function require data storage? • If data storage is required, where will the data store/database be located? • What connectivity to the data location is available at the end-user location? • Will this function require a separate application component server? In our “Enter Orders” example, additional non-functional requirements to address these topographical issues might be defined as shown on the right. Timing Requirements Another non-functional aspect that needs to be addressed involves answering the “When” question. These requirements, usually referred to as timing requirements, describe how time affects the functional requirements and determine parameters for starting and

http://www.testinginstitute.com

stopping functional workflow. In addition, dependencies between the requirements are examined, as well as parallel and mutually-exclusive activities. These factors are then used to modify processing patterns as needed to support alternative paths. Timing requirements are sometimes difficult for end users to identify, as they often involve compounding conditions outside normal boundaries. For example, to the casual observer, a traffic signal has very simple functional requirements (e.g., change from red to green, green to yellow, and back to red). However, few people realize that the dynamics of this interaction are highly dependent on time. For instance, the interval between red and green periods is very different in the evening than it is during the morning commute time. The signal may additionally be programmed to blink red or yellow continuously during the early morning hours or on business holidays. All of these variations are examples of timing requirements.

• All Order Entry functions must execute on the Windows 2000 Professional operating system. • The Order Processing system must require no more than 128MB of system RAM on the end-user machine. • The “Enter Orders” function must be available to all members of the Order Entry staff located in Atlanta (USA), San Francisco (USA), London (England), and Sidney (Australia). • Order data must be stored in a secure facility that provides access by all members of the Order Entry, Accounts Payable, and Shipping /Receiving departments.

Figure D: Sample Topographical Requirements

Journal of Software Testing Professionals

15

The most common qualifying questions used to discover timing requirements include: • Does the time of day, day of week, or date (day, month or year) impact the workflow for this function? • Does the behavior of this workflow change at slack or peak times? • What other event or function does this function depend on? • What other event or function depends on this function? • Are there other functions that can/cannot occur at the same time as this function? In our “Enter Orders” example, additional non-functional requirements to address these timing issues might be defined as shown below: • All customer orders with a status of “Pending” must be approved or deleted by 11:59 PM Eastern Standard Time. • The approval limit for all members of the Order Entry department is doubled on the last business day of each monthly period. • Customer orders for a specific product cannot be entered until that product’s inventory level is verified by the Shipping department. • Only one new order may update a product’s in-stock quantity at a time. Figure E: Sample Timing Requirements Frequency Requirements Frequency requirements are the first subcategory in the “How” category, and they address the question “How Often.” Part of a triad of questions often referred to as “volumetrics,” frequency requirements identify the number of functional transactions that can be expected within a specified timeframe. These statistics are essential when 16

designing the user interface, and also play a very important role when selecting communication, data storage, and other infrastructure components. A simple example might be the functional specifications for a household appliance like an automatic-drip coffee maker. It must be able to heat water, drip the water through a filter of ground coffee, and maintain heat on the brewed coffee. By using these functional requirements as the only measure, any coffee maker is a candidate to satisfy the need. However, once the frequency characteristics are factored in, the choice of possible coffee machines may be radically different. For instance, if the coffee maker is to be used to make 1,000 cups of coffee per day (as in the case of a restaurant), then most “home use” coffee machines are not viable. On the other hand, if the coffee maker is to be used only once per year, it may actually be more cost-effective to rent a machine when necessary and/or to purchase coffee from a third-party provider. The most common qualifying questions to discover frequency requirements include: • What is the period frequencies are measured in (e.g., items per minute, etc.)? • What is the average frequency for this function within the stated period? • What is the minimum and maximum frequency at slack/peak times? • Does the frequency for this function vary by time, day or date? • How many end users will be performing this function? • Does the frequency for this function vary by end user or end-user group? • Does the workflow for this function change at volume thresholds? In our “Enter Orders” example, additional non-functional requirements to address these frequency issues might be

Journal of Software Testing Professionals

defined as shown below: • The standard workday is from 8:00 AM EST to 5:00 PM EST, Monday through Friday. • During the standard workday, the system must be able to record an average of 50 new customer orders per hour per employee, with an expected minimum and maximum of 10 and 80, respectively. • During the standard workday, the system must be able to support an average of 12 employees working simultaneously, with an expected minimum and maximum of 4 and 20, respectively. • From Nov. 15th to Dec. 31st, the system must be able to support triple the number of orders and employees as a standard workday. • When more than thirty (30) employees are logged into the system at one time, order acceptance letters will only be generated for orders with a total value exceeding $25,000 USD. Figure F: Sample Frequency Requirements

Performance Requirements The second part of the “volumetrics” triad, performance requirements address the question “How Fast.” They define the absolute and relative performance constraints that are to be placed on the application. As with frequency requirements, performance requirements are essential to developing an adequate functional workflow and user interface for the application. They will also greatly influence the selection of communication and data storage technologies. Performance requirements are extremely pervasive in both application design and popular culture. Software developers spend a significant amount of effort designing and redesigning program-

http://www.testinginstitute.com

September 2002

ming logic to improve execution performance, and often add extraneous functionality (e.g., progress displays) just to provide visual stimulation during tedious processing. Even auto manufacturers cater to our seemingly instinctive performance desires, prominently advertising a model’s “0-to-60” rating in an attempt to entice customers. Unfortunately, while performance requirements often are discussed and even frequently written down, they are rarely expressed in concrete, testable terms. All too often, incomplete or ambiguous requirement definitions leave so much open to interpretation that end users are disappointed while the development staff is convinced they met the terms of the agreement. As discussed in the Application Verification section above, both the metric and the measurement technique must be specified in the performance requirements. The most common qualifying questions to discover performance requirements include: • What is the period that performance metrics are measured in? • How is the performance data to be captured (e.g., duration or response time)? • What is the absolute minimum standard for functional performance? • What is the absolute maximum standard for functional performance? • Does the performance requirement vary by time, day or date? • Does the performance requirement vary by end user or end-user group?

• All performance measurements shall be based on the response time experienced at the end-user machine. • When the end user presses the “New” button, the order number must be generated and displayed within a maximum of 4 seconds. • From Nov. 15th to Dec. 31st, allowable response times must be 25% less than standard (e.g., 3 seconds instead of 4). • When employees outside of the Order Entry department use the system, allowable response times can be up to 50% longer (e.g., 6 sec. instead of 4). Figure G: Sample Performance Requirements Scalability requirements are similar to the frequency requirements described above, except that scalability requirements focus on the expected future state of the application rather than the current state. Scalability requirements are used to defer “operational obsolescence” by designing for future capacities.

In our “Enter Orders” example, additional non-functional requirements to address these performance issues might be defined as shown above.

An everyday example of this type of thinking might be when young couples begin to look for a home. They not only consider what they need immediately (e.g., a kitchen, one bedroom, one bath, etc.), but also what they will need in the future (e.g., extra bedrooms, baths, etc.). They then balance the cost of purchasing these upgrades immediately against the cost of having to move again when they become a necessity. This process of “planning for the future” is an example of using scalability requirements to influence the decision-making and design process.

Scalability Requirements The third and final part of the “volumetrics” triad is the scalability requirements, which define the characteristics to resolve the question “How Many.”

In application systems, scalability requirements most often specify expected future volumes and performance criteria. In addition, scalability requirements also are used to record archival

September 2002

http://www.testinginstitute.com

and record retention policies so that the “volumetrics” can be adjusted for an anticipated year-on-year growth. The most common qualifying questions to discover scalability requirements include: • How long is the application’s expected lifecycle? • How will the average transaction volume change throughout this lifecycle? • How will the average end-user population change throughout this lifecycle? • What type and how many transactions will be stored? • How long (in duration) must these transactions be stored? In our “Enter Orders” example, additional non-functional requirements to address these scalability issues might be defined as shown below: • The system must be able to operate for a minimum of ten years. • The number of new customer orders entered is expected to increase at an average of 10% per year from the current statistics. • The number of Order Entry employees is expected to remain flat for two years and then to increase an average of 5% per year thereafter. • Approved orders will be retained in active storage for a minimum of one calendar year after they are approved Figure H: Sample Scalability Requirements Usability Requirements The final subcategory within the “How” requirements categories involves identifying the usability requirements. These requirements address the end-user aspects of the application and answer the question “How Easy.” Many times, these requirements are a mix of corporate and/or industry standards and

Journal of Software Testing Professionals

17

application-specific needs. Most often, usability requirements identify necessary traits for the user interface design. This might include standardized look-and-feel characteristics which are specific to the operating platform on which the application will be executed. In addition, the usability requirements may also include specifications about required skill levels and/or training needs. All of these qualities are critical when designing the so-called “man-machine boundary.” The most common example of the need for usability requirements is the modern Video Cassette Recorder (VCR). While the basic functionality of a VCR has not changed in over a decade, when these devices were first introduced they were so difficult to use that the majority of the population avoided them. In fact, it was considered a “badge of honor” if someone was able to successfully program their VCR to record a favorite television show. More recently, however, the VCR manufacturers have learned from their mistakes and have added a wide variety of usability enhancements that make their devices easy and sometimes fun to use. In fact, they have been able to dramatically widen their customer base simply by focusing on usability characteristics rather than functionality. The most common qualifying questions to discover usability requirements include: • What type(s) of interface hardware/ software will the end user have available? • Are there any company/industry interface standards that apply to this function? • What is the average level of skill for this function? • What is the minimum/maximum amount of end-user training expected to be available for this function? • What type(s) of online help is 18

required for this function? • Are there any unique accessibility requirements for this function? • How is this function performed in the current environment? In our “Enter Orders” example, additional non-functional requirements to address these usability issues might be defined as shown below: • All Order Entry functions must utilize a graphical user interface compliant with the Microsoft standard #99.9999 published on 6/1/2001. • End-users must have completed the approved corporate training class for Microsoft Windows 2000 prior to the installation of the new system. • The “Enter Orders” function must require an average of less than 8 hours of formal classroom training to achieve proficiency. • Customer orders must be able to be entered by end-users that use the Microsoft Windows 2000 “StickyKeys” accessibility option. • End-user “drag-and-drop” functionality must not be required. Figure I: Sample Usability Requirements

Requirements Traceability The final step in completing the requirement specification is to establish causality relationships, named “traces,” between the various requirement instances. Specifically, traces must be defined between any related non-functional requirements and between non-functional requirements and the affected functional requirements. Unlike requirements definitions, traces are hard to manage in a textual or document-based format. Because of this, functional specifications that include requirements traceability usually necessitate the use of a commercial

Journal of Software Testing Professionals

Requirements Management (RM) product that stores the requirements definition in a database format. Non-Functional Dependencies The first type of trace, between two non-functional requirements, is usually referred to as a “dependency.” Dependencies denote that one non-functional requirement has a direct and immediate impact on another. They are used to identify the effect that one requirement will have on another. Very often, dependencies between non-functional requirements are used to model calculation or multi-step logic where one fact is based on the definition of another. One obvious example of a dependency is contained in the frequency requirements above where one requirement lists the standard workday transaction volume as “50 new customer orders per hour per employee” and another lists the holiday volume as “triple the number of orders and employees as a standard workday”. Maintaining a dependency between these two requirements is important since the holiday volume must be re-evaluated if the standard workday requirement is modified. Non-Functional Constraints The second type of trace, between functional and non-functional requirements, is usually referred to as a “constraint.” That is, the non-functional requirement constrains, or limits, the functionality described in the functional requirement. Constraints usually include operational and/or performance criteria which circumscribe the conditions under which the function is performed. For example, one performance requirement example listed above stated that “the order number must be generated within a maximum of 4 seconds.” This constraint should be traced back to the functional requirement that described generating the order number. In this

http://www.testinginstitute.com

September 2002

A well-formed system requirements specification must include not only definitions of the functional requirements (the “What”), but also of the non-functional requirements (the “Who,” “Where,” “When” and “How”). manner, when the application designers are developing the system for generating order numbers, they will immediately know: 1) what has to be done; and 2) the constraints within which it has to be done. These constraint relationships are then invaluable for subsequent impact analysis. If the functional requirement changes (e.g., more functionality is added), the designer will immediately know which constraints still apply to the new design. Conversely, if the non-functional requirement changes (e.g., a restriction is eased), the designer can immediately see which functional requirements are affected and may be able to alter the design specifications to take advantage of a new opportunity. Using Traces for Quality Assurance Traces, both dependencies and constraints, also are important during the development of the Quality Assurance plan. When the functional tests are defined, it is usually the non-functional requirements that provide the measurements necessary to properly determine success or failure of the testing process. As described earlier, it is not enough just for the application system to work as designed; rather, it must also work

September 2002

with the customer’s environment and within the environmental pressures that exist.

Summary A well-formed system requirements specification must include not only definitions of the functional requirements (the “What”), but also of the non-functional requirements (the “Who,” “Where,” “When” and “How”). These non-functional requirements then should be organized into appropriate categories and should detail the environmental characteristics that constrain or bound the application. The non-functional requirements then should be related to one another and to the relevant functional requirements. Without both of these components in the requirements specification, the application development team risks delivering an incomplete or unusable application. By defining both components and relating them to each other, the development team is much more likely to deliver an application that not only works as designed but also can operate within the end-user’s dynamic environment.

http://www.testinginstitute.com

About the Author Joseph D. Schulz is a seasoned developer with more than 19 years of professional Information Systems experience spanning a wide range of vertical industries. Currently, Joe is the Senior Product Champion at Starbase Corporation. In this position, he helps organizations around the globe improve their development processes by delivering methods/tool training and providing onsite mentoring. He also supports Starbase’s international distributors, ensuring their ability to support the Starbase family of products.

References 1. Weigers, KE: “Software requirements,” Microsoft Press, Redmond WA USA, 1999. 2. Young, RR: “Effective Requirements Practices,” Addison, Reading MA USA, 2001. 3. Allen, P, and Frost, S: “Component-Based Development for Enterprise Systems: Applying the SELECT Perspective,” Cambridge Univ. Press, Cambridge UK, 1998. 4. Schneider, G, and Winters, JP: “Applying Use Cases: A Practical Guide,” Addison, Reading MA USA, 1998.

Journal of Software Testing Professionals

19