Types Of Testing And Methodologies

Jagan Mohan

TESTING TYPES AND THEIR METHODOLOGIES

Jagan Mohan

Index

1. Unit Testing 2. Integration Testing 3. System Testing 4. Regression Testing 5. Load Testing 6. Performance Testing 7. Stress Testing 8. Model Based Testing 9. Installation Testing 10. Usability Testing 11. User Acceptance Testing 12. Black Box Testing 13. White Box Testing 14. Glass Box Testing 15. Fuzz Testing

Jagan Mohan

Unit Testing In computer programming, a unit test is a procedure used to verify that a particular module of source code is working properly. The idea about unit tests is to write test cases for all functions and methods so that whenever a change causes a regression, it can be quickly identified and fixed. Ideally, each test case is separate from the others; constructs such as mock objects can assist in separating unit tests. This type of testing is mostly done by the developers and not by end-users.

Benefits The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. Unit testing provides a strict, written contract that the piece of code must satisfy. As a result, it affords several benefits. Facilitates change Unit testing allows the programmer to refactor code at a later date, and make sure the module still works correctly (i.e. regression testing). This provides the benefit of encouraging programmers to make changes to the code since it is easy for the programmer to check if the piece is still working properly. Simplifies integration Unit testing helps eliminate uncertainty in the pieces themselves and can be used in a bottom-up testing style approach. By testing the parts of a program first and then testing the sum of its parts, integration testing becomes much easier. Documentation Unit testing provides a sort of "living document". Clients and other developers looking to learn how to use the class can look at the unit tests to determine how to use the class to fit their needs and gain a basic understanding of the API. Separation of interface from implementation Because some classes may have references to other classes, testing a class can frequently spill over into testing another class. A common example of this is classes that depend on a database:

Jagan Mohan in order to test the class, the tester often writes code that interacts with the database. This is a mistake, because a unit test should never go outside of its own class boundary. As a result, the software developer abstracts an interface around the database connection, and then implements that interface with their own mock object. This results in loosely coupled code, minimizing dependencies in the system. Limitations Unit-testing will not catch every error in the program. By definition, it only tests the functionality of the units themselves. Therefore, it will not catch integration errors, performance problems and any other system-wide issues. In addition, it may not be easy to anticipate all special cases of input the program unit under study may receive in reality. Unit testing is only effective if it is used in conjunction with other software testing activities. It is unrealistic to test all possible input combinations for any non-trivial piece of software. A unit test can only show the presence of errors; it cannot show the absence of errors. Applications Extreme Programming While Unit Testing is often associated with Extreme Programming, it existed a long time before Extreme Programming was invented. The unit testing concept is part of the Extreme Programming method of software engineering. Various unit testing frameworks, based on a design by Kent Beck, have come to be known collectively as xUnit, and are available for many programming languages and development platforms. Unit testing is the building block to test driven development (TDD). Extreme Programming and most other methods use unit tests to perform black box testing. Note that many in the Extreme Programming community favor the terms "developer testing" or "programmer testing" over the term "unit testing," since many other test activities (like function or acceptance test) can now be done at "developer-time." Techniques Conventionally and as a well accepted industry practice, unit testing is conducted in an automated environment through the use of a third party supplied component or framework. However, one reputable organization, the IEEE, prescribes neither an automated nor manual approach. This suggests the option to conduct unit testing by following a step-by-step

Jagan Mohan instructional document. Nevertheless, the objective in unit testing is to isolate a unit and verify its correctness. Automation is much more efficient and enables the many benefits listed in this article. In fact, manual unit testing is arguably a form of integration testing and thus precludes the achievement of most (if not all) of the goals established for unit testing. To fully realize the effect of isolation, the unit or code body subjected to the unit test is executed within a framework outside of its natural environment, that is, outside of the product or calling context for which it was originally created. Testing the unit outside of its product context ensures that testing is done without any dependencies between the code being tested and other units or data spaces in the product and that might affect test results. Consequently, unit testing is traditionally a motivator for programmers to create decoupled and cohesive code bodies. This practice promotes healthy habits in software development and in turn supports the software applicable -ilities of system engineering. Design patterns and unit testing often go hand-in-hand.

Integration testing Integration testing (sometimes called Integration and testing and abbreviated I&T) is the phase of software testing in which individual software modules are combined and tested as a group. It follows unit testing and precedes system testing. Integration testing takes as its input modules that have been checked out by unit testing, groups them in larger aggregates, applies tests defined in an Integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing. Purpose The purpose of Integration testing is to verify functional, performance and reliability requirements placed on major design items. These "design items", i.e. assemblages (or groups of units), are exercised through their interfaces using Black box testing, success and error cases being simulated via appropriate parameter and data inputs. Simulated usage of shared data areas and inter-process communication is tested, individual subsystems are exercised through their input interface. All test cases are constructed to test that all components within assemblages interact correctly, for example, across procedure calls or process activations.

Jagan Mohan The overall idea is a "building block" approach, in which verified assemblages are added to a verified base which is then used to support the Integration testing of further assemblages. The different types integration testing are Big Bang, Top Down , Bottom Up and Back bone.

System testing System testing is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of Black box testing, and as such, should require no knowledge of the inner design of the code or logic. Alpha testing and Beta testing are sub-categories of System testing. As a rule, System testing takes, as its input, all of the "integrated" software components that have successfully passed Integration testing and also the software system itself integrated with any applicable hardware system(s). The purpose of Integration testing is to detect any inconsistencies between the software units that are integrated together called assemblages or between any of the assemblages and hardware. System testing is more of a limiting type of testing, where it seeks to detect both defects within the "inter-assemblages" and also the system as a whole. Testing the whole system Generally speaking, System testing is the first time that the entire system can be tested against the Functional Requirement Specification(s) (FRS) and/or the System Requirement Specification (SRS), these are the rules that describe the functionality that the vendor (the entity developing the software) and a customer have agreed upon. System testing tends to be more of an investigatory testing phase, where the focus is to have almost a destructive attitude and test not only the design, but also the behavior and even the believed expectations of the customer. System testing is intended to test up to and beyond the bounds defined in the software/hardware requirements specification(s). One could view System testing as the final destructive testing phase before Acceptance testing. [edit] Types of testing

Jagan Mohan The following examples are different types of testing, that should be considered during System testing: •

Functional testing

•

User interface testing

•

Model based testing

•

Error exit testing

•

User help testing

•

Security Testing

•

Capacity testing

•

Performance testing

•

Sanity testing

•

Regression testing

•

Reliability testing

•

Recovery testing

•

Installation testing

•

Maintenance testing

•

Documentation testing

Although different testing organizations may prescribe different tests, within System testing, this list serves as a general framework, or foundation to begin with.

Regression testing Regression testing is any type of software testing which seeks to uncover regression bugs. Regression bugs occur whenever software functionality that previously worked as desired stops working or no longer works in the same way that was previously planned. Typically regression bugs occur as an unintended consequence of program changes. Common methods of regression testing include re-running previously run tests and checking whether previously-fixed faults have reemerged. Experience has shown that as software is developed, this kind of reemergence of faults is quite common. Sometimes it occurs because a fix gets lost through poor revision control practices (or simple human error in revision control), but just as often a fix for a problem will be "fragile" - if

Jagan Mohan some other change is made to the program, the fix no longer works. Finally, it has often been the case that when some feature is redesigned, the same mistakes will be made in the redesign that were made in the original implementation of the feature. Therefore, in most software development situations it is considered good practice that when a bug is located and fixed, a test that exposes the bug is recorded and regularly retested after subsequent changes to the program. Although this may be done through manual testing procedures using programming techniques, it is often done using automated testing tools. Such a 'test suite' contains software tools that allows the testing environment to execute all the regression test cases automatically; some projects even set up automated systems to automatically re-run all regression tests at specified intervals and report any regressions. Common strategies are to run such a system after every successful compile (for small projects), every night, or once a week. Regression testing is an integral part of the extreme programming software development methodology. In this methodology, design documents are replaced by extensive, repeatable, and automated testing of the entire software package at every stage in the software development cycle.

Load testing Load testing is the act of testing a system under load. In mechanical systems it refers to the testing of a system to certify it under the appropriate regulations (in the UK LOLER - Lifting Operations and Lifting Equipment Regulations) Load testing is usually carried out to a load 1.5x the SWL (Safe Working Load) periodic recertification is required. In software engineering it is a blanket term that is used in many different ways across the professional software testing community. Load testing generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program's services concurrently. As such, this testing is most relevant for multi-user systems, often one built using a client/server model, such as web servers. However, other types of software systems can be load-tested also. For

Jagan Mohan example, a word processor or graphics editor can be forced to read an extremely large document; or a financial package can be forced to generate a report based on several years' worth of data. When the load placed on the system is raised beyond normal usage patterns, in order to test the system's response at unusually high or peak loads, it is known as stress testing. The load is usually so great that error conditions are the expected result, although there is a gray area between the two domains and no clear boundary exists when an activity ceases to be a load test and becomes a stress test. There is little agreement on what the specific goals of load testing are. The term is often used synonymously with performance testing, reliability testing, and volume testing.

Performance testing In software engineering, performance testing is testing that is performed to determine how fast some aspect of a system performs under a particular workload. Performance testing is a subset of performance engineering, an emerging computer science practice which strives to build performance into the design and architecture of a system, prior to the onset of actual coding effort. Performance testing can serve different purposes. It can demonstrate that the system meets performance criteria. It can compare two systems to find which performs better. Or it can measure what parts of the system or workload cause the system to perform badly. In the diagnostic case, software engineers use tools such as profilers to measure what parts of a device or software contribute most to the poor performance or to establish throughput levels (and thresholds) for maintained acceptable response time. It is critical to the cost performance of a new system, that performance test efforts begin at the inception of the development project and extend through to deployment. The later a performance defect is detected, the higher the cost of remediation. This is true in the case of functional testing, but even more so with performance testing, due to the end-to-end nature of its scope. It is important (and often difficult to arrange) for the performance test conditions to be similar to the expected actual use. Testing technology

Jagan Mohan Performance testing technology employs one or more PCs to act as injectors – each emulating the presence or numbers of users and each running an automated sequence of interactions (recorded as a script, or as a series of scripts to emulate different types of user interaction) with the host whose performance is being tested. Usually, a separate PC acts as a test conductor, coordinating and gathering metrics from each of the injectors and collating performance data for reporting purposes. The usual sequence is to ramp up the load – starting with a small number of virtual users and increasing the number over a period to some maximum. The test result shows how the performance varies with the load, given as number of users vs response time. Various tools are available to perform such tests. Tools in this category usually execute a suite of tests which will emulate real users against the system. Sometimes the results can reveal oddities, e.g., that while the average response time might be acceptable, there are outliers of a few key transactions that take considerably longer to complete – something that might be caused by inefficient database queries, etc. Performance testing can be combined with stress testing, in order to see what happens when an acceptable load is exceeded — does the system crash? How long does it take to recover after a large load is reduced? Does it fail in a way that causes collateral damage? Performance specifications It is critical to detail performance specifications (requirements) and document them in any performance test plan. Ideally, this is done during the requirements development phase of any system development project, prior to any design effort. Except in real-time computing, performance testing is frequently not performed against a specification. I.e. no one has expressed the maximum acceptable response time for a given population of users. Performance testing is often used as part of the process of performance profile tuning. The goal is to identify the "weakest links" — there are often a small number of parts in the system which, if they are made faster, will result in the overall system running noticeably faster. It is sometimes difficult to identify which parts of the system represent the critical paths. To help identify critical paths some test tools include (or have as add-ons) instrumentation agents that run on the server and report transaction times, database access times, network overhead, and other server monitors. Without such instrumentation the use of primitive system tools may be required (e.g. Task Manager in Microsoft Windows). Performance testing usually concludes that it is the software (rather than hardware) that contribute most to delays (bottlenecks) in processing data.

Jagan Mohan Performance testing can be performed across the web, and even done in different parts of the country, since the response times of the internet itself vary regionally. It can also be done inhouse, although routers would then need to be configured to introduce the lag what would typically occur on public networks. Loads should be introduced to the system from realistic points. For example, if 50% of a system's user base will be accessing the system via a 56K modem connection and the other half over a T1, then the load injectors (computers that simulate real users) should either inject load over the same connections (ideal) or simulate the network latency of such connections, following the same user profile. It is always helpful to have a statement of the likely peak numbers of users that might be expected to use the system at peak times. If there can also be a statement of what constitutes the maximum allowable 95 percentile response time, then an injector configuration could be used to test whether the proposed system met that specification. Performance specifcations (requirements) should ask the following questions, at a minimum: •

In detail, what is the performance test scope? What subsystems, interfaces, components, etc are in and out of scope for this test?

•

For the user interfaces (UI's) involved, how many concurrent users are expected for each (specify peak vs. nominal)?

•

What does the target system (hardware) look like (specify all server and network appliance configurations)?

•

What is the Application Workload Mix of each application component? (for example: 20% login, 40% search, 30% item select, 10% checkout).

•

What is the System Workload Mix? [Multiple worloads may be simulated in a single performance test] (for example: 30% Workload A, 20% Workload B, 50% Workload C)

•

What are the time requirements for any/all backend batch processes (specify peak vs. nominal)?

Tasks to undertake Tasks to perform such a test would include: •

Decide whether to use internal or external resources to perform the tests, depending on inhouse expertise (or lack thereof)

•

Gather or ilicit performance requirements (specifications) from users and/or business analysts

Jagan Mohan •

Develop a high-level plan (or project charter), including requirements, resources, timelines and milestones

•

Develop of a detailed performance test plan (including detailed scenarios and test cases, workloads, environment info, etc)

•

Choose test tool(s)

•

Specify test data needed and charter effort (often overlooked, but often the death of a valid performance test)

•

Develop proof-of-concept scripts for each application/component under test, using chosen test tools and strategies

•

Develop detailed performance test project plan, including all dependencies and associated timelines

•

Install and configure injectors/controller

•

Configure the test environment (ideally identical hardware to the production platform), router configuration, quiet network (we don’t want results upset by other users), deployment of server instrumentation, database test sets developed, etc.

•

Execute tests – probably repeatedly (iteratively) in order to see whether any unaccounted for factor might affect the results

•

Analyze the results - either pass/fail, or investigation of critical path and recommendation of corrective action

Stress testing Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results. For example, a web server may be stress tested using scripts, bots, and various denial of service tools to observe the performance of a web site during peak loads. Stress testing is a subset of load testing.

Model-based testing Model-based testing refers to software testing where test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT).

Jagan Mohan

General model-based testing setting The model is usually an abstract, partial presentation of the system under test's desired behavior. The test cases derived from this model are functional tests on the same level of abstraction as the model. These test cases are collectively known as the abstract test suite. The abstract test suite cannot be directly executed against the system under test because it is on the wrong level of abstraction. Therefore an executable test suite must be derived from the abstract test suite that can communicate with the system under test. This is done by mapping the abstract test cases to concrete test cases suitable for execution. There are many different ways to "derive" tests from a model. Because testing is usually experimental and based on heuristics, there is no one best way to do this. It is common to consolidate all test derivation related design decisions into a package that is often known as "test requirements", "test purpose" or even "use case". This package can contain e.g. information about the part of the model that should be the focus for testing, or about the conditions where it is correct to stop testing (test stopping criteria).

An example of a model-based testing workflow. IXIT refers to "implementation extra information" and denotes here the total package of information that is needed when the abstract test suite is

Jagan Mohan converted into an executable one. Typically it includes information about test harness, data mappings and SUT configuration. Because test suites are derived from models and not from source code, model-based testing is usually seen as one form of black-box testing. In some aspects, this is not completely accurate. Model-based testing can be combined with source-code level test coverage measurement, and functional models can be based on existing source code in the first place. Model-based testing for complex software systems is still an evolving field. Models Especially in model-driven architecture the model is build before or parallel to the development process of the system under test. The model can also be constructed from the completed system. Recently the model is created mostly manually, but there are also attempts to create the model automatically, for instance out of the source code. Deriving Tests Algorithmically The effectiveness of model-based testing is primarily due to the potential for automation it offers. If the model is machine-readable and formal to the extent that it has a well-defined behavioral interpretation, test cases can in principle be derived mechanically. Often the model is translated to or interpreted as a finite state automaton or a state transition system. This automaton represents the possible configurations of the system under test. To find test cases, the automaton is searched for executable paths. A possible execution path can serve as a test case. This method works if the model is deterministic or can be transformed into a deterministic one. Depending on the complexity of the system under test and the corresponding model the number of paths can be very large, because of the huge amount of possible configurations of the system. For finding appropriate test cases, i.e. paths that refer to a certain requirement to proof, the search of the paths has to be guided. For the test case selection multiple techniques are applied. Test Case Generation by Theorem Proving Theorem proving has been originally used for automated proving of logical formulas. For modelbased testing approaches the system is modelled by a set of logical expressions (predicates) specifying the system's behavior. For selecting test cases the model is partitioned into

Jagan Mohan equivalence classes over the valid interpretation of the set of the logical expressions describing the system under test. Each class is representing a certain system behavior and can therefore serve as a test case. The simpliest partitioning is done by the disjunctive normal form approach. The logical expressions describing the system's behavior are transformed into the disjunctive normal form. The classification tree-method provides a more sophisticated hierarchical partitioning. Also partitioning heuristics are used supporting the partitioning algorithms, e.g. heuristics based on boundary value analysis. Test Case Generation by Constraint Logic Programming Constraint programming can be used to select test cases satisfying specific contraints by solving a set of contraints over a set of variables. The system is described by the means of contraints. Solving the set of constraints can be done by boolean solvers (e.g. SAT-solvers based on the boolean satisfiability problem) or by numerical analysis, like the Gaussian elimination. A solution found by solving the set of constraints formulas can serve as an test cases for the corresponding system. Test Case Generation by Model Checking Originally model checking was developed as a technique to check if a property of a specification is valid in a model. We provide a model of the system under test and a property we want to test to the model checker. Within the procedure of proofing if this property is valid in the model the model checker detects witnesses and counterexamples. A witness is a path, where the property is satisfied, a counterexample is a path in the execution of the model, where the property is violated. This paths can again be used as test cases. Test Case Generation by Symbolic Execution Symbolic execution is often used in frameworks for model-based testing. It can be a means in searching for execution traces in an abstract model. In principle the program execution is simulated using symbols for variables rather than actual values. Then the program can be executed in a symbolic way. Each execution path represents one possible program execution and can be used as a test case. For that, the symbols have to instantiated by assigning values to the symbols.

Jagan Mohan

Installation testing Installation testing (in software engineering) can simply be defined as any testing that occurs outside of the development environment. Such testing will frequently occur on the computer system the software product will eventually be installed on. While the ideal installation might simply appear to be to run a setup program, the generation of that setup program itself and its efficacy in a variety of machine and operating system environments can require extensive testing before it can be used with confidence. In distributed systems, particularly where software is to be released into an already live target environment (such as an operational web site) installation (or deployment as it is sometimes called) can involve database schema changes as well as the installation of new software. Deployment plans in such circumstances may include back-out procedures whose use is intended to roll the target environment back in the event that the deployment is unsuccessful. Ideally, the deployment plan itself should be tested in an environment that is a replica of the live environment. A factor that can increase the organisational requirements of such an exercise is the need to synchronize the data in the test deployment environment with that in the live environment with minimum disruption to live operation.

Usability testing Usability testing is a means for measuring how well people can use some human-made object (such as a web page, a computer interface, a document, or a device) for its intended purpose, i.e. usability testing measures the usability of the object. Usability testing focuses on a particular object or a small set of objects, whereas general human-computer interaction studies attempt to formulate universal principles. If usability testing uncovers difficulties, such as people having difficulty understanding instructions, manipulating parts, or interpreting feedback, then developers should improve the design and test it again. During usability testing, the aim is to observe people using the product in as realistic a situation as possible, to discover errors and areas of improvement. Designers commonly focus excessively on creating designs that look "cool", compromising usability and

Jagan Mohan functionality. This is often caused by pressure from the people in charge, forcing designers to develop systems based on management expectations instead of people's needs. A designers' primary function should be more than appearance, including making things work with people. Caution: simply gathering opinions is not usability testing -- you must arrange an experiment that measures a subject's ability to use your document. 1 Rather than showing users a rough draft and asking, "Do you understand this?", usability testing involves watching people trying to use something for its intended purpose. For example, when testing instructions for assembling a toy, the test subjects should be given the instructions and a box of parts. Instruction phrasing, illustration quality, and the toy's design all affect the assembly process. Setting up a usability test involves carefully creating a scenario, or realistic situation, wherein the person performs a list of tasks using the product being tested while observers watch and take notes. Several other test instruments such as scripted instructions, paper prototypes, and preand post-test questionnaires are also used to gather feedback on the product being tested. For example, to test the attachment function of an e-mail program, a scenario would describe a situation where a person needs to send an e-mail attachment, and ask him or her to undertake this task. The aim is to observe how people function in a realistic manner, so that developers can see problem areas, and what people like. The technique popularly used to gather data during a usability test is called a think aloud protocol. What to measure Usability testing generally involves measuring how well test subjects respond in four areas: time, accuracy, recall, and emotional response. The results of the first test are the baseline or control measurement; all subsequent tests are compared to the baseline. •

Time on Task -- How long does it take people to complete basic tasks? (For example, find something to buy, create a new account, and order the item.)

•

Accuracy -- How many mistakes did people make? (And were they fatal or recoverable with the right information?)

•

Recall -- How much does the person remember afterwards?

•

Emotional Response -- How does the person feel about the tasks completed? (Confident? Stressed? Would the user recommend this system to a friend?)

In the early 1990s, Jakob Nielsen, at that time a researcher at Sun Microsystems, popularized the concept of using numerous small usability tests -- typically with only five test subjects each -- at

Jagan Mohan various stages of the development process. His argument is that, once found that two or three people are totally confused by the home page, little is gained by watching more people suffer through the same flawed design. "Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford." 2. Nielsen subsequently published his research and coined the term heuristic evaluation. The claim of "Five users is enough" was later described by a mathematical model (Virzi, R.A., Refining the Test Phase of Usability Evaluation: How Many Subjects is Enough? Human Factors, 1992. 34(4): p. 457-468.) which states for the proportion of uncovered problems U U = 1 − (1 − p)

n

where p is the probability of one subject identifying a specific problem and n the number of subjects (or test sessions). This model shows up as an asymptotic graph towards the number of real existing problems (see figure below).

Jagan Mohan User acceptance testing User Acceptance Testing (UAT) is a process to obtain confirmation by an SME (subject matter expert), preferably the owner of the object under test, through trial or review, that the modification or addition meets mutually agreed-upon requirements. In software development, UAT is one of the final stages of a project and will often occur before a customer accepts a new system. Users of the system will perform these tests which, ideally, developers have derived from the User Requirements Specification, to which the system should conform. Test designers will draw up a formal test plan and devise a range of severity levels. These tests are not focused on fleshing out simple problems (spelling mistakes, cosmetic problems) or show stoppers (major problems like the software crashing, software will not run etc.). Developers should have worked out these issues during unit testing and integration testing. Rather, the focus is on a final verification of the required business function and flow of the system. The test scripts will emulate real-world usage of the system. The idea is that if the software works as intended and without issues during a simulation of normal use, it will work just the same in production.

Black box testing Black box testing, concrete box or functional testing is used in computer programming, software engineering and software testing to check that the outputs of a program, given certain inputs, conform to the functional specification of the program. The term black box indicates that the internal implementation of the program being executed is not examined by the tester. For this reason black box testing is not normally carried out by the programmer. In most real-world engineering firms, one group does design work while a separate group does the testing. A complementary technique, white box testing or structural testing, uses information about the structure of the program to check that it performs correctly. Equivalence partitioning

Jagan Mohan A technique in black box testing is equivalence partitioning. Equivalence partitioning is designed to minimize the number of test cases by dividing tests in such a way that the system is expected to act the same way for all tests of each equivalence partition. Test inputs would be selected from each partition. Equivalence partitions are designed so that every possible input belongs to one and only one equivalence partition. Disadvantages •

Doesn't test every input

•

No guidelines for choosing inputs

•

Heuristic based

•

very limited focus (by Rahul Saxena,IMT)

Boundary value analysis Boundary value analysis is a technique of black box testing in which input values at the boundaries of the input domain are tested. It has been widely recognized that input values at the extreme ends of, and just outside of, input domains tend to cause errors in system functionality. In boundary value analysis, values at and just beyond the boundaries of the input domain are used to generate test cases to ensure proper functionality of the system. As an example, for a system that accepts as input a number between one and ten, boundary value analysis would indicate that test cases should be created for the lower and upper bounds of the input domain (1, 10), and values just outside these bounds (0, 11) to ensure proper functionality. Boundary value analysis is an excellent way to catch common user input errors which can disrupt proper program functionality. Boundary value analysis complements the technique of equivalence partitioning . Some of the advantages of boundary value analysis are: •

Very good at exposing potential user interface/user input problems

•

Very clear guidelines on determining test cases

•

Very small set of test cases generated

Jagan Mohan Disadvantages to boundary value analysis: •

Does not test all possible inputs

•

Does not test dependencies between combinations of inputs

Smoke testing A sub-set of the black box test is the smoke test. A smoke test is a cursory examination of all of the basic components of a software system to ensure that they work. Typically, smoke testing is conducted immediately after a software build is made. The term comes from electrical engineering, where in order to test electronic equipment, power is applied and the tester ensures that the product does not spark or smoke.

White box testing White box testing, glass box testing or structural testing is used in computer programming, software engineering and software testing to check that the outputs of a program, given certain inputs, conform to the structural specification of the program. The term white box (or glass box) indicates that testing is done with a knowledge of the code used to execute certain functionality. For this reason, a programmer is usually required to perform white box tests. Often, multiple programmers will write tests based on certain code, so as to gain varying perspectives on possible outcomes. A complementary technique, black box testing or functional testing, performs testing based on previously understood requirements (or understood functionality), without knowledge of how the code executes.

Glass Box Testing Black-box and glass-box are test design methods. Black-box test design treats the system as a "black-box", so it doesn't explicitly use knowledge of the internal structure. Black-box test design is usually described as focusing on testing functional requirements. Glass-box test design allows

Jagan Mohan one to peek inside the "box", and it focuses specifically on using internal knowledge of the software to guide the selection of test data. Glass box testing requires the intimate knowledge of program internals, while black box testing is based solely on the knowledge of the system requirements. Being primarily concerned with program internals, it is obvious in SE literature that the primary effort to develop a testing methodology has been devoted to glass box tests. However, since the importance of black box testing has gained general acknowledgement, also a certain number of useful black box testing techniques were developed. It is important to understand that these methods are used during the test design phase, and their influence is hard to see in the tests once they're implemented. Glass box testing definition Software testing approaches that examine the program structure and derive test data from the program logic. Structural testing is sometimes referred to as clear-box testing since white boxes are considered opaque and do not really permit visibility into the code. Synonyms for Glass box testing o

White Box testing o

Structural testing o

Clear Box testing o

Open Box Testing

Types of Glass Box testing o

Static and Dynamic Analysis static analysis techniques do not necessitate the execution of the software, dynamic analysis is what is generally considered as ``testing``, i.e. it involves running the system.

o

Statement Coverage Testing performed where every statements is executed at least once.

o

Branch Coverage Running a series of tests to ensure that all branches are tested at least once.

Jagan Mohan o

Path Coverage Testing all paths.

o

All-definition-use-path Coverage All paths between the definition of a variable and the use of that definition are now identified and tested.

Tradeoffs of Glass box testing Advantages •

forces test developer to reason carefully about implementation

o

approximates the partitioning done by execution equivalence

o

reveals errors in "hidden" code:

o

beneficent side-effects

o

optimizations (e.g. charTable that changes reps when size > 100)

Disadvantages o

expensive

o

miss cases omitted in the code

Fuzz testing Fuzz testing is a software testing technique. The basic idea is to attach the inputs of a program to a source of random data. If the program fails (for example, by crashing, or by failing built-in code assertions), then there are defects to correct. The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior. Uses

Jagan Mohan Fuzz testing is often used in large software development projects that perform black box testing. These usually have a budget to develop test tools, and fuzz testing is one of the techniques which offers a high benefit to cost ratio. Fuzz testing is also used as a gross measurement of a large software system's quality. The advantage here is that the cost of generating the tests is relatively low. For example, third party testers have used fuzz testing to evaluate the relative merits of different operating systems and application programs. Fuzz testing is thought to enhance software security and software safety because it often finds odd oversights and defects which human testers would fail to find, and even careful human test designers would fail to create tests for. However, fuzz testing is not a substitute for exhaustive testing or formal methods: it can only provide a random sample of the system's behavior, and in many cases passing a fuzz test may only demonstrate that a piece of software handles exceptions without crashing, rather than behaving correctly. Thus, fuzz testing can only be regarded as a proxy for program correctness, rather than a direct measure, with fuzz test failures actually being more useful as a bug-finding tool than fuzz test passes as an assurance of quality. Fuzz testing was originated at the University of Wisconsin in 1989 by Professor Barton Miller and the students in his graduated Advanced Operating Systems class. Their work can be found at http://www.cs.wisc.edu/~bart/fuzz/. Fuzz testing methods As a practical matter, developers need to reproduce errors in order to fix them. For this reason, almost all fuzz testing makes a record of the data it manufactures, usually before applying it to the software, so that if the computer fails dramatically, the test data is preserved. Modern software has several different types of inputs: •

Event driven inputs are usually from a graphical user interface, or possibly from a mechanism in an embedded system.

•

Character driven inputs are from files, or data streams.

•

Database inputs are from tabular data, such as relational databases.

There are at least two different forms of fuzz testing:

Jagan Mohan •

Valid fuzz attempts to assure that the random input is reasonable, or conforms to actual production data.

•

Simple fuzz usually uses a pseudo random number generator to provide input.

•

A combined approach uses valid test data with some proportion of totally random input injected.

By using all of these techniques in combination, fuzz-generated randomness can test the undesigned behavior surrounding a wider range of designed system states. Fuzz testing may use tools to simulate all of these domains. Event-driven fuzz Normally this is provided as a queue of datastructures. The queue is filled with data structures that have random values. The most common problem with an event-driven program is that it will often simply use the data in the queue, without even crude validation. To succeed in a fuzz-tested environment, software must validate all fields of every queue entry, decode every possible binary value, and then ignore impossible requests. One of the more interesting issues with real-time event handling is that if error reporting is too verbose, simply providing error status can cause resource problems or a crash. Robust error detection systems will report only the most significant, or most recent error over a period of time. Character-driven fuzz Normally this is provided as a stream of random data. The classic source in UNIX is the random data generator. One common problem with a character driven program is a buffer overrun, when the character data exceeds the available buffer space. This problem tends to recur in every instance in which a string or number is parsed from the data stream and placed in a limited-size area. Another is that decode tables or logic may be incomplete, not handling every possible binary value. Database fuzz

Jagan Mohan The standard database scheme is usually filled with fuzz that is random data of random sizes. Some IT shops use software tools to migrate and manipulate such databases. Often the same schema descriptions can be used to automatically generate fuzz databases. Database fuzz is controversial, because input and comparison constraints reduce the invalid data in a database. However, often the database is more tolerant of odd data than its client software, and a general-purpose interface is available to users. Since major customer and enterprise management software is starting to be open-source, database-based security attacks are becoming more credible. A common problem with fuzz databases is buffer overrun. A common data dictionary, with some form of automated enforcement is quite helpful and entirely possible. To enforce this, normally all the database clients need to be recompiled and retested at the same time. Another common problem is that database clients may not understand the binary possibilities of the database field type, or, legacy software might have been ported to a new database system with different possible binary values. A normal, inexpensive solution is to have each program validate database inputs in the same fashion as user inputs. The normal way to achieve this is to periodically "clean" production databases with automated verifiers.