Testing Types

Black box testing takes an external perspective of the test object to derive test cases. These tests can be functional or non-functional, though usually functional. The test designer selects valid and invalid input and determines the correct output. There is no knowledge of the test object's internal structure. This method of test design is applicable to all levels of software testing: unit, integration, functional testing, system and acceptance. The higher the level, and hence the bigger and more complex the box, the more one is forced to use black box testing to simplify. While this method can uncover unimplemented parts of the specification, one cannot be sure that all existent paths are tested.

Regression testing is any type of software testing which seeks to uncover regression bugs. Regression bugs occur whenever software functionality that previously worked as desired, stops working or no longer works in the same way that was previously planned. Typically regression bugs occur as an unintended consequence of program changes. Common methods of regression testing include re-running previously run tests and checking whether previously fixed faults have re-emerged.

Gray Box Testing: Grey box testing is the combination of black box and white box testing. Intention of this testing is to find out defects related to bad design or bad implementation of the system. In gray box testing, test engineer is equipped with the knowledge of system and designs test cases or test data based on system knowledge. For example, consider a hypothetical case wherein you have to test a web application. Functionality of this web application is very simple, you just need to enter your personal details like email and field of interest on the web form and submit this form. Server will get this details, and based on the field of interest pick some articles and mail it to the given email. Email validation is happening at the client side using Java Scripts. In this case, in the absence of implementation detail, you might test web form with valid/invalid mail IDs and different field of interests to make sure that functionality is intact. But, if you know the implementation detail, you know that system is making following assumptions • • •

Server will never get invalid mail ID Server will never send mail to invalid ID Server will never receive failure notification for this mail.

So as part of gray box testing, in the above example you will have a test case on clients where Java Scripts are disabled. It could happen due to any reason and if it happens, validation can not happen at the client site. In this case, assumptions made by the system are violated and • • •

Server will get invalid mail ID Server will send mail to invalid mail ID Server will receive failure notification

Hope you understood the concept of gray box testing and how it can be used to create different test cases or data points based on the implementation details of the system.

Integration Testing: Objective of Integration testing is to make sure that the interaction of two or more components produces results that satisfy functional requirement. In integration testing, test cases are developed with the express purpose of exercising the interface between the components. Integration testing can also be treated as testing assumption of fellow programmer. During the coding phase, lots of assumptions are made. Assumptions can be made for how you will receive data from different components and how you have to pass data to different components. During Unit Testing, these assumptions are not tested. Purpose of unit testing is also to make sure that these assumptions are valid. There could be many reasons for integration to go wrong, it could be because •

•

Interface Misuse - A calling component calls another component and makes an error in its use of interface, probably by calling/passing parameters in the wrong sequence. Interface Misunderstanding - A calling component makes some assumption about the other components behavior which are incorrect.

Integration Testing can be performed in three different ways based on the from where you start testing and in which direction you are progressing. • • • •

Big Bang Integration Testing Top Down Integration Testing Bottom Up Integration Testing Hybrid Integration testing

Top down testing can proceed in a depth-first or a breadth-first manner. For depthfirst integration each module is tested in increasing detail, replacing more and more levels of detail with actual code rather than stubs. Alternatively breadth-first would proceed by refining all the modules at the same level of control throughout the application. In practice a combination of the two techniques would be used. Entry Criteria

Main entry criteria for Integration testing is the completion of Unit Testing. If individual units are not tested properly for their functionality, then Integration testing should not be started. Exit Criteria Integration testing is complete when you make sure that all the interfaces where components interact with each other are covered. It is important to cover negative cases as well because components might make assumption with respect to the data. Smoke Testing: Smoke testing is done at the start of the application is deployed. Smoke test is the entry point for the entire test execution. When the application is passed under the smoke test then only further system testing or regression testing can be carried out. In general smoke testing is done when the higher version of the build is deployed and is done at each and every time the build is deployed. In smoke testing the main functionalites are tested and the stability of the system is validated. Sanity Testing: Sanity testing is also similar to Smoke testing, but has some minor differences. Sanity testing is done when the application is deployed into testing for the very first time and in smoke testing only positive scenarios are validated but in sanity testing both the positive and negative scenarios are validated. For example, if the new software is crashing systems every 5 minutes, bogging down systems to a crawl, or destroying databases, the software may not be in a 'sane' enough condition to warrant further testing in its current state. Ad-hoc testing Testing carried out using no recognised Test Case Design Technique which is a method used to determine Test Cases. Here the testing is done by the knolwedge of the tester in the application and he tests the system randomly with out any test cases or any specifications or requirements. Security Testing: Security Testing is very important in today's world, because of the way computer and internet has affected the individual and organization. Today, it is very difficult to imagine world without Internet and latest communication system. All these communication systems increases efficiency of individual and organization by multifold. Since every one from individual to organization, uses Internet or communication system to pass information, to do business, to transfer money it becomes very

critical for the service provider to make sure that information and network are secured from the intruders.

Primary purpose of security testing is to identify the vulnerabilities and subsequently repairing them. Typically, security testing is conducted after the system has been developed, installed and is operational. Unlike other types of testing, network security testing is performed on the system on the periodic basis to make sure that all the vulnerabilities of the system are identified. Network security testing can be further classified into following types • • • • • • • •

Network Scanning Vulnerability Scanning Password Cracking Log Review File Integrity Checkers Virus Detection War Dialing Penetration Testing

None of these tests provide a complete picture of network security. You will need to perform a combination of these techniques to ascertain status of your Network Testing Activities. Apart from Network Security Testing, you should also take care of Application security testing. Intruders can target specific applications for unauthorized access or for any malicious reason. It becomes even critical for the web applications, because of the visibility and access of the application through Internet. Web application Security Testing is covered in a different section. System testing: System testing is probably the most important phase of complete testing cycle. This phase is started after the completion of other phases like Unit, Component and Integration testing. During the System Testing phase, non functional testing also comes in to picture and performance, load, stress, scalability all these types of testing are performed in this phase. By Definition, System Testing is conducted on the complete integrated system and on a replicated production environment. System Testing also evaluates that system compliance with specific functional and non functional requirements both.

It is very important to understand that not many test cases are written for the system testing. Test cases for the system testing are derived from the architecture/design of the system, from input of the end user and by user stories. It does not make sense to exercise extensive testing in the System Testing phase, as most of the functional defects should have been caught and corrected during earlier testing phase. Utmost care is exercised for the defects uncovered during System Testing phase and proper impact analysis should be done before fixing the defect. Some times, if business permits defects are just documented and mentioned as the known limitation instead of fixing it. Progress of the System Testing also instills and build confidence in the product teams as this is the first phase in which product is tested with production environment. System Testing phase also prepares team for more user centric testing i.e User Acceptance Testing. Entry Criteria • • • •

Unit, component and Integration test are complete Defects identified during these test phases are resolved and closed by QE team Teams have sufficient tools and resources to mimic the production environment Teams have sufficient tools and resources to mimic the production environment

Exit Criteria • •

Test cases execution reports shows that functional and non functional requirements are met. Defects found during the System Testing are either fixed after doing thorough impact analysis or are documented as known limitations.

Integration Testing: Objective of Integration testing is to make sure that the interaction of two or more components produces results that satisfy functional requirement. In integration testing, test cases are developed with the express purpose of exercising the interface between the components.

Integration testing can also be treated as testing assumption of fellow programmer. During the coding phase, lots of assumptions are made. Assumptions can be made for how you will receive data from different components and how you have to pass data to different components. During Unit Testing, these assumptions are not tested. Purpose of unit testing is also to make sure that these assumptions are valid. There could be many reasons for integration to go wrong, it could be because •

•

Interface Misuse - A calling component calls another component and makes an error in its use of interface, probably by calling/passing parameters in the wrong sequence. Interface Misunderstanding - A calling component makes some assumption about the other components behavior which are incorrect.

Integration Testing can be performed in three different ways based on the from where you start testing and in which direction you are progressing. • • • •

Big Bang Integration Testing Top Down Integration Testing Bottom Up Integration Testing Hybrid Integration testing

Top down testing can proceed in a depth-first or a breadth-first manner. For depthfirst integration each module is tested in increasing detail, replacing more and more levels of detail with actual code rather than stubs. Alternatively breadth-first would proceed by refining all the modules at the same level of control throughout the application. In practice a combination of the two techniques would be used. Entry Criteria Main entry criteria for Integration testing is the completion of Unit Testing. If individual units are not tested properly for their functionality, then Integration testing should not be started. Exit Criteria Integration testing is complete when you make sure that all the interfaces where components interact with each other are covered. It is important to cover negative cases as well because components might make assumption with respect to the data. Performance test: Load test Load tests are performance tests which are focused on determining or validating performance characteristics of the product under test when subjected to workload models and load volumes anticipated during production operations.

What are the benefits? Helps • • •

Evaluate the adequacy of a load balancer. Detecting functionality errors under load. Determine the scalability of the application OR for capacity planning purposes as the need may be.

What risks does it address? • • •

How many users can the application handle before “bad stuff” happens How much data can my database/file server handle? Are the network components adequate?

Stress test Performance tests focused on determining or validating performance characteristics of the product under test when subjected to workload models, and load volumes beyond those anticipated during production operations. These tests are all about determining under what conditions an application will fail how it will fail and what indicators can be monitored to warn of an impending failure. What are the benefits? Helps • • • • •

Determining if data can be corrupted by over stressing the system Estimating how far beyond the target load an application can go before causing failures and errors in addition to slowness Establishing application monitoring triggers to warn of impending failures Ensuring that security holes are not opened up by stressful conditions. Determining the side effects of common hardware or supporting application failures.

What risks does it address? • • •

What happens if we underestimated the peak load? What kind of failures should we plan for? What indicators should we be looking for to intervene prior to failure?

Equivalence Class Partitioning (ECP): An Equivalence Class Partitioning (ECP) approach divides the input domain of a software to be tested into the finite number of partitions or eqivalence classes. This method can be used to partition the output domain as well, but it is not commonly used. Boundary value testing (BVA) is a technique to find whether the application is accepting the expected range of values and rejecting the values which falls out of range. Ex. A user ID text box has to accept alphabet characters ( a-z ) with length of 4 to 10 characters.

BVA is done like this, max value = 10 pass max-1 = 9 pass max+1 = 11 fail min = 4 pass min+1 = 5 pass min-1 = 3 fail. Like wise we check the corner values and come out with a details whether the application is accepting correct range of values. Sanity testing: Sanity testing is

product or project is sane enough(Marked by

sound judgment) or not ? just checking the major functionality navigations, screens displaying or not

Retesting: Is a type of testing that is performed to check for the functionality of an application by using different inputs after the fix is done for the bugs that were recorded during the earlier testing. Fuzz testing or fuzzing: It is a software testing technique that provides random data ("fuzz") to the inputs of a program. If the program fails (for example, by crashing, or by failing built-in code assertions), the defects can be noted. The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior. Fuzz testing was developed at the University of Wisconsin-Madison in 1989 by Professor Barton Miller and the students in his graduate Advanced Operating Systems class. Bug Life Cycle & Guidelines In this tutorial you will learn about Bug Life Cycle & Guidelines, Introduction, Bug Life Cycle, The different states of a bug, Description of Various Stages, Guidelines on deciding the Severity of Bug, A sample guideline for assignment of Priority Levels during the product test phase and Guidelines on writing Bug Description. Introduction: Bug can be defined as the abnormal behavior of the software. No software exists without a bug. The elimination of bugs from the software depends upon the efficiency of testing done on the software. A bug is a specific concern about the quality of the Application under Test (AUT). Bug Life Cycle: In software development process, the bug has a life cycle. The bug should go through the life cycle to be closed. A specific life cycle ensures that the process is standardized. The bug attains different states in the life cycle. The life cycle of the

bug can be shown diagrammatically as follows: The different states of a bug can be summarized as follows: 1. New 2. Open 3. Assign 4. Test 5. Verified 6. Deferred 7. Reopened 8. Duplicate 9. Rejected and 10. Closed Description of Various Stages: 1. New: When the bug is posted for the first time, its state will be “NEW”. This means that the bug is not yet approved. 2. Open: After a tester has posted a bug, the lead of the tester approves that the bug is genuine and he changes the state as “OPEN”. 3. Assign: Once the lead changes the state as “OPEN”, he assigns the bug to corresponding developer or developer team. The state of the bug now is changed to “ASSIGN”. 4. Test: Once the developer fixes the bug, he has to assign the bug to the testing team for next round of testing. Before he releases the software with bug fixed, he changes the state of bug to “TEST”. It specifies that the bug has been fixed and is released to testing team. 5. Deferred: The bug, changed to deferred state means the bug is expected to be fixed in next releases. The reasons for changing the bug to this state have many factors. Some of them are priority of the bug may be low, lack of time for the release or the bug may not have major effect on the software. 6. Rejected: If the developer feels that the bug is not genuine, he rejects the bug. Then the state of the bug is changed to “REJECTED”. 7. Duplicate: If the bug is repeated twice or the two bugs mention the same concept of the bug, then one bug status is changed to “DUPLICATE”. 8. Verified: Once the bug is fixed and the status is changed to “TEST”, the tester tests the bug. If the bug is not present in the software, he approves that the bug is fixed and changes the status to “VERIFIED”. 9. Reopened: If the bug still exists even after the bug is fixed by the developer, the tester changes the status to “REOPENED”. The bug traverses the life cycle once again. 10. Closed: Once the bug is fixed, it is tested by the tester. If the tester feels that the bug no longer exists in the software, he changes the status of the bug to

“CLOSED”. This state means that the bug is fixed, tested and approved. While defect prevention is much more effective and efficient in reducing the number of defects, most organization conducts defect discovery and removal. Discovering and removing defects is an expensive and inefficient process. It is much more efficient for an organization to conduct activities that prevent defects. Guidelines on deciding the Severity of Bug: Indicate the impact each defect has on testing efforts or users and administrators of the application under test. This information is used by developers and management as the basis for assigning priority of work on defects. A sample guideline for assignment of Priority Levels during the product test phase includes: 1. Critical / Show Stopper — An item that prevents further testing of the product or function under test can be classified as Critical Bug. No workaround is possible for such bugs. Examples of this include a missing menu option or security permission required to access a function under test. . 2. Major / High — A defect that does not function as expected/designed or cause other functionality to fail to meet requirements can be classified as Major Bug. The workaround can be provided for such bugs. Examples of this include inaccurate calculations; the wrong field being updated, etc. . 3. Average / Medium — The defects which do not conform to standards and conventions can be classified as Medium Bugs. Easy workarounds exists to achieve functionality objectives. Examples include matching visual and text links which lead to different end points. . 4. Minor / Low — Cosmetic defects which does not affect the functionality of the system can be classified as Minor Bugs.