Software Testing Report

Software Testing 2009

1. Introduction In a software development project, errors can be creeping at any stage during the development. For each phase we have discussed different techniques for detecting and eliminating errors that originates in that phase. During testing, the program to be tested is executed with a set of test cases, and the output of the program for the test case is evaluated to determine if the program is performing as it is expected to. Due to its approach, dynamic testing can only as certain the presence of error in the program; the exact nature of the error is not usually decided by testing. Testing forms the first step in determining the errors in a program. Clearly the success of testing revealing errors in programs depends critically on the test cases.

Page | 1

Software Testing 2009

2. Need for Testing Testing is the process of running a system with the intention of finding errors.Testing enhances the integrity of a system by detecting deviations in design and errors in the system. Testing aims at detecting error-prone areas. This helps in the prevention of errors in a system. Testing also adds value to the product by conforming to the user requirements.

2.1 Causes of Errors The most common causes of errors in an e-commerce system are: • • •

Communication gaps between the developer and the business decision maker. Time provided to a developer to complete the project. Over commitment by the developer.

2.1.1 Communication gaps between the developer and the business decision maker A communication gap between the developer and the business decision maker is normally due to subtle differences between them. The difference can be classified into five broad areas: • • • • •

Thought processes Background and experience Interests Priorities Language

For example, an entrepreneur with a financial marketing background wants to set up an online share-selling site. The developer with an engineering back ground might be unaware of the intricate details of the financial market. Therefore the developer may not know to incorporate those features that can add value to other site visitors into the site such as the prices of mutual funds.

2.1.2Time provided to a developer to complete the project Page | 2

Software Testing 2009 A common source of errors in projects comes from time constraints in delivering a product. At best, a project schedule provides an educated guess that is based on what is known at the time. At worst, a project schedule is a wish-derived estimate with no basis in reality. Assuming the best, previously unknown problems may present themselves during development and testing. This can lead to problems maintaining the schedule. Failing to adjust the feature set or schedule when problems are discovered can lead to rushed work and flawed systems.

2.1.3Over commitment by the developer High enthusiasm can lead to over commitment by the developer. In these situations, developers are usually unable to add here to deadlines or equality due to the lack of resources or required skills on the team.

2.2 Objectives of Testing Testing is essential because of: • Software reliability • Software quality • System assurance • Optimum performance and capacity utilization • Price of non-conformance

2.2.1 Software reliability E-commerce requires software that performs critical tasks. Such as creating storefront and a shopping cart, collecting customer data, and providing the payment gateway. This software needs to function correctly. Testing assures the organization of the quality and integrity of the e-commerce solution.

2.2.2 Software quality Software quality is characterized by the correctness of program logic and implementation. It begins with testing the software during development. The developer must test each module to make sure that it functions correctly at the time it is written or modified. Test values and boundary conditions must both be verified. Next, the module Page | 3

Software Testing 2009 should undergo interface testing to check for functional errors. Only after the module works correctly can it be released for testing to the larger system. Early detection of errors saves rework and prevents a problem from becoming more complex in nature. As a result, error detection during the operation of a system incurs greater direct and indirect costs. At a higher level, the interaction of individually correct components must be tested. For example, if a customer enters the details of their credit card payment and are disconnected before the order confirmation, the software must indicate the status of the transaction when the customer reconnects to the e-commerce site. If the software functions otherwise, it does not meet organization requirements. Another instance of the quality of software is that of accurate tax and shipping calculations. Because all states have different tax systems and some of them are complex, it becomes difficult for the developer to integrate all the tax structures with multi-location shipping. This raises the complexity of the software and increases the chance of errors.

2.2.3 System assurance The main purpose of system assurance is to deliver a quality product. Conformance to requirements increases the organization’s confidence in the system. An e-commerce over the Internet is more critical than in real life. If the faith of any of these parties dwindles in the e-commerce site, the entrepreneurs can lose lot money, as well as their reputation. For example, in the case of a faulty e-commerce system, the credit card of the customer may be billed immediately for the complete order, when only a partial order has been filled. Testing must assure that partial order fulfillment and billing are done correctly.

2.2.4 Optimum performance and capacity utilization Another purpose of testing is to ensure optimum performance and capacity utilization of e-commerce system components. The purpose of stress or capacity testing/planning is to make sure that the web site is able to perform acceptably at peak usage. For example, during the Christmas shopping season the web site loads increase significantly. To handle this, the e-commerce solution must be able to handle the anticipated load with minimal degradation.

2.2.5 Price of non-conformance Page | 4

Software Testing 2009 The main purpose of testing is to detect errors and error-prone areas in a system. Testing must be through and well planned. A partially tested system is as bad as an untested system. And the price of an untested and under-tested system is high. The following list suggests some of the potential fallouts of an untested or under-tested any system: • Legal suits against the entrepreneur due to a faulty transaction system. This type of system may not have been tested for transaction software functionality. • Loss of critical data from the database can result in untraceable transactions. This could again invite legal action and losses due to the digression of the site visitors to competitors’ sites. • Insecure transactions can cause losses to customers and may result in the withdrawal of certification by the security certification agency. • System may breakdown if the system has not been tested. A system breakdown results in loss of time due to restoration of service. Fixing the error can involve both direct and indirect costs.

2.3 Best Practices While testing a system steps are to be followed: •

• • • • •

Prepare comprehensive test plan specifications and test cases for each level of testing. Supplement these with the test data and test logs. Test plans for system testing may involve operators and test plans because acceptance testing involves customers. Design the test cases to test system restrictions, such as file and database size (stress testing). Develop the data to test specific cases. Copies of live files must not be used except for Acceptance testing. Do not use confidential data for testing without written authorization, especially in the case of Acceptance testing. Follow relevant standards. Perform Regression testing on each component of the system. This ensures that no anomalies have crept into the system because of the changes made to the system. Page | 5

Software Testing 2009 • Make sure to document and set up the test environment for each level in advance of testing. Test environments specify the preconditions required to perform the tests. • Specify the intended test coverage as part of the test plan Test coverage is the degree to which Specific test cases address all specified requirements for a specific for a specific system or component.

3. Heuristics of Software Testing Software testability is how easily, completely and conveniently a computer program canbe tested.Software engineers design a computer product, system or program keeping in mind theproduct testability. Good programmers are willing to do things that will help the testingprocess and a checklist of possible design points, features and so on can be useful in negotiating with them. Here are the two main heuristics of software testing. 1. Visibility 2. Control

3.1 Visibility Visibility is our ability to observe the states and outputs of the software under test. Page | 6

Software Testing 2009 Features to improve the visibility are:

3.1.1 Access to Code Developers mustprovide full access (source code, infrastructure, etc) to testers.The Code, change records and design documents should be provided to thetesting team. The testing team should read and understand the code.

3.1.2 Event logging The events to log include User events, System milestones, Error handling andcompleted transactions. The logs may be stored in files, ring buffers in memory and/or serial ports. Things to be logged include description of event, timestamp,subsystem, resource usage and severity of event. Logging should be adjusted bysubsystem and type. Log file report internal errors, help in isolating defects, andgive useful information about context, tests, customer usage and test coverage.The more readable the Log Reports are, the easier it becomes to identify thedefect cause and work towards corrective measures.

3.1.3 Error detection mechanisms Data integrity checking and System level error detection (e.g. Microsoft Appviewer) are useful here. In addition, Assertions and probes with the followingfeatures are really helpful. Code is added to detect internal errors. Assertions abort on error. Probes log errors. Design by Contract theory---This technique requires thatassertions be defined for functions. Preconditions apply to inputand violations implicate calling functions while post-conditionsapply to outputs and violations implicate called functions. Thiseffectively solves the oracle problem for testing.

3.1.4 Resource Monitoring Memory usage should be monitored to find memory leaks. States of runningmethods, threads or processes should be watched (Profiling interfaces may beused for this.). In addition, the configuration values should be dumped.Resource monitoring is of particular concern in applications where the load onthe application in real time is estimated to be considerable.

3.2 Control Page | 7

Software Testing 2009 Control refers to our ability to provide inputs and reach states in the software under test.The features to improve controllability are:

3.2.1 Test Points Allow data to be inspected, inserted or modified at points in the software. It is especially useful for dataflow applications. In addition, a pipe and filters architecture provides many opportunities for test points. Custom User Interface controls:Custom UI controls often raise serious testability problems with GUI test drivers.

•

Ensuring testability usually requires: Adding methods to report necessary information Customizing test tools to make use of these methods •

Test Interfaces: Interfaces may be provided specifically for testing e.g. Excel and Xconq etc. Existing interfaces may be able to support significant testing e.g. Install Shield, AutoCAD, Tivoli, etc.

•

Fault injection: Error seeding---incrementing low level I/O code to simulate errors---makes itmuch easier to test error handling. It can be handled at both system andapplication level, Tivoli, etc. •

Installation and setup: Testers should be notified when installation has completed successfully. Theyshould be able to verify installation, programmatically create sample recordsand run multiple clients, daemons or servers on a single machine.

3.3 Categories of Heuristics of software testing 3.3.1 Observability What we see is what we test. Distinct output should be generated for each input Current and past system states and variables should be visibleduring testing All factors affecting the output should be visible. Incorrect output should be easily identified. Page | 8

Software Testing 2009 Source code should be easily accessible. Internal errors should be automatically detected (through self-testingmechanisms) and reported.

3.3.2 Controllability The better we control the software, the more the testing process can be automated and optimized. Check that All outputs can be generated and code can be executed throughsome combination of input. Software and hardware states can be controlled directly by thetest engineer. Inputs and output formats are consistent and structured. Test can be conveniently, specified, automated and reproduced.

3.3.3 Decomposability

By controlling the scope of testing, we can quickly isolate problems and perform effective and efficient testing. The software system should be built fromindependent modules which can betested independently.

3.3.4 Simplicity

The less there is to test, the more quickly we can test it.The points to consider in this regard are functional (e.g. minimum set of features), structural (e.g. architecture is modularized) and code (e.g. a codingstandard is adopted) simplicity.

3.3.5 Stability The fewer the changes, the fewer are the disruptions to testing.The changes to software should be infrequent, controlled and not invalidating existing tests. The software should be able to recover well from failures.

3.3.6 Understandability The more information we will have, the smarter we will test.The testers should be able to understand the design, changes to the design and the dependencies between internal, external and shared components.Technical documentation should be instantly accessible, accurate, well organized, specific and detailed.

3.3.7 Suitability Page | 9

Software Testing 2009 The more we know about the intended use of the software, the better we can organize our testing to find important bugs.The above heuristics can be used by a software engineer to develop a software configuration (i.e. program, data and documentation) that is convenient to test and verify.

4. Types of Testing Testing is usually applied to different types of targets in different stages of the software’s delivery cycle. The stages progress from testing small components (unit testing) to testing completed system (system testing).

4.1 Unit Test Unit test, implemented early in the iteration, focuses on verifying the smallest testable elements of the software. Unit testing is typically applied to components in the implementation model to verify that control flow and data flow are covered and function as expected. These expectations are based on how the components participate in executing a use case. The implementers perform unit test as the unit is developed. The detail of unit test is described in the implementation work flow.

4.2 Integration Test Integration testing is performed to ensure that the component in the implementation model operate properly when combined to execute a use case. The target-of-test is a package or a set of packages in the implementation model. Often the packages being combined come from different development organizations. Integration testing exposes incompleteness or mistakes in the package’s interface specifications.

4.3 System Test Page | 10

Software Testing 2009 System testing is done when the software is functioning as a whole or when- defined subsets of its behavior are implemented. The target in this case is the whole implementation model for the system.

4.4 Acceptance Test Acceptance testing is the final test action prior to deploying the software. The goal of Acceptance testing is to verify that the software is ready and can be used by the endusers to perform those functions and tasks the software was built to do. In the introduction to Test, it was stated that there is much more to testing software than testing only the functions, interface and response time characteristics of a target-of-test. Additional test must focus on characteristics/ attributes such as the target-as-test: • Integrity (resistance to failure) • Ability to be installed/executed on different platforms • Ability to handle many requests simultaneously In order to achieve this, many different tests are implemented and executed each with a specific test objective. Face focused on testing only one characteristics or attribute of the large-of-test. Often individual tests are categorized, implemented and executed in groups, most commonly arranged by similarities in their test objectives or the quality dimension they address, such as: Quality dimension Functionality

Type of Test •

•

Function test: tests focused on verifying the target-of-test functions as intended, providing the required service(s), method(s), or use case(s). This test is implemented and executed against different target-of-tests including units, integrated units, applications(s), and systems. Security test: tests focused on ensuring the target-of-test Page | 11

Software Testing 2009

•

Usability

•

Reliability

•

•

•

data (or systems) is accessible to only those actors intended. This test is implemented and executed various target-of-test. Volume test: testing focused on verifying the target-of-the ability to handle large amount of data, either as input and output or resident within the database. Volume testing includes test strategies such as creating queries that[would] return the entire contents of the database or have so many restrictions that no data is returned or data entry of the maximum amount of data in each field. Usability test: Tests which focus on:  Human factors,  Aesthetic,  Consistency in the user interface,  Online and context-sensitive help,  Wizards and agents,  User documentation and  Training materials Integrity testing: tests which focus on assessing the target-of-test’s (resistance to failure)and technical compliance to language syntax, and resource usage. This test is implemented and executed against different targetof-test including units and integrated units. Structure test: tests that focus on assessing the target-oftest’s adherence to its design and formation. Typically this test is done for web-enabled application ensuring that all links are connected. Appropriate content in displayed and there is no orphaned content. See concepts: structure testing for additional information. Stress test: a type of reliability test that focuses on ensuring the system functions add intended when abnormal conditions are encountered stresses on the system may include extremes workloads, insufficient memory, and unavailable services hardware or diminished shared resources. Typically, these tests are performed to determine when the system does break, how it breaks. Page | 12

Software Testing 2009 Performance

•

•

•

Benchmark test: a type of performance test that compares the performance of a [new or unknown] target-of-test’s to a known reference-workload and system. Contention test: test focused on verifying the target –oftest’s can acceptably handle multiple actor demands on the same resource (data, records, memory,etc…..) Load test: a type of performance test to verify and assess acceptability of the operational limits of a system under varying workloads while the system-under –test remains constant. Measurements include the characteristics of the work load and the response time. When system incorporate distributed architectures or load balancing, special tests are performed to ensure the distribution and the load balancing methods function appropriately.

Performance profile: a test in which the target-of-test’s timing profile is monition including execution flow, data access function and system calls to identify and address performance bottlenecks and inefficient process. Supportability

•

•

Configuration test: tests focused on ensuring the targetof-test functions as intended on different hardware and or software configuration. This test may also implement as a system performance test. Installation test: test focused on ensuring the target-oftest installs as intended on different hardware and or software configuration and under different condition (such as insufficient disk space or power interrupt). This test is implemented and executed against application(s) and system

Page | 13

Software Testing 2009

5. Test cases A Test case a set of test inputs execution conditions and expected results developed for a particular objective such as to exercise a particular program path or to verify compliance with a specific requirement. Unit testing is implemented against the smallest testable element(units) of the software and involves testing the internal structure such logic and data flow and the unit function and observable behaviors. Designing and implementing tests focused on a unit internal structure relies upon the knowledge of the unit’s implementation (white- box approach). The design and implementation of test s to verify the unit’s observable behaviors and functions does not rely upon knowledge of the implementation and therefore is known as black-box approach.

5.1White-Box Test Approach A white-box test approach should be taking to verify a unit’s internal structure. Theoretically, testers should test every possible path through the code but that is possible only in very simple units. At the very last testers should exercise every decision-to-decision path (dd-path) at least once because testers are then executing all statements at least once. A decision is typically an if-statement and a dd-path is a path between two decisions. To get this level of test coverage, it is recommended that testers should choose test data so that every decision is evaluated in every possible way. Reliability testing should be done simultaneously with the white box testing.

5.2 Black-Box Test Approach The purpose of a black-box test is to verify the unit’s specified function and observable behavior without knowledge of how the unit implements the function and behavior. Block-box tests focus and relay upon the unit’s input and output.Deriving unit test based upon the black-box approach utilized the input and output arguments of the unit’s operation and / or output state for evolution. For example,the operation may include an algorithm (requiring two values as input and return a third as output) or initiate change Page | 14

Software Testing 2009 in an object’s or component’s state such as adding or deleting a database record. Both must be tested completely. To test an operation you should derive sufficient test cases to verify the following: • For each valid used as input .an appropriate value was returned by the operation • For each valid input state , an appropriate output state occur • For each invalid input state , an appropriate output state occur

5.3Test cases based upon input Arguments An input argument is an argument used by an operation. By using input arguments test cases should be created for each operation, for each of the following input condition: ➢ Normal values form each equivalence class. ➢ Values on the boundary of each equivalence class. ➢ Values outside the equivalence classes. ➢ Illegal values. Remember to treat the object state as an input argument. If, for example, test an operation add on an object Set, test add with values from all of Set’s equivalence classes, that is, with a full Set, with some element in Set, and with an empty Set.

5.4 Test Cases based upon Output Arguments An output argumentis an argument that an operation changes. An argument can be both an input and an output argument. Select input so that output is according to each of the following. ➢ Normal values from each equivalence class. ➢ Values on the boundary equivalence class. ➢ Values outside the equivalence classes. ➢ Illegal values.

5.5 Test Cases for Regression Test Regression testing compares two builds or versions of the same target-of-test and identifies differences as potential defects. It thus assumes that a new version should behave like an earlier one and ensures that defects have not been introduced as a result of the changes. Ideally, you would like all the test cases inone iteration to be used as test cases in the later iterations. The following guidelines should be used to identify, design, and Page | 15

Software Testing 2009 implement test cases that maximize the value of regression testing and re-use. While minimizing maintenance: ➢ Ensures the test case identify only the critical data elements(those needed to create/support the condition being tested). ➢ Ensure each test case describes or represents a unique set of inputs or sequence of events that result in a unique behavior by the target-of-test. ➢ Eliminate redundant or equivalent test cases. ➢ Group together test cases which have the same target-of-test initial state and state of the test data.

Page | 16

Software Testing 2009

6. Finding Faults It is commonly believed that the earlier a defect is found the cheaper it is to fix it. For example, if a problem in the requirements is found only post-release, then it would cost 10–100 times more to fix than if it had already been found by the requirements review.

6.1 Testing Tools Program testing and fault detection can be aided significantly by testing tools and debuggers. Testing/debug tools include features such as: • Program monitors, permitting full or partial monitoring of program code including: ○ Instruction Set Simulator, permitting complete instruction level monitoring and trace facilities ○ Program animation, permitting step-by-step execution and conditional breakpoint at source level or in machine code ○ Code coverage reports • Formatted dump or Symbolic debugging, tools allowing inspection of program variables on error or at chosen points. • Automated functional GUI testing tools are used to repeat system-level tests through the GUI. • Benchmarks, allowing run-time performance comparisons to be made. • Performance analysis (or profiling tools) that can help to highlight hot spots and resource usage.

6.2 Measuring Software Testing Usually, quality is constrained to such topics as correctness, completeness, security but can also include more technical requirements such as capability, reliability, efficiency, portability, maintainability, compatibility, and usability.

6.3 Testing Artifacts Software testing process can produce several artifacts. Page | 17

Software Testing 2009 6.3.1 Test Plan A test specification is called a test plan. The developers are well aware what test plans will be executed and this information is made available to management and the developers. The idea is to make them more cautious when developing their code or making additional changes. Some companies have a higher-level document called a test strategy. 6.3.2 Traceability matrix A traceability matrix is a table that correlates requirements or design documents to test documents. It is used to change tests when the source documents are changed, or to verify that the test results are correct.

6.3.3 Test script The test script is the combination of a test case, test procedure, and test data. Initially the term was derived from the product of work created by automated regression test tools. Today, test scripts can be manual, automated, or a combination of both. 6.3.4 Test suite The most common term for a collection of test cases is a test suite. The test suite often also contains more detailed instructions or goals for each collection of test cases. It definitely contains a section where the tester identifies the system configuration used during testing. A group of test cases may also contain prerequisite states or steps, and descriptions of the following tests. 6.3.5 Test data In most cases, multiple sets of values or data are used to test the same functionality of a particular feature. All the test values and changeable environmental components are collected in separate files and stored as test data. It is also useful to provide this data to the client and with the product or a project. 6.3.6Test harness

Page | 18

Software Testing 2009 The software, tools, samples of data input and output, and configurations are all referred to collectively as a test harness.

Page | 19

Software Testing 2009

7. Conclusion Software testing is an art. Most of the testing methods and practices are not very different from 20 years ago. Good testing also requires a tester's creativity, experience and intuition, together with proper techniques. Testing is more than just debugging. Testing is not only used to locate defects and correct them. It is also used in validation, verification process, and reliability measurement. Testing is expensive. Automation is a good way to cut down cost and time. Testing efficiency and effectiveness is the criteria for coverage-based testing techniques. Complete testing is infeasible. Complexity is the root of the problem. At some point, software testing has to be stopped and product has to be shipped. The stopping time can be decided by the trade-off of time and budget. Or if the reliability estimate of the software product meets requirement.

Page | 20

Software Testing 2009

8. Bibliography Books ✔

An Integrated Approach to Software Engineering by Pankaj Jalote

Websites www.4shared.com ✔ www.google.com ✔ www.pdfcoke.com ✔ www.wikipedia.org ✔

Page | 21