An Introduction To Testing Life Cycle Process

An Introduction to Testing Life Cycle Process

Contents  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle Overview  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle Overview •Normally, testing of any Large Systems will be in TWO parts. The functional verification and validation against the Requirement Spec and Performance evaluation against the indicated requirements Testing activity is involved right from the beginning of the project V&V Process model to successfully deliver a project on time This model is followed for a complete project life cycle model

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - V&V Process Model Business requirements, Solution Architecture

Release

Master Test Strategy

Project Planning

HLD/LLD for each application involved Interface design

Test Planning

Strategy for individual applications, Test case design

Solution Mapping/ Development/ Customization

Integrated Solution

Application Integration

System Testing (Application level Testing)

Acceptance Tests & Certification

Performance Testing, Business Cycle testing

Incremental Integration Testing

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - V&V Process Concept The "V &V " concept relates the build components of development to the test components that occur during that build phase. Testing related activities during Requirement phase Creation and finalization of testing templates Creation of over-all Test Plan and Test Strategy Capturing Acceptance criteria and preparation of Acceptance Test Plan Capturing Performance criteria of the software requirements

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - V&V Process Concept … • Testing activities in System Testing phase – System test is done for validating the product with respect to client requirements – Testing can be in multiple rounds – Defect found during system test should be logged into Defect Tracking System for the purpose of tracking. – Test logs and defects are captured and maintained.  Testing Life Cycle – Review of all the test documents Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - V&V Process Concept… Testing activities in Design phase Develop Test cases to ensure that product is on par with Requirement Specification document. Verify Test Cases & test scripts by peer reviews. Preparation of traceability matrix from system requirements  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - V&V Process Concept … • Testing activities in Integration Testing Phase – This testing is conducted in parallel with integration of various applications (or components) – Testing the product with its external and internal interfaces without using drivers and stubs. – Incremental approach while integrating the interfaces.  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - V&V Process Concept … • Performance Testing – This is done to validate the performance criteria of the product / application. This is non-functional testing • Business Cycle testing – this refers to end to end testing of real life-like business scenarios. • Testing activities during Release phase – Acceptance testing is conducted at the customer location. – Resolves all defects reported by the customer during Acceptance Testing – Conduct Root Cause Analysis (RCA) for those defects reported by customer during acceptance testing

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - Team Structure • An effective testing team includes a mixture of members who has – Testing expertise – Tools expertise – Database expertise – Domain/Technology expertise

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - Team Structure (Contd…) • The testing team must be properly structured, with defined roles and responsibilities that allow the testers to perform their functions with minimal overlap. • There should not be any uncertainty regarding which team member should perform which duties. • The test manager will be facilitating any resources required for the testing team.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - Roles and Responsibilities • Clear Communication protocol should be defined with in the testing team to ensure proper understanding of roles and responsibilities. • The roles chart should contain both on-site and off-shore team members.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - Roles and Responsibilities • Test Manager – Single point contact between onsite and offshore team – Prepare the project plan – Test Management – Test Planning – Interact with onsite lead, Client QA manager – Team management – Work allocation to the team – Test coverage analysis

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - Roles and Responsibilities • Test Manager cont.. – Co-ordination with onsite for issue resolution. – Monitoring the deliverables – Verify readiness of the product for release through release review – Obtain customer acceptance on the deliverables – Performing risk analysis when required – Reviews and status reporting – Authorize intermediate deliverables and patch releases to customer.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - Roles and Responsibilities • Test Lead – – – – – – – – –

Resolves technical issues for the product group Provides direction to the team members Performs activities for the respective product group Review and Approve of Test Plan / Test cases Review Test Script / Code Approve completion of Integration testing Conduct System / Regression tests Ensure tests are conducted as per plan Reports status to the Offshore Test Manager

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - Roles and Responsibilities • Test Engineer – – – –

Development of Test cases and Scripts Test Execution Result capturing and analysing Defect Reporting and Status reporting  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - Term Definitions • Testing: It is a process of executing software in a controlled manner, in order to answer the question “Does the software behave as specified?” • Verification: Says “Are we building the product right?” All QC activities throughout the life cycle that ensure that interim deliverables meet their input specification. • Validation: Says “Are we building the right product?” The test phase of the life cycle which assures that the end product meets the user’s needs. • Test Case: “An input operation and the corresponding expected output” in order to test a small unit of work. • Test Script: “A logical group of test cases which, when taken together, test a particular function or unit of a system”

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - Term Definitions (Contd.)

• Test Bed: “A group of test scripts which, when taken together, test all functions of an entire system” • Audit: An inspection/assessment activity that verifies compliance with plans, policies and procedures • Baseline: A quantitative measure of the current level of performance • Bug: A catch all term for all software defects or errors • Certification: Acceptance of a software by an authorized agent or its validity been demonstrated to an authorized agent • Cost of Quality (COQ): Money spent above and beyond expected production costs, this involves prevention, appraisal, internal & external failure costs • Cyclomatic Complexity: The number of decision statements plus one.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - Term Definitions (Contd.)

• Defect: It is “A deviation from specification or standard” or “Any thing that causes customer dissatisfaction”. • Desk Check: Verification technique conducted by the author of the artifact to verify the completeness • Dynamic Assertion: A dynamic analysis technique that inserts into the program code assertions about the relationship between program variables • Force field Analysis: A group technique used to identify both driving and restraining forces that influence a current situation • Inspection: A formal assessment of a work product conducted by one or more qualified independent reviewers to detect defects, violation of development standards, etc. Inspection identifies defects but does not attempt to correct them.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - Term Definitions (Contd.)

• Instrumentation: The insertion of the additional code into a program to collect information about a program behavior during program execution. • Life Cycle Testing: The process of verifying the consistency, completeness and correctness of software at each phases of the development life cycle. • Phase Containment: Method of control put in place within each stage of the development process to promote error identification and resolution so that defects are not propagated downstream to subsequent stages of the development process • Productivity: The ratio of the output of a process to the input

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Testing Life Cycle - Term Definitions (Contd.)

• Quality Assurance: The set of activities (including facilitation, training, measurement, and analysis) needed to provide adequate confidence that process are established continuously improved to produce products that meet specifications and are fit for use. • Quality Control: The process by which product quality is compared and detected w.r.t requirements and other relevant specifications, focus is in detection and removal • Test Driver: A program that directs the execution of another program against a collection of test data sets • Test item: A software item that is an object of testing • User: The customer that actually uses the product received

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Strategy  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Strategy • Test strategy is statement of overall approach of testing to meet the business and test objectives. • It is a plan level document and has to be prepared in the requirement stage of the project. • It identifies the methods, techniques and tools to be used for testing . • It can be a project or an organization specific. • Developing a test strategy which effectively meets the needs of the organization/project is critical to the success of the software development • An effective strategy has to meet the project and business objectives • Defining the strategy upfront before the actual testing helps in planning the test activities

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Elements of Test Strategy • A test strategy will typically cover the following aspects – – – – – – – – –

Definition of test objective Strategy to meet the specified objective Overall testing approach Test Environment Test Automation requirements Metric Plan Risk Identification, Mitigation and Contingency plan Details of Tools usage Specific Document templates used in testing

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Strategy - Project/Business Objectives • All the business and test objectives are identified and test strategy shall be defined in order to meet those objectives. • As far as possible all the test objectives to be met are defined in quantitative terms in the test strategy. • Examples of test objectives can be like 90 % of code coverage, finding at least a minimum of 100 defects etc.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Strategy - Test Approach • Test approach will be based on the objectives set for testing • Test approach will detail the way the testing to be carried out • Types of testing to be done viz Unit, Integration and system testing • The method of testing viz Black–box, Whitebox etc., • Details of any automated testing to be done

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Strategy - Test Environment • All the Hardware and Software requirements for carrying out testing shall be identified in detail. • Any specific tools required for testing will also be identified • If the testing is going to be done remotely, then it has to be considered during estimation

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Strategy - Metric Plan • Metric plan will contain the measures that will indicate the effectiveness of testing . • All the metrics to be tracked for testing like defect density ,residual defect density ,code coverage etc are identified in test strategy . • The objectives set for testing are also measured in the metric plan.

Note: More details are covered in Test Execution section

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Strategy - Risks • Risk analysis should carried out for testing phase • The risk identification will be accomplished by identifying causes-and-effects or effects-and-causes • The identified Risks are classified into to Internal and External Risks. The internal risks are things that the test team can control or influence. The external risks are things beyond the control or influence of the test team • Once Risks are identified and classified, the following activities will be carried out – – – –

Identify the probability of occurrence Identify the impact areas – if the risk were to occur Risk mitigation plan – how avoid this risk? Risk contingency plan – if the risk were to occur what do we do?

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Strategy - Tools Usage • Tools required in the project and rationale to use that particular tool(s) should be documented. • Automation requirements should be identified and analyzed – Which phases need to be automated – How much of testing needs to be automated

• Based on the automation requirements, tools strategy should be prepared.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Strategy - Templates • Any specific templates to be used for testing apart from the normal templates based on client or project requirements are identified in the test strategy. • In some case client provides the templates and we’ll use that.

Sample Strategy doc

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Strategy - Acceptance criteria • Acceptance test plan can be prepared either by us or by the customer, as agreed. • Customer will be involved in preparation, review and approval of acceptance test cases • This plan specifies the criteria for client acceptance of the final tested product including Features/functionalities to be tested and traceability information.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Strategy - Summary • Test strategy is a high level document which states in detail way testing is to be carried out. • It is done in line to meet the test and business objectives • Tools ,Methods ,and techniques to be used are stated in the test strategy.  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning - Documenting assumptions • If any assumptions are there, it needs to be documented to avoid any confusions – Testers has understanding of V&V-process model or any SDLC model – Testers are aware of test phases of the project – A good understanding of the Domain – Standard naming conventions are followed in documentation

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning -Traceability • Requirements tracing is the process of documenting the links between the user requirements for the system you are building and the work products developed to implement and verify those requirements. • Work products include software requirements, design specifications, software code, test plans and other artifacts of the systems development process

Traceability Matrix

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning -Traceability • Traceability is documenting of – the dependencies and logical links between individual requirements and other system elements – traceability to greatly facilitate change impact analysis. – traceability is that it can help identify the propagation of change that can result when a specific requirement is deleted or modified. – specific requirements mapped into tasks in a workbreakdown structure, you know those tasks that will be affected if and when a requirement is changed or deleted.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning - Schedule/ Milestones • Test plan captures the details about different milestones of the project and the schedule details. • Define the schedule for each event for offshore and onsite. Events may include familiarization, training, test case preparation, test scripting, test execution etc., • Completion criteria for each milestone should be identified and documented

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning - Resources • Staffing: – Check what types of personnel should be considered for inclusion on the test team – Check for the skills of the tester – Ensure a good ratio of testers to developers. • Steps to form the best team: – Identify potential test team members – Recruit test team members and develop tentative test assignments – Define individual work assignments

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning - Test run plan Release Plan 4 Cumulative TCs Executed

• The execution order of test cases depends on many factors and is shown as below:

1400 1200 1000 800 600 400 200 0 7/4

Impacted Functionality

7/5

7/6

7/7

7/8

7/9

7/10 7/11

7/12 7/13

7/14

7/15 7/16

7/17

4-Jul

5-Jul

6-Jul

9-Jul

10-Jul

11-Jul

12-Jul

13-Jul

16-Jul

17-Jul

Plan

106

257

575

669

767

877

983

1065

1119

1119

Actual

297

483

632

737

855

939

1015

1079

1140

1147

# of Days

Unstable functionality

TC priority

Execution Order

Customer Priority

Complexity

Execution rate

Functionality Dependency

TC Interdependency

Execution Order

Test Execution Run Plan

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning - Change management • Version control - Versioning test cases before and after release to customer • Change control - Authority to approve and prioritize changes • Configuration auditing - Ensure changes made properly • Reporting • Addition/Updating of test cases/plans because of bugs which are identified by the customer and that are not available in test plans

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning – Suspension and Resumption criteria

• Deadlines (release/testing deadlines etc.) are met and quality of the product is as per the agreed norms. • Test cases completed with certain percentage passed • Test budget depleted • Coverage of code/functionality/requirements reaches specified point • Beta or Alpha testing period ends

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning – Suspension and Resumption criteria

Determination of Test Completion criteria • Most popular method of establishing a test phase completion criteria is by detecting a predefined number of bugs or when a pre-defined time elapses • E.g.: Unit testing of a unit is not deemed complete unless at least 5 errors are detected • Testing completion criteria could be detection and fixing of 30 bugs, or elapsed time of 30 days, whichever is later.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning – Suspension and Resumption criteria

Estimation of number of errors

• How to determine the number of errors to be detected? – An estimate of total number of errors in the program. – An estimate of what percentage of these errors can be feasibly found through testing. – Estimates of what fraction of errors originated in a particular phase, during what testing phases these are likely to be detected.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning – Suspension and Resumption criteria

• Methods of estimation

– Rough estimate of total number of errors can be obtained by previous experience on similar programs. – Error-seeding techniques – involves intention introduction (seeding) of error in the code. After some period of testing time, the product of the number on non-seeded errors found during testing and the ratio of total seeded errors to the number of seeded errors detected during testing, gives an estimate of the number of non-seeded errors remaining in the program.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning – Suspension and Resumption criteria

• Estimation of errors in a particular phase - Example

– Existing data reveals that in large programs, 40% of errors are coding and logic-design mistakes and the rest are generated in earlier phases. – Example to illustrate estimation of total number of errors – – We have a 10,000 statement program, and the number of errors remaining in the program after code review is 5 per 100 statements. Our objective is to find 98% of coding and logic design errors and 95% design errors. Therefore• • • • • •

Total number of estimated errors = 500 Coding and Logic design errors = 200 (40% of 500) Design errors = 300. We need to find at least: 196 Coding and Logic Design errors 285 Design errors.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning – Suspension and Resumption criteria Estimation of error distribution in different phases

Unit Test

Coding & Logic Design errors

Design Errors

65%

0%

Module 30% Integration Test

60%

System Test

3%

35%

Total

98%

95%

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning – Suspension and Resumption criteria

Establishment of completion criteria • Exit Criteria for Unit testing would be detection and correction of 130 errors (65% of estimated 130 coding and design logic errors) • Exit Criteria for Module Integration testing would be detection and correction of 240 errors (30% of 200 plus 60% of 300) or elapse of one month; which ever is later. • Exit Criteria for System Testing is when 111 errors are detected or when 2 months elapses; which ever is later. Assumption: We have established one month for Integration Testing and 2 months for System Testing.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning – Suspension and Resumption criteria

• Resolution in case of over estimation of defects • In the previous example, let us say that 240 errors could not be detected even after one month of integration testing is completed. Then the project manager should take the following action – With the help of the appropriate tool, he/she should analyze the test cases and decide whether • There are inadequate Test cases, or • Excellent Test cases, but lack of errors to detect

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

•

Determination of completion criteria through a graphical representation A graph is plotted using number of errors found per unit time during a testing phase. By examining the shape of the curve, one can determine whether to continue the test phase or end it. Assume a program is being Integration tested, and the number of errors per day is plotted. Lets say on the 7h day the graph looks as follows:

Number of errors

• •

Test Planning – Suspension and Resumption criteria

No. of days

7

AT the end of 7th day, even if the number of errors detected has crossed the criteria mark, it would be wise to continue Integration Testing. This is because, the error rate is still very high.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning – Suspension and Resumption criteria

Number of errors

• Determination of completion criteria through a graphical representation • On the hand if the graph looks as shown, then this shows that error rate has decreased, and that testing can be stopped.

No. of days

7

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Planning – Document templates • Typical Test Plan – – – – – – – – – – – – – – –

Test Plan identifier Introduction Test items Features to be tested Features not to be tested Approach Testing strategy Item pass/fail criteria Suspension criteria and resumption criteria Test deliverable Automation scope Testing tasks Effort estimation Environmental needs Roles and Responsibilities

testplan template

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Designing  Test Execution  Testing Types  Testing Tools Overview

Test Case Design  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Case Design (Contd…) Development Of Test Objectives: • At very First stage of testing , you need to set the goals and objectives of what you need to achieve. • This Simplifies the limits or borders of testing process. • Test Objectives will be contain the list of tasks you are going to do and the achievements. Test Case Writing: • A specific set of steps and data along with expected results for a particular test objective. A test case should only test one limited subset of a feature or functionality.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Case Design (Contd…) A typical test case sheet contains these fields… • Test Case ID: It is unique number given to test case in order to be identified. • Test description: The description if test case you are going to test. • Revision history: Each test case has to have its revision history in order to know when and by whom it is created or modified. • Function to be tested: The name of function to be tested. • Priority/Build/time: The priority of the Test Case, Build version of the test case and the Minimum timeto-execute the test case is also mentioned.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Case Design (Contd…) • Environment: It tells in which environment you are testing. • Test Setup: Anything you need to set up outside of your application for example printers, network etc., • Test Execution: It is detailed description of every step of execution. • Expected Results: The description of what you expect the function to do. • Actual Results: Pass /Failed. If pass – What actually happens when you run the test If failed - Description of what you've observed.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Case Design (Contd…) Characteristics of a Good Test: • Tests are likely to catch bugs • No redundancy • Not too simple or too complex. TIP: Test case is going to be complex if you have more than one expected results.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Case Design (Contd…) Test Data Inputs: • Before testing, the tester should plan what kind of data he is giving for test. • Give data inputs as functional, boundary, stress, performance, usability values etc.  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Case Design (Contd…) Test Scripts: • A test script is the executable form of a test. • It defines the set of actions to carry out in order to conduct a test and it defines the expected outcomes and results that are used to identify any deviance in the actual behavior of the program from the logical behavior in the script (errors during the course of that test). • In essence it is a program written for a human computer (tester) to execute. Tes tCas e Template

Tes tCas e Status Template

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution • Test execution is culmination of testing activities which involves executing the planned test cases and conducting of the tests. • Test execution phase broadly involves execution and reporting.  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution - Activities • Test execution consists of following activities to be performed – – – – –

Creation of test setup or Test bed Execution of test cases on the setup Test Methodology used Collection of Metrics Defect Tracking and Reporting

– Regression Testing

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution Cycle

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution - Testing • Only exhaustive testing can show a program that it is defect free ,but exhaustive testing is impossible • Tests should exercise the systems capabilities fully • Testing typical situations is more important than boundary value conditions  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution -Test Data and Test Cases

• Test Cases: Inputs to test the system and predicted output from the system are detailed, further will contain the expected result and test topology • Test Data: Inputs which are to exercise the system under test.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution -Types of Testing and Methodologies • There are three major types of testing and methods used are based on the testing type – Unit testing – Integration testing – System testing

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution - Unit Testing • Unit testing is typically done as white box testing. • Unit testing is done at module level. • Derivation of Test cases are based on the program structure .Knowledge of the program is used to identify the test cases . • Objective is to exercise all parts of the program . • Test coverage measurements are done to verify that all program parts are exercised • Drivers are used to feed the data to the program to test it, wherever required. • Test stubs are used to stub for the functions which are outside the module

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution - Integration Testing

• Tests partial systems composed of integrated components • Integration testing is black box testing • Main difficulty in integration testing is localizing errors • There two approaches to integration testing – Top-down approach – Bottom - up approach

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Integration testing- Top Down approach • Start from the top level module and integrate individual components and stub the rest of the system

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Integration testing - Bottom – Up approach • Integrate individual components from lower level module until whole system is integrated .

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution - System Testing

• System testing is typically “Black Box “ testing where the system is taken as black –box and tested based only on its functionality • Test cases are based on the system specification • It is critical part of the testing process as the system is verified as a whole  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution - Regression Testing

• Regression testing is done after fixing all the defects. • It is re-testing after fixing the bugs found during testing and also when there is change in the test environment. • The impacted portions of the code after fixing the bug is re-tested. • Automation of testing helps in faster regression testing.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

• • • • •

Test Execution - Defect Tracking and Reporting All the defects/bugs found during testing is

logged and they are classified based on the severity and priority. They are assigned to developers for fixing. Each defect found is logged and tracked to closure. Test summary report shall be prepared containing the details like numbers of test cases passed/failed ,number of cycles of testing. Generally test defect logging and tracking tools are used for this purpose.

Test report template

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution - Start / Stop Criteria

• Testing is done in rounds. Software passes though several rounds before it gets released. • Testing is stopped when there are many major bugs and critical system errors, when further testing is not feasible • Testing is stopped when the system becomes un stable during test. • When those critical errors are fixed and system is stable, the next round of testing is started . • The test cases are executed all over again in each round.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution - Metrics • Metrics are important to collect for any project as it helps in monitoring the health of project /product /application under test. • Metrics are collected during testing depending on the objective of testing. • There are different kinds of metrics which can be used in testing. – For example LOC (Lines of Code) is a metric for Unit testing. i.e., Code related metrics comes in the category of Unit testing.

• Test case related metrics can be used for all kinds of testing types, since for every type of test, we will write test cases. • Normally source of errors/defects is requirements, design or Implementation. Testing can not be the source of errors/defects.

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution - Metrics • Typical testing metrics – – – – – – – – – – – –

No of LOC tested per day No of test cases executed per day Number of test cases failed Number of bugs reported Number of rejects of bugs reported Number of persons working on the test plan during the month Total Effort spent on test execution in person hours Number of simulations made during the month Number of out of norms per month Average time taken to find a problem categorized by priority Defect density Review efficiency

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution - Metrics • Having a complete Metrics data helps in building the “Trustworthy, Capable, Reliable and Predictable” Product or Application. – – – – – – – – –

Productivity Increase Field error rate Decrease Customer Overhead reduces Improvement release on release On time delivery Risks in Program identified and proactively intimated Decrease in Cost of Quality Decrease in Re-work Helps in identifying the area which needs focus in the project/organization for improvement – Helps in arriving at better estimates for the project

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution – Good test engineer?

• What makes a good test engineer? – – – – – – – – – – –

A good test engineer has a 'test to break' attitude. An ability to take the point of view of the customer A strong desire for quality, and an attention to detail. Tact and diplomacy for maintaining a cooperative relationship with developers. An ability to communicate with both technical (developers) and nontechnical (customers, management) people. Previous software development experience can be helpful, gives the tester an appreciation from the developers' point of view, and reduce the learning curve in automated test tool programming. Judgment skills are needed to assess high-risk areas of an application on which to focus testing efforts when time is limited. Notice little things that others miss/ignore (See symptom not bug) Good testers use files, Databases and all the other accoutrements of an organized mind Testers are fundamentally honest and incorruptible Last but not the least is “Patience”

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution – Testing Rules? • Software testing 10 rules? – Test early and test often. – Integrate the application development and testing life cycles. – Formalize a testing methodology; you'll test everything the same way and you'll get uniform results. – Develop a comprehensive test plan; it forms the basis for the testing methodology. – Use both static and dynamic testing. – Define your expected results. – Understand the business reason behind the application. You'll write a better application and better testing scripts. – Use multiple levels and types of testing (regression, systems, integration, stress and load). – Review and inspect the work, it will lower costs. – Don't let your programmers check their own work; they'll miss their own errors. Source: SQAtester.com

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution - Summary • Test execution consists of preparing of test bed/setup ,execution of planned test cases ,verification of results ,Test tracking and reporting and metrics collection. • Apart from that ,it involves applying different test methods like black box to system testing ,white box at unit testing level and top –down and bottom –up in case of integration testing .

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Test Execution – Summary Contd..

• Testing will involve several rounds based on the stability and bugs found on the software and start /stop criteria identified . • Regression testing will be carried out when ever the new bugs are found and fixed in the tested software to verify the impacted items. • Test execution will involve preparation of test reports ,collection of different metrics and meeting the test objectives .

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Testing Tools Overview  Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Testing Tools Overview (contd…)

• Testing tools are used to automate the testing process. • These tools will have features like – Record and play back – Test Script generation – Virtual users And so on…

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Testing Tools Overview (contd…) •

Different vendors released different testing tools in the market. Some of them are

Functional / GUI Testing Tools: • Win Runner • Rational Robot • Visual Test

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Testing Tools Overview (contd…) Performance Testing: • Rational Quantify • True time Web Testing: • Silk Test • Astra Quick Test • eTest Suite

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Testing Tools Overview (contd…) Java Testing Tools: • • • • •

Jtest JProbe QStudio Optimizeit JStyle

 Testing Life Cycle Overview  Test Strategy  Test Planning  Test Case Design  Test Execution  Testing Types  Testing Tools Overview

Appendix 1: Example for test life cycle • To understand the different stages of testing, consider an example in which you need to test a Webpage, say HRWEB->resume part. • To test this Webpage following activities needs to be done – – – –

Prepare Test Strategy for testing HRWEB->Resume section Prepare Test Plan for testing HRWEB->Resume section Develop Test Cases for testing HRWEB->Resume section Execute the test cases and report the results, including bug reporting, re-test and tracking.

Appendix 1: Example for test life cycle cont.. •

Test strategy to test HRWEB->Resume page includes overview of – – – – – – – –

•

Document the test requirements User Interface testing Test for associated links Error Handling and Exceptions Functionality testing Boundary conditions Test for Load, Performance, Security issues Deciding of tools and test set-up to test

Test Planning to test HRWEB->Resume includes – – – –

Resume page and all associated links will be tested Specify how it will be tested - Test Cases Identify resources for testing and finalize timeline Specify Hardware, Software configuration requirement

Appendix 1: Example for test life cycle cont.. • Test Case Development – Several Test Cases will be generated as per the test strategy and plan for User Interface, Functionality, Error Handling of Resume Section in HRWEB – Integration and System Test cases will be generation to test for the whole application including Resume Section.

• Test report: – Test Cases will be executed and Test Results will be recorded and tracked. – After bugs are fixed, the system is tested again to verify the same – The whole system is also tested to verify that new bugs have not been introduced.

Appendix 1: Example for test life cycle cont..

Sample User Interface Test Case for Resume Section • Perform the necessary steps to view the Resume Section on TEDWEB • Verify that the menu TEDWEB Menu displays at the top and the Resume Menu displays at the left of the screen • Verify the presence of the following buttons - Add Resume, Modify Resume, Add project, Modify Project etc.. Sample Functionality Test Case for Resume Section • Click on ‘Add Resume’ link • Verify a new screen appears to the right that allows the user to add his resume details • Put the cursor in Educational Qualifications field and verify it is enabled for user input • Put the cursor in Summary field and verify it is enabled for user input.

Appendix 1: Example for test life cycle cont.. Sample Error Handling Test Case for Resume Section • Click on ‘Add Resume’ when the resume already exists. Verify the message displayed is ‘Profile Already Exists’ • Click on ‘Modify Resume’ when no resume has been added. Verify that a proper error message is displayed that no resume exists.

Appendix 2: Testing Standards • IEEE Standard for Software Test Documentation (ANSI/IEEE Standard 829-1983). This is a summary of the ANSI/IEEE Standard 829-1983. • It describes a test plan as: “A document describing the scope, approach, resources, and schedule of intended testing activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning.”

IEEE testplan template

Appendix 3: Additional readings • Please go through the following documents, which are available in Interops KM Page. – – – – – – – – –

Inter101_002-Intro Software Testing Inter101_003-Test Plan & Test Case Inter101_004-SDLC&TLC Inter101_005-Unit & Integration Testing Inter101_006-Functional & Regression Testing Inter101_007-General Testing Types Inter101_008-Test Report Automation Tools Inter101_009-Review & Walk thru Inter101_010-Reliability terminology