System Testing Methodology

System Testing Methodology QMITST01 / Version 1.0

Effective: June 15, 2005

Purpose The purpose of system Testing methodology is • To define how system testing projects are executed and • To serve as a basis for different phases in a system testing project

Scope During project planning, this methodology shall be used to define the activities and work products for each phase. Only the Testing specific activities are detailed in this methodology. For all other activities like project management, configuration management, etc. the project shall follow the processes defined as part of Qualify. Each system testing project which is taken up may go through all the phases of the Software Testing Life Cycle which are detailed in this methodology, or may go through only certain phases of Testing life cycle. For instance, a project may encompass only test execution based on the customer’s requirements. The phases that will be followed during the course of the project will have to be detailed in the Software Project Plan (SPP) –Testing of the project. The applicable phases of execution, the associated work products, verification and validation criteria for each of the relevant phases shall be at par with this methodology. The system testing methodology offers the flexibility of entry and exit through any of the defined phases. The user can use this flexibility by following the Entry Input Task Validation Output Exit (EITVOX) criteria defined for each phase.

Definitions & Acronyms Definitions Keyword Bug Tracking

Definition Bug Tracking process covers the defect lifecycle from identification of a defect to its resolution / closure. Functional Testing Process to determine that the features / functionality of the application / product is as per the requirements Load testing Testing the application behavior under varying acceptable loads Performance Testing conducted to evaluate the compliance of a system or Testing component with specified performance requirements Regression Testing • Process to ensure that the earlier applications/ products still work with the new changes. • Tests ensure that changes do not introduce unintended behavior or additional errors. Smoke testing Smoke testing to ensure the build version is ready for undertaking testing System Testing Process to determine that the system functions as intended or documented.

Acronyms Acronym STC STP

Explanation System Test Cases System Test Plan

Overview The system Testing process starts with analyzing/ Understanding of Business/ Functional requirements. System testing Objectives are identified specified and prioritized. (Depending on the client requirements System test may include different tests like Functional Test, Regression Test and Load-performance Test). The System Test plan is created. Test Cases are prepared to validate if requirements are met. System testing Environment is identified and setup is done to proceed with Test Execution. Tests are executed to ensure that the application meets the System requirements. Test Automation follows the manual Test Execution as per Customer Requirement Defects identified during the system Testing are entered in to Defect tracking process. Test Result Reports are generated after completion of Testing. The System Testing project comprises of the following phases: 1. 2. 3. 4. 5. 6. 7.

Analyze Requirements Create System Test Plan Create System Test Cases Ensure System Testing Environment Execute System Test Cases Review System Test Results. Prepare Test Report

References QEdge Testing Methodology Functional Testing Methodology Regression Testing Methodology Load-Performance Testing Methodology

Tailoring Guidelines It is recommended that this methodology be followed during the course of the project. If any tailoring is to be made due to the customer’s requirements, then the tailoring shall be documented along with justification in the Project Tailoring Record (PTR). (Ref… Tailoring Guidelines)

Requirements Analysis This phase consists of requirements from BRS/ SRS and Design documents to understand the application functional areas and prioritizing test objectives Entry

Requirements Specification and Design Specifications are available

• • • • • •

Inputs

Tasks

• • • • • • • • • • • • • •

Verification / Validation Outputs Exit

Requirements Specification (BRS/ SRS) Design Specification User Guide Installation Guide Study and review the system Requirements document Understand application/product architecture in Design Specification document Specify Test Objectives Define Strategy/ Approach Specify Acceptance Criteria and completion criteria for each test objective. Prioritize the Test Objectives as High and Medium High - Most important tests: must be executed Medium - Second-level objectives: should be executed after highpriority tests Identify scope of Automation. Identify Test Automation Tools as required Review of Test Objectives Review of Scope. Identification of critical areas Prioritized test objectives The test objectives are prioritized Approval of Scope

Create System Test Plan This phase consists of creation of System Test Plan after understanding the requirements. The plan basically consists of Test Criteria, Scope, Environment, Estimations, Risks, Defect Tracking, Build Management (Refer… Guidelines for Build and Release Management, Guidelines for Metrics Definition Standard, Guidelines for Defect Management) Entry Inputs

The test objectives are prioritized and the Scope is approved • Requirements Specification (BRS/ SRS) • Design Specification • User Guide • Installation Guide

•

Tasks

Verification / Validation Outputs Exit

Plan System Test* According to the required tests specified by the Client. Identify all the features to be covered in each kind of test. Estimate the time to prepare Test Cases Estimate the time to execute the Test Cases. Identify Risks. Analyze and document effort estimation Identify HW/SW requirements. Identify the resources requirements. Assign responsibilities to the resources. Plan to Track and control Effort estimations Plan Build Management/ tracking Plan Defect Tracking

• • • • • • • • • • • • System test plan review

System test plan Completion of the System Test Plan and sending for approval

* Refer to the Procedure Documents for different types of testing

Create System Test Cases This phase consist of preparation of Test Cases following different techniques EP, BVA, Cause Effect, etc. Test Cases are prepared based on the identified Functional scenarios which will be useful for Test Automation. (Refer… Guidelines for Software Testing) Entry Inputs

Tasks

Verification / Validation Outputs Exit

Approved System Test Plan • Design document • Use cases/ System requirements • Design Specification • Design System Test Cases • Identify Test Scenarios for Automation • Identify and Prepare cases/ Scenarios for Automated Testing Scripts • Create Test Cases from Use Cases • Test Cases review • Automated Test Scripts review • System Test Cases • System Test Scripts • Completion of designing of System Test Cases • Completion of Automated Test Scripts • Send Test Cases and Test Scripts for approval

Ensure System Testing Environment This phase consist of Environment setup to ensure readiness for execution. (Refer… Guidelines for Test Data Management) Entry

The test objectives should be prioritized and the scope is approved

Inputs Tasks

Verification / Validation Outputs

Prioritized test objectives •

Ensure the required test environment for System testing is setup as identified in STP. • Execute the smoke test <When a build is received, a smoke test is run to ascertain if the build is stable and it can be considered for further testing. Smoke testing can be done for testing the stability of any interim build.> Test Environment review System testing environment • •

Exit

The required test environment for System testing is setup The smoke test is executed successfully

Execute System Test Cases This phase consists of execution of Test Cases and entering the identified defects in to Defect Tracker. Entry Inputs

Tasks

Verification / Validation

Approved System Test cases and System Test Scripts • • • • • •

System Test Cases System Test Scripts System Test Plan Execute System Test Cases manually Record all System Test runs in the System Test Report. Enter “P” (Pass) in the observed behavior column if Actual behavior matches Expected behavior; otherwise enter “F” (Fail). • Record observed behavior for every run and repeat the same until all the defects are closed. • Record all defects for tracking • Maintain System Test Reports as a quality record in the project folder. • Report and Close Defects using Defect Form/ Defect tracking Tool • Execute Automated Test Scripts as required. • Fill in Test Evaluation form Test Evaluation using test evaluation form

Outputs Exit

• • • •

Defect Log and Defect Summary Report Automated Test Reports/ Scripts Defect Log and Defect Summary Report is prepared Automated Test Reports/ Scripts

Review System Test Results This phase consists of review of test results.

Entry Inputs Task

Validation Output

Exit

• Completion of the execution of the system test cases • Defect Log is prepared Defect log •

Review of the system Test results. Reviewer may take some sample test cases and conduct testing to verify the correctness. • The reviewer shall verify the following: • All the relevant test cases are executed • The choice of test cases covers the main scenarios • The testing is as per the corresponding test plan • Testing shall be signed off and all the review comments are closed. Test Report evaluation and Review • Test evaluation reports • Defect Log • Defect summary report • Completion of the execution of the System test cases • Defect Log and Defect summary is prepared

Preparation of Test Report This phase consists of preparation of Test report. It contains summary information such as defect data, test evaluation summary. Entry

Defect Log and Defect Summary Report is prepared and Test Reports/ Scripts are Automated

Inputs

• Test Evaluation Forms • Defect Summary Reports • Calculate Test Completion time • Prepare Metrics • Analyze Defect data Test Report evaluation and Review System Test Report Test Report Completion

Task

Validation Output Exit