Automation Test Strategy

Automation Test Strategy Document – Contract Management System

Automation Test Strategy Contract Management System

Revision / Document History 12/9/2009

Page 1 of 14

Automation Test Strategy Document – Contract Management System Version 0.5 1.0

12/9/2009

Date 6 Feb 2009 26 Feb 2009

Author Makesh Makesh

Description Initial draft Updated as per the comments captured during team discussion w.r.t strategy

Page 2 of 14

Automation Test Strategy Document – Contract Management System

Table of Contents INTRODUCTION............................................................................................................4 SCOPE .........................................................................................................................4 APPROACH...................................................................................................................4 PHASE PHASE PHASE PHASE

I – APPLICATION UNDERSTANDING & AUTOMATION TEST ENVIRONMENT SETUP ......................................4 II – FUNCTIONAL TEST CASE REVIEW................................................................................5 III.....................................................................................................................6 IV.....................................................................................................................6

TEST ENVIRONMENT SETUP.........................................................................................7 HARDWARE CONFIGURATION...................................................................................................8 SOFTWARE CONFIGURATION...................................................................................................9 TASK & BASELINE TASKS...........................................................................................10 REVIEW PROCESS......................................................................................................10 TEST CASE REVIEW ........................................................................................................10 AUTOMATABLE/NON- AUTOMATABLE REVIEW PROCESS......................................................................11 TEST SCRIPT REVIEW.......................................................................................................11 BATCH MODE TEST EXECUTION REVIEW PROCESS..........................................................................12 APPROVAL PROCESS .................................................................................................12 TEST CASE APPROVAL.......................................................................................................12 AUTOMATABLE/NON- AUTOMATABLE APPROVAL..............................................................................12 BATCH MODE TEST EXECUTION APPROVAL PROCESS........................................................................12 SCRIPT MAINTENANCE...............................................................................................12 REPORTING AND STATUS UPDATES............................................................................13 DELIVERABLES...........................................................................................................13 RISK & MITIGATION PLAN.........................................................................................13 ----------------------------------------------------END OF DOCUMENT---------------------------------------------------------------------.............................14

12/9/2009

Page 3 of 14

Automation Test Strategy Document – Contract Management System Introduction This is the Automation Test strategy document which will dedicate the task & activities involved to complete the Functional Test Automation for Contract Management systems (hereafter referred as CMS). Scope The scope of the project is to perform the Functional Test Automation for CMS using QTP 9.5. In addition to the above, the existing test cases will be reviewed and should be tweaked to improve the quality of the test cases. Also wherever required new cases have to be created to ensure that maximum functional test coverage is obtained. Up on completion of reviewing/enhancing/creating new test cases, the team will automate the feasible test cases and deliver the test scripts to ABC. Also Automation suite should help ABC QA team to perform the regression testing for upcoming releases. Approach Phase I – Application Understanding & Automation Test Environment Setup The first phase of the project is Application Understanding. The set of identified automation test engineers will take part in a series of session to understand the application functionality existing in CMS application. Also it has been planned in such a way that ABC will send Functional QA Lead and Develop Manager to conduct series of demo and interactive session to spread across the core functionality of the application to the automation team. Moreover after each session, the automation team will share the understanding on the application what has been covered for that day via email. Any questions/clarification will also be asked in the same email so that ABC QA Lead can clarify them on the next day. The above exercise will definitely help the team to clarify the doubts on the functional features and at the same time it will allow ABC whether the automation team has really understood the core functional features of the application. This phase is planned for 3 weeks and at the end of first phase, the automation team should be having indepth functional understanding of the application. Parallely the Automation Project Manager will work with ABC Infrastructure team and XYZ Infrastructure team to get the necessary test environment ready before the start of phase3. Deliverables for Phase1: 1. Understanding email on the functional features from automation team ABC QA Lead 2. Automation Test Plan

Required from ABC/XYZ: 1. Application Demo and Session on Functional Features 2. Establishing VPN connectivity for the accessing of server from both locations 12/9/2009 Page 4 of 14

Automation Test Strategy Document – Contract Management System 3. Getting the Automation Test Environment Setup 4. Getting the perforce setup for ongoing script maintenance 5. Review of Automation Test Plan Phase II – Functional Test Case Review The phase will involve the Functional Test Case Review. In order to start with the review, ABC should provide access to the existing test cases that has been already written by internal QA team. Automation team will take stock of all the test cases and segregate them as per the functional understanding which has been achieved in phase1. The test cases will be allocated to the automation team and the information will be shared with ABC so that it is clear that who owns which feature and so on. This will really help to get in touch with the right resource for any clarifications/questions/suggestions. Each member of the automation team will review the test cases and capture the review comments, if necessary information is missing like incomplete test procedure, not clearly defined expected results, invalid test cases, missing test cases for the functional features and so on. The phase is planned for 4 weeks, on the first 2 weeks the automation team will work on reviewing the test cases and capture the review comments in the spreadsheet. The comments will be shared on weekly basis to ABC. After completing the above exercise, will definitely help to quantify the efforts involved in updating existing test cases and creation of new test cases. On the third week, the team will start enhancing the existing test cases to improve the quality of the test cases and the same will be sent for review to ABC. On the fourth week, creation of new test cases will happen and it will be shared with ABC for approval While coming to classification of test cases, Automation team recommends in such a way that there will be 3 categories on how the test cases will be looked into. a. Sanity Level Test Cases: This will cover the basic functionality touching almost all the modules to ensure the application works fine and it can be drilled down for further testing. For example, it takes a tour of creating company, accounts, rules, approvals, authoring, creation of contracts and ends up finally checking the reports. Bulk import and Client tool will be checked separately during sanity. The coverage will be around 20% of test cases. b. Confidence Level Test Cases: Confidence level cases will cover all positive scenarios of the test cases with different test conditions. Also it may cover 40%-60% c. Regression Level Test Cases: The Regression cases will cover the negative scenarios as well as the out of box cases to check the application behaviour and this will be around 20%-30% of test cases. After the completion of four week exercise, the below deliverables should be achieved and delivered to ABC 1. 2. 3. 4. 5. 12/9/2009

Test Case Review Report Share the updated test cases Share the new test cases Share the Automation Feasibility Report Share the Automation Framework document Page 5 of 14

Automation Test Strategy Document – Contract Management System Required from ABC 1. 2. 3. 4. 5. 6. 7.

Access to Test Case Repository Clarifications on test cases posted by Automation Team Review of Test Case Review Report shared by Automation Team Review of Updated Test Cases Review of New Test Cases Review of Automation Feasibility Report Review of Automation Framework document.

Phase III The phase 3 of the project is the creation of automated test scripts using QTP 9.5. The team will follow the instructions as per the framework. The framework will take care of the folder structure, shared modules and naming conventions of variables that are planned to be planned during the automation of test cases. XYZ assume that 65%-75% of test cases can be automated that are available for CMS. Also it is evident that the automated scripts will be developed as per the classification developed so that it can be executed on need basis. The duration for completing 65%-75% of automation test cases has been estimated as 12 weeks from the completion of phase 2. Team will make a weekly delivery of test scripts and share the execution of the batch test results to ABC team for approval. Any modification/suggestions will be made on the successive week and the same will be verified during the next batch execution. Deliverables 1. Weekly Script delivery along with Batch mode execution results 2. Reporting of Functional Automation coverage achieved on weekly basis Required from ABC: 1. Acknowledgement of Test Script deliver on weekly basis 2. Review of Batch mode execution results 3. Provide suggestion on the test scripts delivered on weekly basis

Phase IV Finally all the test scripts will be consolidated and executed in the batch mode for a final run and the results will be shared with ABC. Up on delivery of automated test scripts will require a project signoff from ABC. The verification of test scripts will be performed on application hosted environment.

12/9/2009

Page 6 of 14

Automation Test Strategy Document – Contract Management System Deliverables 1. Final Delivery of Test Scripts 2. Understanding Document Required from ABC

1. Acknowledgement on Final delivery of Test Script 2. Comments on Understanding Document 3. Project Sign-off Test Environment Setup The test environment will consists of local CMS setup at XYZ to verify the test scripts developed by the team before making a weekly delivery to ABC. Also the infrastructure at XYZ has not been made according to Hardware configuration/ hosted application setup shared by ABC. Moreover the objective of having the local setup is to ensure the test scripts works fine and it can be further tested on ABC application hosted environment. The perforce client will be installed in automation engineer desktop machines in order to perform the check-in/check-out of test scripts to perforce server. In order to perform the above activities a VPN connectivity setup will be made so that so that any issues related to application can be solved immediately using RDC or VPN access. Please find the Test Environment Diagram below.

12/9/2009

Page 7 of 14

Automation Test Strategy Document – Contract Management System

A utomation T est E nvironment S etup

X Y Z C hennai A pplication / D atabase S erver W eb S erv er

D esk top1

D esktop2

D es ktop3

D esk top4

D esktop5

C M S S erver

V P N C onnectivity

ABC S an Francisco

C M S S erv er at A B C

QTP S erver

Build S erver

P erforc e Server

Hardware Configuration 12/9/2009

Page 8 of 14

Automation Test Strategy Document – Contract Management System 1.1.1

Workstations Intel Pentium Core 2 Duo 2GB RAM 150 GB HDD 10/100 Ethernet Care

1.1.2

Web Server Intel Pentium Core 2 Duo 2GB RAM 150 GB HDD 10/100 Ethernet Care

1.1.3 Application /Database Server (Local Setup) Intel Pentium Core 2 Duo 32 Bit Windows 4GB RAM 150 GB HDD 10/100 Ethernet Care 1.1.4

QTP Server (Onsite Setup) Intel Pentium Core 2 Duo 8GB RAM 150 GB HDD 10/100 Ethernet Care

Software Configuration 1.1.5

Workstations WinXP SP2 QTP 9.5 Perforce Client Internet Explorer 7.0 Microsoft Office 2003

1.1.6

Web server WinXP SP2 Tomcat

1.1.7 Application/ Database Server (Local Setup) Windows 2003 Server Tomcat Oracle 10G 12/9/2009

Page 9 of 14

Automation Test Strategy Document – Contract Management System 1.1.8

QTP Server (Onsite Setup) Windows 2003 Server QTP 9.5 QTP License Management Microsoft Office 2003 or 2007 Internet Explorer 7.0

Task & Baseline Tasks Project official Start Date: 27th Jan 2009 Sl.No Tasks Start Date End Date Duration Phase I 3 Weeks 1. Application Understanding 27th Jan 2009 13th Feb 2009 Milestone 1: Delivery of Automation Test Plan Phase II 4 Weeks 2. Test Case Review 16th Feb 2009 27th Feb 2009 3. Enhancing Existing Test Cases 2nd Mar 2009 6th Mar 2009 th 4. Creating New Test Case in addition to 9 March 2009 13th March Existing Test Cases 2009 Milestone 2: Delivery of Test Case Review Report – 16th March 2009 Milestone 3: Delivery of Feasibility Report – 16h March 2009 Milestone 4: Delivery of Framework Document – 9th March 2009 Review of Frame work document by ABC - 9th March to 13th March 1 Week Review of Feasibility Report by ABC -16th March to 19th March 4 Days Phase III 12 Weeks 5. Developing Test Scripts using QTP 9.5 16th March 5th Jun 2009 2009 Milestone 5: Weekly deliverables of Test Scripts with Batch Mode Results Phase IV 1 Week 6. Verifying Test Scripts in Batch Mode and 8th Jun 2009 12th Jun 2009 fixing issues. Milestone 5: Final Deliverables of Automated Test Scripts – 12th Jun 2009 Milestone 6: Understanding Document/ KT on Automation framework – 8th Jun to 12th Jun Note: The above timelines subjected to change if there is any change in scope. Also the timelines are defined based on the assumption that 30%-40% of test cases may require enhancements and the creation of new test cases ranging from 400 -500. Moreover the timelines for test script development has been estimated based on the assumption that 65-75% of the test cases can be automated.

Review Process The below Review process will be carried out internally to ensure the quality has been met at each and every phase of the project. Test Case Review

12/9/2009

Page 10 of 14

Automation Test Strategy Document – Contract Management System Automation team will take stock of all test cases that are already prepared by ABC for CMS 3.2.5. The review will happen based on the application understanding the team has achieved in Phase1. The review will ensure that the following items. a. Valid Test Cases with Test Procedures written against the Application. b. Should validate the expected results dedicate correctly that needs to be checked. c. Should capture the incomplete procedures with proper comments on what is missing. d. Invalid Test Case with respect to the current application e. Gaps in test cases missing few features/out of box scenarios All the above will be captured and the same will be shared with ABC on a periodical basis during phase 2. Also ABC & XYZ will have a conference call whenever required, if there are any clarifications to be asked from both sides. All the comments/suggestions will be captured and the same has to be implemented. Automatable/Non- Automatable Review Process The classification of automatable/non-automatable will be carried out based on the POC report will has been shared with ABC. Also the team will be educated on the POC report so that the classification is done properly. Moreover the team will categorize the test cases that are automatable based on the core functionality of the application and see the benefit that can be achieved if those test cases are automated. As a first step the sanity test cases will be automated followed by Confidence and Regression test cases. At the end of Phase2, XYZ will share the feasibility report to ABC for a review. ABC will review the feasibility report and provide their approval based on the report. If any clarifications are required then ABC & XYZ will have a conference call to complete the review of feasibility report. Also XYZ will confirm the percentage of test cases that can be automated in the total test cases that can be automated. Note: Test cases related to installation, upgrades and online help supported for different languages, comparing Bar & Pie Charts may not be automatable due to limitation with the tool. Test Script Review The team will follow the instructions/steps that have been called out clearly in the Automation Framework document. Up on completion of test scripts for the automatable test cases, there will be a peer review of test scripts. Followed by Automation Lead/Manager will review test scripts and post comments accordingly. The owner of the test script will modify them according to the review comments. All the comments will be captured in the review document for future reference. The exercise will cover the following a. b. c. d. e. 12/9/2009

Naming conventions of scripts. Adding proper comments for future reference. Coding Standards defined as per the framework document. Validating the re-usable functions/VB scripts for handling few challenges during automation. Tweaking the scripts for better execution. Page 11 of 14

Automation Test Strategy Document – Contract Management System Batch Mode Test Execution Review Process The batch mode execution will be performed at each and every week during the course of phase3. At each and every week after the completion of batch test the results will be reviewed. The review will ensure that all the test cases have been executed successfully and there were no scripts issues. Subsequently the failures test scripts will be analyzed for the root cause. If there is an application issue, the team will send across the information to ABC and defect Id will be associated with test cases so that it need not be looked in when same test scripts fails during successive batch test execution. By performing this activity will help the team for a long run, since the scripts are verified each and every week to ensure the final delivery is met in time. Approval Process The approval process will be carried out for the below sections to ensure that XYZ & ABC where in the same page during different phases of the project. Test Case Approval Based on the test case review report, the team will update/modify the existing test cases. Up on completion of test case enhancement for each and every day will be shared with ABC for Review. ABC should review with their counter parts and ensure the test case captures as per application functionality. They should provide the acknowledgement on the enhancement made to the existing test cases and also on the new test cases developed by the team. The approval should also say that if there is any changes/modification required to the test cases that has been shared for review to ABC. Automatable/Non- Automatable Approval XYZ will share the Feasibility report based on the expert review of the test cases that are available for CMS. ABC should provide the acknowledgement on the feasibility report which will be considered as an input to start the development on test scripts. The feasibility report will confirm the percentage of test cases that can be automated, percentage that cannot be automatable due to tool limitation, application environment and so on. Batch Mode Test Execution Approval Process Up on completion of batch mode execution on every week, XYZ will share the batch mode results to ABC with the review report on the number of test cases passed /Failed due to Application issues. ABC should provide the acknowledgment on the batch mode execution. This will be captured on weekly basis to ensure that weekly milestones are achieved with better quality. Note: The script failed due to script issue will be fixed and the same will be updated during the weekly project status meeting. Script Maintenance 12/9/2009

Page 12 of 14

Automation Test Strategy Document – Contract Management System The test scripts will be checked-in in the perforce server. Also the scripts will be modified in such a way that all comments will be noted before enhancing the scripts for scripts issues or with respect to any change in application UI/functionality. All team members will have their own perforce account to check-in the scripts after completion. Reporting and Status Updates There will be a weekly status meeting with ABC to update on the progress made on weekly basis, plan for the next week and deliverables expected the following week. Also any issues causing Roadblock will be discussed during the weekly status call. If there is any impact to the schedule will be intimated to the team. Deliverables

Sl.No. 1 2 3 4 5 6

Deliverables Automation Test Plan Test Case Review Report Automation feasibility Report Automation Framework Document Automation Test Scripts Understanding Document

Deliverable Mode Word Document Spreadsheet Document Spreadsheet Document Word Document QTP Scripts bundled Word Document

Risk & Mitigation Plan Sl.No 1

Risk Any change in scope of project / change in application functionality

2

More than 40% of effort involved in test cases enhancement

3 4

5

12/9/2009

Any script issue during the execution of test scripts in batch mode More than automating 75% of test cases will have an impact to the timelines Any application issues automation of test cases

during

the

Mitigation Plan XYZ will review the scope/change in functionality update ABC whether it can be accommodated with the existing schedule or it will impact the current timelines. Also it can set the priorities correctly so that it is clear whether it has to be addressed immediately or can be considered at the end during completion of project XYZ will review the effort required to accommodate more effort that is required to modify the test cases. Also intimate ABC about the increase in scope of work with the updated timelines XYZ will get the script issue resolved on or before the next batch test execution XYZ will check with ABC on the test cases and set the priority accordingly and update ABC on impact of project completion date. ABC should help XYZ in fixing the issue, since there is time zone there will be a delay in fixing the issue. In that case XYZ will try to cover it up within the current Page 13 of 14

Automation Test Strategy Document – Contract Management System schedule. In worst case there will be an impact to the current project timelines.

----------------------------------------------------End Document---------------------------------------------------------------------

12/9/2009

of

Page 14 of 14