Tool Categories

Tips for Success when Selecting Testing Tools

Marie Ackley ackley @ promail.com SASQAG panel discussion October 18, 2001

Different Categories of Testing Tools ◆ ◆ ◆ ◆ ◆ ◆

◆

Requirements Management Configuration Management Test Planning and Test Case Management Test Evaluation - Coverage Analysis Automated Testing Systems - Front End testing Load Simulators and Performance Measurement Tools Defect Tracking

others: Cyclomatic Complexity; Bounds Checkers; Comparators - Files, Screens, Reports; Data Generators - Combinatoric, Cause / Effect Node

2

Leading Cause of SHELF WARE in Testing Tools is Putting the CART before the HORSE Don't invest in Tools without having good QA Methods and Practices. A good tool will compliment and augment good processes but not even the greatest tool can overcome process chaos. 3

Requirements Management ◆

Supports multi levels of linking and tracing: Requirements to Specifications to Test Cases, including hierarchies within those groups

◆

Provides traceability - both forward and backward

◆

Provides coverage analysis

◆

Provides multi relationships: one-to-many, many-to-one and coverage is tracked for partials when relationship is one-to-many

◆

Provides Change Management of requirements along with audit history and maybe security privileges for modifications

◆

Import and export features so stuff came go and come between the tool and word processors or spreadsheets 4

Requirements Management Links to tool comparisons the "Volere" Requirements Method - by Atlantic Systems Guild http://www.systemsguild.com/GuildSite/Robs/Template.ht ml Automating Requirements Management - by Karl E. Wiegers Detailed feature comparisons of many requirements management tools and guidance on selecting a requirements management tool http://www.processimpact.com/articles/rm_tools.html 5

Configuration Management "To change and to change for the better are two different things" - German Proverb ◆ Provides Identifications (indexing of artifacts) ◆

Controls not only software code but documentation, hardware, firmware, and system environment configuration

◆

Provides audit trail and history tracking

◆

Provides customizable reports

◆

High ease of use for View Only functions

◆

Sufficient security levels for the Promote functions

◆

Accounting and reporting for different ”Status Levels" WIP, Waiting Room, Release/Production

6

Configuration Management Links to tool comparisons Configuration Management Vendor Links - by UCM Central http://www.snuffybear.com/ucmcentral_new_vendorlinks.htm config-mgmt Newsgroup http://www.faqs.org/faqs/by-newsgroup/comp/comp.software.configmgmt.html

7

Test Planning and Test Case Management ◆

Import test cases from other file formats such as word processing or spreadsheet

◆

Customizable reporting

◆

Tracks history and status of test cases

◆

Allows file attachments

◆

◆

◆

Integration - can tool link to other tools such as your Requirements tool and your Automated Testing tool Provide a Test Execution Results repository Maintains Master Set of test cases and then can build “subsets” for project specific needs

8

Test Planning and Test Case Management Links for tool comparisons Testing and Test Management Tools http://www.methods-tools.com/tools/frames_testing.html Information About Testing Tools http://www.cs.queensu.ca/home/shepard/testing.dir/under.construction/tool _list.html

9

Test Evaluation

- Coverage Analysis

◆

Identifies what "unique" data combinations are needed

◆

Uses an appropriate measurement technique Decision; Condition/decision; Path; Predicate

◆

Types of coverage included in the reporting Branch; Loop; Multi; Operator; Operand

◆

Integration - is it compatible with your other tools such as Automated Test Execution tools

◆

Implementation - is the tool "invasive" (must be compiled along with the source code)

10

Test Evaluation

- Coverage Analysis

Links to tool comparisons Experience With the Cost of Different Coverage Goals for Testing - by Brian Marick http://www.testing.com/writings/experience.pdf Coverage Analysis - by Steve Cornett http://www.bullseye.com/webCoverage.html

11

Automated Testing Systems Front End testing ◆

Maintenance - object recognition when the application changes

◆

Ease Of Use - complexity matches the programming talent of QA persons

◆

Tool language and syntax matches what company uses: Visual Basic or C

◆

Extensible Language - can use external subroutines or DLLs

◆

Integration with other tools - Code Coverage, Test Case Management, Defect Tracking

◆

Compatibilty - platforms (Client/Server, Web, Mainframe) Operating Systems(Windows, Mac, Unix, OS/2, Mainframe) and GUI Development (PowerBuilder, C, Visual Basic, 12 Galaxy

Automated Testing Systems Front End testing ◆

Supports Data Driven inputs from flat files, spreadsheet files, and database connections

◆

For Evaluation/Proof of Concept - include automating a Level One test Links to tool comparisons

Making the Right Choice - by Elisabeth Hendrickson http://www.qualitytree.com/feature/mtrc.pdf Short Sharp Advice about how to choose a testing tool - Mark Fewster and Dorothy Graham http://www.grove.co.uk/Tool_Information/Choosing_Tools.html

Tool Comparison between Rational, Mercury, Segue, and Compuware - by Ray Robinson

13

Automated Testing Systems Links to articles Lessons in Test Automation - by Elfriede Dustin http://www.sqa-test.com/articles.html The Truth About Automated Test Tools - by Linda Hayes http://www.worksoft.com/resources/4-97.htm http://www.worksoft.com/resources/5-97.htm Test Automation - by Bret Pettichord http://www.io.com/~wazmo/qa/#test_automation Testers' Network - VeriTest website http://www.veritest.com/testers'network Test Automation Frameworks - by Carl J. Nagle http://members.aol.com/sascanagl/FRAMESDataDrivenTestAutomationFramew orks.htm 14

Load Simulators and Performance Measurement ◆

Reporting includes graphical charts

◆

Debugging features for script creation

◆

Reporting includes graphical charts and logs of performance for server speed and responsiveness

◆

Quantity of simulated users

◆

Variations in session-specific items such as cookies, usernames, passwords, proxies and ASP-session IDs

◆

Support for your middleware - SQLnet, DBLib or CBLib, SQL Server, ODBC, Telnet

◆

Challenges include SSL sessions, cookies, forms, proxies, password authentication, dynamic HTML, full HTTP, SSL, XML, Javascript, ActiveX, WAP, Java

◆

Platforms and protocols 15

Load Simulators and Performance Measurement Link to tool comparisons Load and Performance Test Tools - by Rick Hower http://www.softwareqatest.com/qatweb1.html Software Research - TestWorks http://www.methods-tools.com/tools/testing.html

16

Defect Tracking ◆

Determine how many current users you have and consider and scaling for future users

◆

Web based or client-server

◆

Features: Customizable Fields; Customizable Reports; File Attachments; Audit Trail; Email linking; Security Levels

◆

Provides customizable rules for the workflows, security privileges and notifications

◆

Integration - does it link to your other tools 17

Defect Tracking Links to tool comparisons Tracking Down a Defect Management Tool - by Hung Quoc Nguyen http://www.qacity.com/testgen/DefectTrackingNguyenSTQE 701.pdf The Problem with Problem Tracking - by Linda Hayes http://www.worksoft.com/resources/3-99.htm

18

Websites for Tool Information Vendor contact information http://www.testingfaqs.org/tools.htm http://www.ondaweb.com/sti/stivend.htm#L http://www.dbmsmag.com/pctest.html#P0890 Bulletin board forums and articles http://www.qaforums.com http://community.sqatester.com http://www.qualitytree.com/links/links.htm http://www.stqemagazine.com

19