Wit Magazine April 2008

What Is

Test ng India’s 1st Software Testing Magazine April’08, Issue

Reduce number of testing scenarios with this insight on -

Cover Story

Pg

21

Risk based Testing

Inside this issue Putting the “Test” back in “Test plan” Some things I have learned in Software Testing Morning Ideas Have I got a deal for you? Finally Usability Testing

Are you TEAM ready? Pg 57 Pg 44 Pg 04 Pg 28 Pg 40 and more...

Pg

14

With little Preparation & Teamwork you can pave the way for new Team Memers.

Welcome to



 is the first conference being organized by PureConferences in India. 

We intend making  an experience for all participants by creating a platform for professionals and experts from the testing community of the Globe

 will revolve around ’Agility in Testing’ The theme will explore and expand on the need for testing teams to adapt to the changing demands of release and quality.

 will cover diverse topics. Keynotes, Tutorials, and Paper presentations by renowned speakers known nationally and internationally will be one of the highlights of the conference. Let’s mark the conference program in our calendar today itself.

 will offer opportunities to the partners to associate themselves with this endeavor and showcase their products in the conference. We are sure you want to participate and/or present a paper or tutorial.

Do visit us at http://www.test2008.in

Issue1: Vol 1: April 2008

Cover Story

Contents

Is bad testing eating your

pg

Pr fits? 04 08

12 21 28 33

21

FROM THE DESK: Morning Ideas by Sunil Gupta Good design is key for a good and reliable product

THE PROCESS ROOM: Software Testing: ~ A field par excellence by Sudhir & Anjan Software testing as a field has brought in a high level of confidence in customers and users and has thus far provided far more challenges to the new age engineers and academicians.

TEAM MANAGEMENT: Are You Ready? by Michael Bolton With a little preparation and teamwork, you can pave the way for a new team member.

COVER STORY: Is Bad Testing Eating Your Profits? by Rex Black Testing should focus on mitigating specific risks to the quality of the system. Sequence of test execution should be driven by associated risk...

THAT TIME OF YEAR: Have I Got A Deal For You by Matthew Heusser The methods and extend of our testing are all choices; you could even say they are trade-offs.

LEARN HOW TO TEST: What a Tester Should Know, even After Midnight by Hans Schaefer Testing The Normal Way is Not Enough.... Where you find defects, dig deeper!

40

LEARN HOW TO TEST: Finally Usability Testing? by Erik van Veenendaal Will it finally happen!

Other Stories 44 FROM DEV DESK: Some Things I’ve Learned in Software Testing by Jonathan Kohl

50 CONTROVERSY CORNER: QA is More Challenging than Development by Mrityunjay

53 Is scripted testing bad? by Vipul Kocher

57 Putting the “Test” back in Test Plan by Paul Carvalho

WhatIsTesting.com

1

Tea testers look at aspects of dried & brewed tea leaves and the liquor

Likewise, PureTesting goes into various aspects & nuances of soware testing We build innovati ve, end-to-end solutions, and manage critical testing processes, and reduce total cost of producing quality soware

•

Banking & Finacial Services • Pharmaceuticals • eLearning

• Datacom & Telecom • Embedded Systems • EAI & ISVs

Test Consulting • Testing Services • Testing Products

India • USA • UK • NZ www.puretesting.com

+91 (120) 4621010; [email protected]

Global Software Test Consulting & Services company

Editor’s Note

Release of any software is a moment of mixed emotions, pleasure and pain, joy and anxiety -- joyousness of seeing the hard work resulting in a release and the anxiety of the unfound bugs that could have been found and fixed. After a lot of effort, this magazine is also being released with similar emotions. We take great pleasure in introducing Version 1.0 of WIT magazine. We hope to have many more versions of this magazine, both major and minor, in years to come and hopefully with minimum number of patches. We also hope that the future versions of the magazine will have more features, will be more robust, will have an expanded user base and establish itself as a dominant player. Instead of the usual Editorial about the contents of the magazine which you anyway can find out from the table of content, we take this opportunity to share with you the reasons for this magazine and who are the people who have contributed to it. Many of the technical magazines “grow-up” with time. The articles become more “philosophical” and more abstract. Readers who are regular subscribers too grow with the magazine but most of the new comers start finding the magazine “too abstract” for their liking. WIT magazine hopes to keep itself relevant to both old readers who grow up with the magazine and new readers who are either new to the testing profession or new to the magazine. Our sincere thanks to the initial group of people who helped brainstorm the concept of the magazine. They are – Bernard Homès, Danny R. Faught, Johanna Rothman, Kamesh Pemmaraju, Matthew Heusser, Michael Kelly, Rex Black, Scott Barber, Stefan Steurs. Without them the magazine would not have become a living reality. All good things about the magazine are because of this group, the authors and other well wishers. All bugs are because of us. For putting together this version of the WIT magezine, I acknowledge the efforts of Ashwin Razdan for collaborating with the authors and collecting all the articles; Parag Sapre for designing the elaborate graphics; Anushree Tewari for providing editing support, and Satish Thakur for giving it the final shape in which you see the WIT magezine. Hope you find this version readable and useful and we hope to have a better V 1.1. ~ Vipul

WhatIsTesting.com

3

From the Desk

Daily LifeIdeas Morning

Morning

Ideas Though a good design is key for a good and reliable product, a robust test and evaluation of the product only ensures that the product meets all-explicit and implicit requirements and is ready for deployment and usage.

O

ne morning when I was engrossed in my newspaper, my daughter created panic as her school bag had given way at one of the shoulder belts. I was annoyed at her letting me know only when I was completely torn while she would have seen this happening over the last few days but my bigger reason of annoyance was due to the fact this was the 3rd or 4th time the different school

4

bags had torn almost at the same place. Since it was not possible to repair the bag in ten minutes, thus she carried her stuff in another bag but this episode left me thinking. I realized that the bag was made to meet the functional requirements of the student such as adequate space to carry stuff, compartments for lunch box, pencil box and water bottle. The bag was also providing a good aesthetics but lacked the robustness in terms of handling the stress that comes on the bag with daily usage....

WhatIsTesting.com

by Sunil by Gupta, Head of Testing Practice : Flextronics Software Systems

Composition and organization of the test team plays a very crucial role in the success of the product.

For product organizations, the composition and organization of the test team plays a very crucial role in the success of the product. Enough time and energy needs to be spent by senior test folks to • Visualize the hardware)

product

(software

and

• Look at the various field configurations Two things emerged when I extended my thought process further. One, product making whether it is a school bag or a large and complex telecom product requires good understanding and research in terms of features, stress areas, usability, etc without which the product would give numerous problems to the user as the school bag gave to my daughter. Second, testing plays a very crucial role to ensure that the product meets all requirements including the ability of the product to handle normal and abnormal stress conditions. If we take the school bag example, it is very important for the product manufacturer to ensure that the various stitches in the bag take up the full bag load else there are going to be numerous dissatisfied customers like me. Though a good design is key for a good and reliable product, a robust test and evaluation of the product only ensures that the product meets all-explicit and implicit requirements and is ready for deployment and usage. It is seen across the IT and software industry that the organizations give a lot of focus to hiring and developing software development teams but do not focus enough on developing testing teams. The test team is put in place either very late on the programs or the test teams are not organized well enough to provide the vital value expected from the Test team.

• Identify the tools required for testing • Look at the FOUR views: o User view o Maintenance View o Functional/Operational View o Performance View In order to achieve a good understanding of the product before the exhaustive test cycle commences, a few people from the test team need to be dedicated to carry out the Test Engineering tasks (as defined above). This set of people should have the mandate to carry out the verification and validation tasks. In the school bag example, the User View would be to look at all the ways in which the school bag would be used by a user. There would be several user scenarios such as: • Student lifting the bag from one belt • Student lifting the bag using the side belt and loading on its back • Student fully loading the bag with books but weight may not be too much • Student partially loading the books but the weight may have exceeded the

WhatIsTesting.com

5

Morning Ideas

Product testing and deployment is a very involved and rigorous task recommended weight

Suddenly I realized that it was 9:00 hrs by the watch and I had a 9:30 am meeting with the Quality Group to review the Post Release Defect Density of the products that we shipped in the last quarter and had only 30 minutes to reach office. Thanks to god, for my home is only a 7 minutes drive from office. Bye for now……

Maintenance View would be to look at aspects such as ease of cleaning and washing the bag; ease of repairing the bag and how easy it is to find the spares such as zippers, buckles, etc. Functional or Operational view is another important aspect of testing. Tester has to consider all the functional aspects of the product. In this example, aspects such as following would be tested: • Ease of keeping books in the bag • Access to all the pockets of the bag • Ease of closing and opening the bag • Ease of lifting the bag on shoulder and on the back • Enough space to keep books, lunch box, and water bottle, etc. Last but not the least, the Performance view of the product in terms of load handling, ability to bear stress due to regular use through load balancing is considered. In the example, tester has to take following aspects into consideration: • Testing the bag for various load and stress situations • Testing the bag for full load capacity • Testing the key joints/stitches under the stress situations • Identify the weak areas and strong areas of the product Product testing and deployment is a very involved and rigorous task and involves the best of people and the best of the practices.

6

WhatIsTesting.com

Neilsoft is a specialist engineering services & solutions company focused on helping our clients enhance their product engineering efficiency. Our services in the software product engineering domain span the entire product lifecycle including development, testing, and localization engineering.

T

I

N

G

E

e rtis e p ex

Glo ba l

Compatibility Testing

P

E

R

T

de liv e

Do m ai

l de

In te Te gra st ti in on g

mo

n tio la g al in st st In Te

S

C AE

PL M

ty

Functionality Testing

uri ec

CA M

[email protected] Bangalore

|

n& e v Pro

sse ro ce

st p

ers ad

te by ti- ng ul ti M Tes

y le ustr

s

C AD A u Te tom st at in ed g

Partnerships with ind

rage Sto

Certification Testing

Pune

X

ry

n

S

Performance Testing

E

bu ro

www.neilsoft.com/testing.htm

Canton

Focus verticals: Construction

Chicago

Cambridge

)

T

Dalian

Industrial Machinery

+91-20-2605 3003

Software

Transportation

Energy

I

S

E

The Process Corner

Software Testing: ~ A field par excellence

Software Testing

~ a Field Par Excellence

Apart from giving fillip to the economy, Software testing as a field has brought in a high level of confidence in customers and users and has thus far provided far more challenges to the new age engineers and academicians. Introduction Awareness on Software Quality has, over the last couple of decades, become one of the defining features not only in Indian software sector but also from a global perspective. Thanks to the dotcom crash which was an eye opener for everyone in the field to look back and focus on enhancing the quality of the products that are already in the market for many years. The quality initiatives taken through the last couple of decades, have now become all pervasive and thus given rise to a new avenue within the IT

8

and ITeS segment known as Software Testing. Renewed growth of the Software Industry has helped testing achieve the position of being called the watchdog of quality. Apart from giving fillip to the economy, Software testing as a field has brought in a high level of confidence in customers and users and has thus far provided far more challenges to the new age engineers and academicians. Looking at some of the astounding figures: 1. Bangalore has seen a 24% growth in its recruitment drive in software testing. It is

WhatIsTesting.com

by Sudhir & Anjan, Product Testing & Documentation : Accelrys, ITPL, Bangalore

Facts from Industry

Composition and organization of the test team plays a very crucial role in the success of the product. forecasted that India needs about 18000 test engineers in the year 2005-2006. 2. With India having more than 89 companies at SEI CMM Level 5 assessment; 275 Indian software and BPO companies acquired quality certification; and more and more IT-ITES companies have dedicated quality departments responsible for developing and deploying the quality policies and reviewing them. 3. NASSCOM survey has indicated that the road ahead for the Indian software and services companies within the quality arena was largely dictated and tuned to the developments taking place in the global marketplace. 4. The future would see organizations move from quality assurance to business assurance and focus on information security complying with international legislations, developing new global delivery capabilities and enhancing quality processes in new business areas. This is testimony to the fact that software testing is poised to grow at a phenomenal rate in future. With all these happenings, no one can deny the conclusion that software testing has grown as an independent field within IT/ITeS service segments. And it has given a clarion call to everyone within the professional arena, specifically academicians, to closely monitor, analyze and push forward the new horizons of the technology movement that software testing as a field is going to provide. Looking at some more interesting statistics on testing industry:

1. $3.0 billion of $4.6 billion in outsourced testing is sent offshore, throwing up opportunities for companies in India that thrive on low-cost knowledge workers. ~ Source Aztech Software - Bangalore 2. Testing could make up to 25-50 percent of software budgets. Independent testing is growing at 50-65 percent while the part of work done offshore is growing at 35-40 percent. ~ Source - Partha Iyengar, vicepresident at industry researcher Gartner 3. Testing service team at Wipro has jumped four-fold to 2,400 in two years. In the nine months to December, revenue grew 90 percent to $64 million, three times the industry average. ~Source - C.P. Gangadharaiah, VP Testing services, Wipro 4. The global market for software testing is around $13 billion. In India alone, the demand for software testing professionals is expected to touch 20,000 to 30,000 by December 2006. . ~Source – Internet Search

Recent Trends With this background, we can ask ourselves - what conclusion we want to drive from it? Why suddenly is everyone so statistical about gathering all these figures? To take a close look below we see from 50’s onwards till 80’s either the products were not tested or if they were then it was out of compulsion. Quality as a term itself took the real shape under the leadership of Borris Beizer who gave the awareness to the industry and the world about its very existence. The role of quality has become a necessity than a need in the software industry. If we start looking into history, we will find thousands of such cases where a small mistake in the software has brought havoc to life and money. The most recent one is the Columbia Space Shuttle crash. All such incidents have

WhatIsTesting.com

9

Software Testing: ~ A field par excellence

espoused within the IT-geeks eagerness towards “checking before delivering” approach. On the contrary, there was time when products were released without or with minimal testing done. Of course, many a time, these are dictated by how the market wants them. Time-to-market was probably more important than quality of the product. But, time has changed post dotcom crash era and software quality has the maximum thrust in the market today, be it consumer products, electronics, manufacturing or software. And quite rightly, software testing is enjoying its share of this quality conscious market. Testing as a field has now been more strategic and approach-centric thus adding new dimensions as well as many challenges to it. Previously testing used to be an afterthought activity in the post release phase when the clients call and say “we found something wrong in your delivered software”. But with time, it is no longer the case. Any small error today is a big issue from a client perspective. With more and more third party software testing companies grooming overtime, the availability of a talented resource pool, making timely deliveries and the commitment to meet the quality expectations of the customer has added many feathers to India’s software testing market. Needless to make a note that Software testing as a field within IT/ITeS has now emerged as a strong area of business solutions and a source of major revenue generator and emerging more as a serious business goal as well an intrinsic part within the IT/ITeS organization culture.

Future Challenges Domain and Technology Perspective The growing demand from within the customer groups as well the growing demand of the organizations to meet and extend their client support has made software testing as a field nothing but a paramount knowledge pool. There is software in almost every aspect of

10

human life such as Insurance, Healthcare, Banking, Financial services, Pharmaceuticals, logistics etc. What this means is that, software developed in each of these Industry verticals need testing – some rigorously and some which can give adequate confidence in the product. At the end, software testing too is spanning across all these Industry verticals. When it comes to domains, the field looks even more lucrative by the amount of challenge each one of them provide to the individuals involved in testing. Therefore, as software testing started to support the development activities in these domain areas, it became increasingly necessary for the test engineers to gain knowledge and specialize within each of these domain areas. This is another area where there is a huge potential for test engineers to grow and make an impact. This led to different teams being formed, some heterogeneous and some homogeneous groups to cater to testing software in these domains. The requirements of the client keep on growing and we as organizations need to meet the demands of the end users to stay in this business. This translates to the whole gamut of technology issues starting from software, hardware to third-party products. Software can be legacy application running on a mainframe or the latest technologies such as .NET, J2EE etc... Technology as we know has advanced from the days of mainframes to today’s applications running on a PC based Linux systems. However, at the same time, applications developed on the legacy technologies have not gone away or have not been replaced with new technology; they are still being used in Banks, Insurance companies

There is software in almost every aspect of human life.... software developed... need testing -- some rigourously and some which can give adequate confident in the product.

WhatIsTesting.com

by Sudhir & Anjan, Product Testing & Documentation : Accelrys, ITPL, Bangalore

and many others. This even complicates the scenarios wherein the vendors have to support both old applications as well as new applications. This tells us how complex it can be for testing to handle such complex set-ups. Besides, the applications run on varied platforms from different vendors including: Windows, Linux, Macintosh, IRIX, AIX, unix & Mainframe. We are currently looking from a macro level. Since with the growing numbers of different flavors of an individual OS (e.g. Linux as such has Linux Workstation2.0/3/0, Advanced Server 2.0/3.0, Enterprise server 2.0/3.0 etc.) the coverage from testing point of view becomes immeasurable. Does that not give a test engineer enough challenge to work on so many platforms? Of course yes, it’s not that developers only code for making an application run over different platforms but the test engineers too have their own way of reaching end to end. Apart from this we can also predict a good deal of exploration in areas like Client-Server, Web (2-tier, 3-tier) applications, Mainframe application and an endless list of such applications.

Conclusion Moving a little off track to give a comparative analysis we see in the past Public Administration has struggled a lot to give itself an individual identity and for a long time shared a part of its glory with either Political science or Sociology. But today it has gained a public reputation and has become a part of everyone’s life. So does it hold true for software testing which has remained cocooned within Software Engineering or Software Development Life Cycle. But now the field has not only gained reputation but also has stood on its own. Software testing as a field has come of age bearing the pain of “We can do without you” to share the glory of “With you we are right through”. A field that is still growing, shining and awaiting to flourish still needs a lot of planning & strategizing from all sections in the Industry and academicians. As we need to remember the following lines: “The Road to Excellence is a never ending Road” ~H. James Harrington, The Improvement Process

What we discussed as far as technology is concerned is only the tip of the iceberg. There seems to be an endless list and mushrooming application areas to be explored in the time to come.

WhatIsTesting.com

11

Team Managment

Are You Ready?

Are You Ready? With a little preparation and teamwork, you can pave the way for a new team member. Use a checklist to make sure you have everything ready before the new team member arrives. ● Compile all necessary documents into one binder that can be given to any new arrivals. ● Be sure to provide some context about the company and the team to help the newbie get acclimated.

12

WhatIsTesting.com

by Michael Bolton, DevelopSense, a Toronto-based consultancy and training firm.

I

f your company is typical, it is hiring more and more temporary workers while scrambling for every bit of efficiency and value. Hiring a contractor to help with temporary problems can be cost-effective, but wasting his or her time undermines the purpose of the exercise and costs hundreds of dollars per day. Moreover, people simply don’t work as well when they feel uninformed, frustrated, or stymied. On the other hand, a contractor who can hit the ground running will make better relationships, will be more effective, and will use your time and hers productively. This article describes ways to prepare for a new contractor; however, the tips here could just as easily be applied to a temporary employee, a new permanent employee, or in some cases, a transfer into your department. I don’t assume any particular job description or title for the person, nor do I assume anything about the person’s technical skill in any given category. Neither should you make such assumptions. A crackerjack developer may be completely oblivious to network configuration issues — especially in a new environment — and a brilliant documentation writer might be flummoxed by a cryptic voicemail system. Your contractor’s time is best used to solve the problem for which you’re hiring her, not her own infrastructural or contractual issues.

agreement. Make sure this, too, is prepared in advance, and that all agree on each of its points before the contractor’s first day. If your human resources department has policies and procedures manual appropriate for your hire, make a copy available to the contractor early in the game. Human resources will typically require various kinds of information from your new hire. Be sure to ask in advance what information they’ll need so that you can pass on any questions to the contractor prior to her arrival. Human resources or security will typically supply ID badges, pass cards, access codes, and keys to the office, restrooms, or other secure facilities. These items should be available and tested. If the contractor is arriving from out of the country, make sure that your human resources department and the contractor have fulfilled all requirements related to visas, work permits, and the like. The accounting people will need to deal with administrative and taxation issues. Make their jobs easier by coordinating and relaying the required information in advance, both from your staff and from the contractor. Issues related to eligibility, tax numbers, tax forms, and withholding information should be sorted out concurrently with the contract.

Contracts and Administrivia Before beginning work, your contractor will typically sign a contract outlining the scope of the work to be done, the deliverables expected, and a schedule detailing when each element of the work should be completed. The more specific the contract, the less opportunity there is for miscommunication and disagreement, and the better your interests are protected. Your contractor may have requirements that are outside the scope of your company’s standard contract, or she may have to provide certain kinds of documents relating to work eligibility. Make sure these kinds of issues are ironed out before the contractor is slated to begin work. Most contracts come with a non-disclosure

WhatIsTesting.com

Restrooms with locks but no keys are scourges upon the land. When I started at one small outfit, it was a week before security could produce a restroom key for me. It wasn’t only a bother for me to ask my new colleagues for a key—I was interrupting their work, too. At this company, the restrooms were locked, but the door to the server room was always open.

13

Are You Ready?

Elizabeth, a European contractor of my acquaintance, was hired to work in an American company. She submitted an invoice, covering salary and expenses, which was rejected on the basis of a couple of questions from the accounting department about her tax withholding status. She dropped in to investigate the problem by speaking with Mark, the accountant who had sent her the mail rejecting the invoice. “I don’t have time,” said Mark. “Not even for just a couple of questions?” asked Elizabeth. “No,” Mark replied. “It would be a different story if you were an employee rather than a contractor. But,” he said imperiously, “I have more important strategic things to do.” This incident occurred at a company where more than half the staff was made up of contractors. Eventually the problem was sorted out, but it meant inconvenience and exasperation for Elizabeth and her manager, and embarrassment for Mark’s manager. On the other hand, Elizabeth’s final report included a subtle recommendation—which Mark’s manager followed—to give Mark less strategic work to do. Whether Mark is currently doing strategic work or not, he’s doing it somewhere else.

14

There are few things that make a contractor happier than being paid on time and with a minimum of hassle. Ask your accounting department what they’ll need from the contractor to pay invoices promptly. Ensure that expense and reimbursement policies are clear from the get-go. If your company uses a standard expense reconciliation form, make sure that a few copies or an electronic template are available to your contractor. Again, your most friendly and helpful contact person in accounting should be available to the contractor to help sort out confusion or difficulties.

IT Issues Notify the IT support department that a new hire is on her way, and request the specific things you’ll need and the date you’ll need them. Most new arrivals will need at least one computer set up, linked to the network, loaded with tools, attached to printers, and tested. If you’re hiring a developer or tester, additional terminals or workstations may be necessary. Don’t forget to tell the IT department about this, and above all, don’t assume that they’ll know what the new arrival needs. Your new arrival will need access to various areas of the network, so make sure she has the appropriate set of access rights. If your company’s security is very restrictive—for example, limited Internet access—notify your contractor beforehand so that other arrangements can be made for Net-based research tools and resources. An analog line might be required for some purposes. If specific tools—examples include compilers, configuration management programs, and testing utilities—are required, have these loaded and tested on the contractor’s machine in advance. Be sure to follow up, verifying that the work has been done and that resources, configuration settings, network policies, and default passwords are documented. Make sure that email accounts have been set

WhatIsTesting.com

We challenge you. At GrapeCity, we challenge you with assignments on the latest .NET and Java technologies, enabling you to emerge a winner in this game called life.

Lines of Business Business Solutions GrapeCity is a software development multinational company

Custom Applications

with over 650 employees across Asia and the United States. For

Mobile Applications

more than 25 years, we have brought optimized software solutions and services to enterprises around the world.

CRM ERP

We rely on four key principles to help our clients achieve their goals: thoroughly understanding our clients’ business objectives, providing highly personalized experiences, maintaining a strong emphasis on quality, and adhering to the highest ethical standards.

Technical Services Mobile Porting Quality Assurance Technical Support

Send your resume to [email protected] now!

GrapeCity India Pvt. Ltd. A-15, Sector 62, Noida 201307 Phone: +91 120 247 0123 Fax: +91 120 247 0124 www.grapecity.com

Component Development Technical Tools

Are You Ready?

up and checked by sending a message to the new arrival’s address, and then check with the IT department to be sure the message has been received in the new mailbox. In addition, obtain a document from the IT department on how to change network login and email passwords.

Note the people who are most likely to be helpful in answering questions or fixing problems, and provide their contact information.

Everyone needs a working telephone and

The new arrival can often help people to notice old bumps in the road. I was brought into a test organization to help identify problems in testing efficiency. There were two types of network drops in the test lab—one for the company’s internal network and one for the test network. These weren’t labeled, since everyone who used the lab knew which was which. I suggested that this shouldn’t be taken for granted as one of the locals helped me find the right connection, and I used a bit of masking tape to label the drops nearest to my workstation. The next day, a fairly senior tester reported a defect that was later found to be spurious, at a fairly significant waste of time and effort. The misunderstanding happened because he had plugged the test machine into the wrong network connection. The day after that, labels magically appeared on all of the network drops.

16

voicemail system. Have the telephone installed, and a number ready for the contractor when she arrives. Make sure that this number is listed in the company directory — and remember that directories can exist on the network, in the voicemail system, and with the receptionist. Despite everyone’s best intentions, a company’s information infrastructure can be complex and confusing to a new arrival. Have the IT department provide the name of a specific person—a guide or sponsor who is helpful and knowledgeable about the company—to help the contractor navigate through network access, email configuration, and printer problems. In addition, make sure that your contractor will be able to find her default printer, email address, network login name and password, and IP address. Try to counsel colleagues against sending email announcements saying things like “This week’s build can be found in the usual places.” Instead, make the details explicit.

Providing Context The contractor has been hired to accomplish some task or solve some problem. If people within your company are having difficulty sorting out an issue, imagine the difficulty that a stranger will face! Even people who have worked inside your company and who know the general environment need background information when they arrive in a new position. This is yet another motivating force for your company to develop clear specifications, design documents, plans, schedules, budgets, and so on. Start by writing a brief description of what the company and your division do, and prepare an organizational chart or departmental roster. At the very least, identify yourself and your direct superiors and reports, along with the other people with whom you expect the contractor will be working. As you’re

WhatIsTesting.com

by Michael Bolton, DevelopSense, a Toronto-based consultancy and training firm.

preparing this information, note the people who are most likely to be helpful in answering questions or fixing problems, and provide their contact information, so as to minimize the amount of time that the contractor must wait—unproductively—for answers. Document the key business processes and workflows associated with the new arrival’s task at hand. If you provide answers early, there will be less time spent wondering and asking questions.

The processes and documents that you prepare for the arrival of one contractor will be highly recyclable. and you’ll quickly have a list that is easily maintained and printed. One of the benefits of traveling, from the point of view of many contractors, is using free time

Don’t get hung up on tools or formatting: A document composed using a text editor or even a pen and paper is infinitely superior to a document that doesn’t exist at all. When offered a choice between a detailed Notepad file and an empty Gantt chart, I’ll pick the former every time. Simply make sure that the information exists, and that it can be found and read easily. Make it a mandate from the beginning, and check with the new arrival often. If the new arrival feels that she is being forced to do detective work, encourage her to keep a log of what she needed to know and what she found. Make sure that one of her deliverables is a list of the things that will be useful to her successor, and be sure to budget time for her to prepare it.

City Guide If your new arrival is coming in from out of town, she’ll feel more welcome with a guide and some idea of where important services can be found. If your company is providing accommodations, make sure that the contractor’s lodgings are quiet, secure, comfortable, and reliable, and that they’re ready for the contractor by the end of the first day of work, at the latest. If arranging accommodations is the contractor’s responsibility, provide a list of facilities in a variety of price ranges. While it’s nice to be close to work, don’t strand the contractor near an industrial park without food or services nearby. Prepare a list of restaurants in the area that cater to a variety of tastes and budgets.

At one company where I was a full-timer, we hosted a number of developers during an outsourced project. We created a list of local restaurants and tourist attractions by sending out email to everyone in the department, soliciting one suggestion from each person. I compiled the suggestions into a single document, removed the duplicates, and posted the results to an internal discussion board. A few of the suggestions inspired still more. Some might suggest that discussion of restaurants on company time and company bulletin boards is wasteful. Our experience was that the onesuggestion technique cost next to no time and was a way of getting to know people—and people who knew each other better almost always worked smarter and harder. We also found that a handy list made it easy to book company gatherings, to provide directions, and so on.

Ask your department for recommendations,

WhatIsTesting.com

17

Are You Ready?

to explore their surroundings. Survey your staff for a number of interesting— and possibly undiscovered— places to visit in the area. If those places are somewhat off the beaten path, ask the person making the recommendation to provide a link to one of the several useful Webbased mapping services. If everyone pitches in a suggestion or two, compiling them into a helpful guide shouldn’t be burdensome. When you’re finished, you should have a complete package of general information that can be passed to any new employee or visitor on paper, via email, or on an internal Web site. Ask the new arrival frequently for feedback on the checklist, and refine it accordingly.

On the Job Make sure that you schedule some time on the contractor’s first day for introductions and a tour of the essentials. You should be prepared to walk around for at least an hour with the new arrival, introducing her to the

The kitchen or lunchroom is not only the place for refueling but also a good place to meet and converse with others. Linger for a while and encourage chatting, even if it’s not directly related to the task at hand. Spend a little petty cash on group lunches, outside coffee breaks, or visits to the pub with your contractor and the rest of your staff. If these informal approaches seem of questionable value, consider that productivity and effectiveness are often based on personal relationships and rapport—do what you can to foster them.

18

people she’ll need to know. An introduction from you is likely to carry more significance than one from a subordinate, so be prepared to do this yourself. Even if you are routinely busy, getting your employees and contractors to work together is a tremendously worthwhile investment. A contractor’s effectiveness often depends upon being able to meet and connect with other people. Most contractors meet their new colleagues in business meetings, but more informal circumstances can lead to friendlier relationships. On the introductory tour, take your time. Cover the important places and people, but be relaxed and try to avoid a hard-and-fast agenda. Spend a few moments introducing the new contractor to each staff member. Explain quite generally what the new arrival is there to do, and describe your staff member’s role, and how each may be able to help the other. Don’t just concentrate on peers and colleagues. Spend a few moments chatting with other useful contacts with whom your new arrival will need to interact—people in network services, reception, the mail room, and accounts payable. As you’re walking around, show the new person all of the useful and necessary places around the office. An early stop should be the restroom. If keys or pass cards are required, make sure that the contractor is given them upon her arrival. The IT department should have set up a default printer and informed you which one it is. Include this on the tour, along with specialty printers and other equipment that might be useful. Also on the tour, include visits to the office supply cupboard, the photocopier, the first aid kit, and the lunchroom. For the duration of the contract, the contractor is a member of your company and your department, and thus should be accorded at least the same kind of assistance and respect given to your permanent employees. If you are the contractor’s supervisor, you must be prepared to go to bat for her, and the rest of your organization should be prepared to take the same approach. No matter what the task, there’s nothing worse than having to research and answer the same questions over and over again. Keep a list of

WhatIsTesting.com

by Michael Bolton, DevelopSense, a Toronto-based consultancy and training firm.

frequently asked questions, their answers, and where someone can go for more information. In many cases the FAQ information can be posted on an internal Web site, captured in a database within workgroup software, or collected in a piece of email. If you don’t have such things set up, consider making the collection of FAQs a task for the contractor, making such a document part of the contractor’s package of deliverables, since your contractor will have plenty of insight on what’s important for a new arrival to know.

complete, can be reused and refined, but rarely will you have to start it again from scratch. You’ll need to review things periodically, but the benefits will be immediate: happier, betterprepared, and—most importantly— more productive workers.

Reuse and Recycle All this might sound like a lot of work— and the first time you do it, it will be. Luckily, the processes and documents that you prepare for the arrival of one contractor will be highly recyclable; they can be used again and again, regardless of whether the new arrival is a temporary or full-time employee. Since you’ll be consulting with other departments, you can exchange ideas and plans, and improve the lot of new arrivals companywide. The work, once

Checklist for a New Team Member

* Policies and procedures manual

Prepare for Arrival

* Expense forms

* Signed contract

* Telephone and voicemail instructions

* HR info from contractor

* Telephone directory, with key contacts highlighted

* Keys, badges, and codes ready and tested

* Expense and reimbursement policy

* Internet policy

* Visa, work permit, and eligibility confirmed

* Password and login instructions

* Resolution of tax issues * Telephone and voicemail set up

* Names of key contact people in IT, HR, Accounting, etc.

* Phone number listed in directory

* Organizational chart

* Set up computer:

* Division responsibilities

* Software resource manuals

networked

* Overview of company

all applications installed and tested

* Key business processes and workflows related to job

email account login passwords printer connection

* Guide to city, if applicable * List of frequently asked questions (with answers)

Documents to Provide

WhatIsTesting.com

19

CORPORATE OVERVIEW

Hexaware is a leading global provider of IT and BPO services. Over 40 of our clients are Fortune 100 companies. We enable clients achieve competitive advantage by co-developing innovative IT/Process capabilities delivered through flexible business models. We focus and have achieved leadership positions in domains such as HR, Banking and Financial Services, Insurance, Leasing and Transportation. On the technology front we specialize in Business Analytics, Enterprise Applications, Application Modernization/Management and Independent Testing. Founded in 1990, we are a US$ 153 Million company with over 4000 employees in 16 locations worldwide. We are currently ranked as No. 11 Software Company in India. TECHNICAL EXPERTISE

Our Key Offerings Industry Industry Specific Specific Solutions Solutions Airlines Manufacturing & High technology Banking & Securities Insurance

Technology Technology Practices Practices

Business Requirement Analysis

BR / FS/SR Specification Gap Analysis Functional Support

Software Testing Process Consulting

Existing Methodology Review Existing Tools Evaluation Process Recommendations

System Integration Testing (SIT)

Black Box - Functional Testing White Box - Interface Testing Grey Box - Regression Testing

Performance Testing

Load Testing Stress/Volume Testing Analysis and Interpretation

Application Management Enterprise Application Integration

Test Automation

People Soft Oracle

Test Management Functional Automation Load Test Automation

UAT Support

UAT Preparation & Execution Roll Out / Production Support User Training

Fail Over / Availability Testing

Multi-tier Availability Low Resource Endurance Tests

Siebel E-Business solutions

‰ Technology – Legacy Systems, Client Server, J2EE and .NET. ‰ Operating Systems – Multiple Virtual System (MVS), UNIX variants and Window based. AUTOMATION TOOLS EXPERTISE

‰ Win Runner, Load Runner and Test Director - Mercury Interactive. ‰ Functional, Performance and Test Management Tools - Rational Software. ‰ Silk Test and Silk Performer from Segue Software.

HITS(HEXAWARE INDEPENDENT TESTING SERVICES) OVERVIEW

¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾

Team of 250+ with varied skill sets Domain focused, Automation driven E2E testing solutions Independent testing labs based in Chennai and Mumbai Showcases projects spanning a diverse spectrum of ¾ Domains – BFS, Insurance, ERP and Airlines ¾ Technologies - Mainframe, J2EE, Siebel, People soft and SAP Expertise in test automation tools like Win runner, Quick Test Pro, Load Runner, Test Director, Rational suite. More than 35% have hands on experience in Mercury tools. Proven Test Methodologies from end-user perspective to meet the customers expectations under real-world business scenario. Offshore – Onsite Delivery model providing significant cost savings More than 1000 Man years experience

Hexaware Technologies Hexaware Towers, 51/3, G.N Chetty Road, TN

Bldg. No. 152, Millennium Business Park, TTC Industrial Area, S 3 "A" Bl k M h

Cover Story

by Rex Black, President, RBCS, Inc., Bulverde, TX

Is bad testing eating your

Pr fits? Using Quality Risks to Guide Testing Effort Testing should focus on mitigating specific risks to the quality of the system. Sequence of test execution should be driven by associated risk...

S

ince the 1980s, when Bill Hetzel and Boris Beizer published their influential books on software testing, we have known that testing should be risk-based. Testing should focus on mitigating specific risks to the quality of the system. The sequence of test execution and the total effort expended on any given test should be driven by the level of risk associated with that test. In the 1990s, people like Rick Craig, Paul Gerrard, and I independently created some ways to achieve systematic risk-based testing, often by adapting ideas from other forms of engineering, such as insurance and medicine. In this article and the next one, we’ll talk about ways to perform risk based testing.

21

Categories of Quality Risks Computer software, hardware, and systems can fail in the most amazing and various ways. Some bugs are not functional problems, but fall into other quality risk categories. We’ll take a look at some categories of quality risks to stimulate our thinking about what to test.

Functionality The most obvious quality risk category is that of functionality. There’s always a chance that the system does not provide some function it should. The function can include a capability, feature, processing, input or output. For example, if you’re testing a calculator

WhatIsTesting.com WhatIsTesting.com

21

Using Quality Risks to Guide Testing Effort

Unacceptable performance load above 500

3

2

Unacceptable performance at any load Unacceptable performance degeneration above time

1

Acceptable performance

Transaction Arrival Rate

1000

Transaction Processing Time

4

Figure 1: Performance bugs program, the lack of an add capability would fall into this category. So would a situation where the add capability was implemented, but the “+” key didn’t cause the function to be activated. The add capability might be implemented and accessible, but might work only on integers, not real numbers. The add operation, when carried out, might give the wrong result, as in 2+2=5. Or there could be some strange side effect where the operation is handled but the result is not at all as expected, as in a divide function where 2 divided by 2 returned one, but in Roman numeral format.

Performance and Reliability In Figure 1, you see a graphical representation of three types of quality risks in the area of

22

performance. The shaded area at the bottom of the figure represents the required performance, which is one second or less transaction processing time at a load of up to 1,000 transactions per minute. Risks to system quality in the area of performance include the possibility that the system responds to input, produces output, or processes data too slowly under all levels of load. The straight dashed line shows this possibility. The system might perform fine up to some level of load, but have an unacceptable non-linearity in the performance curve, as shown by the steep curved dotted line. Finally, the system might perform within specifications during an initial test run, but subsequent tests—when the

WhatIsTesting.com WhatIsTesting.com

22

by Rex Black, President, RBCS, Inc., Bulverde, TX

Reliability problems exist when the system fails to function normally each and every time. system is not rebooted between tests—might reveal unacceptable performance degradation, as shown in the family of three gently curved lines. The risk of performance degradation—and indeed, behavioral degradation of any sort— over time brings us to the reliability category of risks to system quality. Reliability problems exist when the system fails to function normally each and every time. Such a reliability bug can exhibit itself as an intermittent functional problem. Sometimes, such bugs appear after a long period of runtime or after running under heavy load. I had such a problem with Windows 2000 while writing this article. Extended uptime between reboots usually resulted in characters in some applications taking on the wrong font or the wrong font size. You might have seen another type of reliability bug, too, where the system functions normally when it functions, but crashes or hangs unpredictably. Such bugs can occur after some long period of runtime or after running under heavy load. Alternatively, such bugs can be truly random. I recently saw such a bug when a security patch caused reboots on a system every third or fourth time the system booted.

Stress, Capacity, and Volume Risks to system quality in the area of capacity include seeing functionality, performance, or reliability problems due to resource depletion. For example, the performance of operating systems often starts to degrade once the system consumes more than 80% of the available hard drive or memory space. Risks to system quality in the area of volume

include seeing functionality, performance, or reliability problems due to the rate of computational, data, and communication flows. For example, the performance of database management systems often start to degrade when the database management system load exceeds 80% of the rated transactions per minute or the allowed number of simultaneous connections.

Installation and Deinstallation Various things can go wrong when installing programs. These include situations where the program won’t install or the installation process causes damage to the system. The latter kind of problem once was ubiquitous on Windows systems, with the appropriate nickname “DLL hell.” Here, one program overwrote libraries used by another program during its installation process. Deinstallation can also create problems. Sometimes, deinstallation does not completely remove files and undo changes. Sometimes, deinstallation causes damage to the system.

Various things can go wrong during installing & deinstalling programs. Operations We also have those risks associated with recurring operating activities. These include activities carried out at the end of some operational period, such as end-ofquarter or end-of-month closing out of accounts. Problems can arise especially with failures to archive inactive logical records in a safe and recoverable manner, as well as archiving of data that should remain active. In addition, the

WhatIsTesting.com

23

Using Quality Risks to Guide Testing Effort

The effects of changes, even small, localized, isolated changes, are not always small, localized, or isolated. timing of archiving and the calculation of when a period has ended and a new period begun can fail. For systems with databases, there can be regular requirements to compact or repair the databases. In addition, for such systems, operators often must back up data and configurations regularly, along with verifying the restore process. There’s a strong risk of side effects like performance problems related to backup or restore operations, the inaccessibility of some or all system functionality during backup or restore operations, and the failure of backup or restore operations under full system capacity conditions. Sometimes, operations activities happen automatically, in the background. For example, rollback logs for a database often need to be purged automatically to avoid major performance and reliability bugs.

Regression By regression, I mean when a previously-working feature, function, capability, or behavior fails following a change. The problem we have in system testing is that software/hardware systems, being digital, tend to be discrete rather than continuous in their failure modes. The effects of changes, even small, localized, isolated changes, are not always small, localized, or isolated. How many times have you heard a statement like, “How could that have happened? I only changed one line of code.” A small change, even one line of code, can affect the behavior of the rest of the system. A small change can cause incompatible data to enter shared data sets, including dynamic dataflows and static databases, resulting in bugs that

24

exhibit their effects a long way from their origin. Side-effects of changes can impair or even break cohabiting, interacting, or underlying software. The adding of yet-another-feature can lead to systemic performance and capacity issues.

Usability and User Interface One special category of non-functional quality risks relates to the usefulness and usability of the system by the intended user or operator. Specific usability and user interface bugs can vary. In some cases, systems present cumbersome interfaces that do not follow normal or expected workflows, leading to user frustration, confusion, and mistakes. In other cases, functionality, while present in the system, is practically inaccessible to the user

Data quality poses a major category of quality risks for many systems. or operator. Systems can be inappropriately difficult for the users to learn, leading to abandonment, mistakes, and inefficiency. Finally, users and operators can find instructional, help, and error messages that are misleading, confusing, or misspelled.

Data Quality Many systems exist primarily to do interesting things with data. Inputs are processed, transformed, and stored. Databases are queried, records linked through foreign keys, and integrity constraints enforced. Outputs are displayed, printed, and transmitted to other system. So, data quality poses a major category of quality risks for many systems. The system might corrupt or lose data. The system might store bad or nonsense data in a database without integrity constraints. Databases shared across multiple systems can allow data which is

WhatIsTesting.com

by Rex Black, President, RBCS, Inc., Bulverde, TX

valid for one system to be accessed by another system which does not know how to handle that data, resulting in failures that are removed in time and feature space from the source of the problem. Further complicating these situations, it is sometimes difficult to restore the system to working state after a badly-handled error condition. In some cases, the system, in the course of succumbing to the error, damages configuration files or static data stores in a way that’s hard to fix later. The data quality bug in the expense reporting package I mentioned earlier is an example of this kind of misbehavior.

Date and Time Handling In addition to having to handle common errors, many systems must also handle dates. This has created significant information-technology problems, including the infamous “Y2K bugs” that consumed huge proportions of company and government IT budgets in the late 1990s and contributed to the depression in the high-technology sector in the early 2000s. In addition to having to handle once-in-a-hundred-lifetimes type of events like a new millennium, systems must frequently handle events like leap years. Many systems must deal with expiration dates. Licenses expire. Credit cards expire. Insurance policies expire. After some period of time, the right of a bank to disqualify a borrower based on a derogatory like a bankruptcy can also expire. Failure to handle such expiry events properly can expose a company to serious financial and legal risks.

Localization Localization refers to the ability of a system to support local customs, languages, norms, and laws. There are two broad categories of quality risks related to localization. One category relates to the user interface, and the other to operations.

Of course, if the system does not support the character sets used by the local language, we have a localization problem. Those languages that use the Roman alphabet, such as English, German, and Spanish, have single-byte character sets. Some languages that use other alphabets, such as Russian and Hebrew, have single-byte character sets, too. Localization also has operational implications. For example, time zones and time formats vary based on locale. Does the time change in the summer? (In the United States, this is called “daylight savings times,” but it goes by different names around the world.) If so, when does adjusted time begin and end? Are dates written “month-day-year” (as in the United States) or in the more logical “day-month-year” format (as in much of the rest of the world)?

Configuration and Compatibility

Quality risks related to Localization may be in the categories of GUI or operations. Another family of risks to system quality lives in the areas of configuration and compatibility. Different users configure both the hardware and software of systems differently. A family of systems may support various hardware configurations. Often, the number of potential combinations of configurations is huge and testing inappropriate combinations can result in risk of failure in the field.

Standards and Regulatory Compliance More risks to the quality of the system exist in the areas of standards and regulations. A system can work properly but be excluded from the market by failing to meet standards, either

WhatIsTesting.com

25

Using Quality Risks to Guide Testing Effort

effectively (no one will buy it) or legally (the government won’t let anyone buy it or sell it).

Security With the growth of the Internet, awareness of the risks associated with security has increased. Security risks are legion. They include viruses and worms, criminals breaking into servers, vandals causing denial of service attacks, bugging and intercepting e-mail and Internet communications, and more.

Timing and Coordination When you have communicating components, or shared components, timing and coordination between those components is a concern. For example, with an e-commerce system, how long should the system wait between events like mouse-clicks and submitted screens? At what point can it safely conclude that the customer has abandoned the transaction? As another example, consider the automated teller machine. What happens when the central computer times out or the network goes down? What if that happens in the midst of a transaction? If we’re testing an inventory system, what happens when two salespeople try to sell the same item at the same time?

When you have communicating components, timing and coordination between them is a concern. Can You Think of Other Quality Risks? Yes, and so could I. But isn’t this list already too long? What if you tried to cover all the quality risks discussed so far? Could you test them all? Would they fit into the budget and schedule context of the project? Would you end up wasting time and effort on unimportant problems? Surely any attempt to test this whole set of quality risks would be neither effective nor efficient! So, what shall we do? In the sequel to this article, we will discuss some ideas of how to trim the infinite number of things you could test into a finite list of risks you should mitigate.

About the Author:

Documentation For many people the quality of the documentation significantly affects their experience of quality when using the system. Documentation quality problems include being technically incorrect, of course, especially the examples. The user can find the documentation insulting or offensive, teeming with grammatical or spelling errors, or afflicted with distracting cosmetic or formatting problems. Documentation refers not only to hard copy, but also to electronic documentation. Help screens, installation instructions, error messages, and wizards are a form of documentation.

26

With a quarter-century of software and systems engineering experience, Rex Black is president and principal consultant of RBCS, Inc., a leader in software, hardware, and systems testing.

This article is based on an excerpt from Rex Black’s book, Pragmatic Software Testing, published by Wiley

WhatIsTesting.com

That Time of Year

Have I got a deal for you?

Have I got a DEAL for you …? A

s this is our first issue, we are in a unique place—: This issue will set the ground rules for every issue that follows. Future issues will build on this one, so we have to lay the right foundation. I would like to start that foundation with a simple assertion:

The methods and extend of our testing are all choices; you could even say they are trade-offs. Life is all about trades. We trade a “work week” of our lives for a pay check, then we trade that pay check in for a home, for food, and utilities or clothing. We trade some of our freedom for a family. At each stage of the game, we are

28

giving something up – time, money, energy, or effort, in exchange for something we value more. If you don’t think it’s like that in the world of software testing … think again. Having an independent test group splits management attention and costs money, but it also decreases risk and provides an impartial assessment of the quality of the software. Using bug-tracking software costs money and creates the risk of “information overload”, but also ensures that every incident is logged and tracked. With a complete list of incidents, a change control board (CCB) can determine which defects should be fixed in the next release, by priority, but that invariably slows down bug fixes and decreases the personal touch.

WhatIsTesting.com

by Matthew Heusser, Socialtext, California, US

Documented test cases are great, right? We couldn’t live without documented test cases, could we? Well, maybe. Time spent documenting test cases is time that could be spent testing. There is also some evidence to suggest that following the same test scripts again and again leads to thinking about problems in the same way, which could cause a miss. Many times, finding a bug (or a repeatable error condition) is like playing a game of twenty questions; how likely are those scripted test cases to get the right answer if they lack feedback? Requirements-driven testing requires written requirements, and “locks in” the feature set at the beginning of the cycle, when we have the least information about what the users actually need and how the software will support them. Reviews and Inspections of documented test cases slow down the test process, again taking time away from test execution. Exploratory testing requires talent and practice; exploratory testing in the hands of someone without experience could just be a waste of time. How can your team get “better?” Mentoring, Coaching, and Skill Development are all very hard to measure and hard to evaluate. Test Process Improvement is easier to measure, but also of more questionable value. Not all trade-offs are equal; the trick is to carefully weigh the pros and cons before making a decision – to consciously realize what you are trading away, what you are trading for, and if the switch is worth it. The question is, what do you value more? Which outcome is “better” for your company, with your customers, selling your product, at this point in time? I’m harping on this point for a reason. Not because it’s so blatantlyzing obvious (it is), but because so few people are talking about it. Instead of talking about trade-offs, they are talking about “Best Practices”, which, apparently, always solve all of your problems without introducing any new problems. The fact that this defies common sense doesn’t seem

to enter into the equation, and that worries me a bit. Okay, more than a bit. It worries me a lot. It’s downright irresponsible to suggest that there is

Not all trade-offs are equal; the trick is to carefully weigh the pros and cons before making a decision.... one single way to do it – a single one-dimensional scale for goodness. The real suggestion that the “Best Practice” crowd is that you listen to them, doing things the way they suggest, instead of thinking for yourself. The trade-off is your brain for their brain. Keep in mind, the guru never set foot in your office and doesn’t even know what product you make. When pressed on this issue, most gurus admit that, of course, “best” doesn’t really mean “best.” To them, “Best Practice” is just a short way of saying “A practice that is good enough for enough people enough of the time that it’s probably worth considering.” If that were really the case, then there are a number of alternatives; James Bach of Satisfice recently listed several on his blog, which you can find at www.satisfice.com: • “Here’s a practice I find interesting” • “Here’s what I would recommend for this situation” • “Here’s my favorite practice for dealing with {x}” • “Person {x} attributes Practice {Y} for his success. Maybe you’d like to learn about it” Notice that all of those examples included conditionals, and that best practices are free of all conditionals. When someone consistently

WhatIsTesting.com

29

Have I got a deal for you?

At each stage of the game, we are giving something up... in exchange for something we value more. uses the phrase “best practices” and never seems to use any conditionals, my radar goes up … then I go to protect my wallet. Some gurus admit that they don’t know your situation. They explain when a practice might fit and when it might not; they talk about the pros and cons of the practice. Some experts and consultants admit that all new practices involve a trade-off, and they are up-front about it. At WhatIsTesting, we want to follow that example. Every issue of WhatIsTesting will contain some ideas, strategies, techniques, practices, and perhaps some philosophy. The ideas might fit into your workplace, they might not, or perhaps you can use them later, but not now.

7) Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products - http://www.context-driven-testing.com/ The greek philosopher Socrates is credited with saying that the unexamined life is not worth living. We spend thousands of hours a year at work. Examining our work life and striving to constantly improve would seem a natural extension of that, and it is what this magazine is all about. Of course, the choice is yours. Each issue will cost a little bit of your money, and a little bit more of your time. Would you like to trade?

About the Author:

As I mentioned at the beginning of this essay, this issue will set the ground rules for everything that follows, so I want to start it out right. I consider myself a member of the context-driven school of testing, that has the following basic principles: 1) The value of any practice depends on its context. 2) There are good practices in context, but there are no best practices. 3) People, working together, are the most important part of any project’s context. 4) Projects unfold over time in ways that are often not predictable. 5) The product is a solution. If the problem isn’t solved, the product doesn’t work. 6) Good software testing is a challenging intellectual process.

30

Matt Heusser is a Member of the Technical Staff at Socialtext, where he is a lead in the QA group. You can read Matt by email at Matt.Heusser@ gmail.com <mailto:[email protected]> or read his blog online at xndev.blogspot.com

WhatIsTesting.com

Learn How To Test

by Hans Schaefer, Cairman : Norwegian Software Testing Qualifications Board

What a

Tester Should Know, even After Midnight Testing The Normal Way is Not Enough. Systematic testing of software or systems can be learned, just like any engineering discipline. There are tester knowledge certification schemes like the ISTQB certification, which is an internationally recognized, independent and non-profit organization, and there are standards (ISTQB Glossary, BS 7925, IEEE 829, 1008, 1012). At least the books and most of the standards have been around for a long time, and many techniques are widely accepted. This means testing can actually be studied and then executed in some systematic way. This does not mean that the typical testing methods described in the

literature are widely practiced. As test execution is generally one of the last activities in a project, it is very often run under severe budget and time pressure. This means that the testers have a hidden or open motivation to compromise on the thoroughness of their job. For example, as detecting defects always delays both testing and the delivery or acceptance of the product, there is a hidden pressure not to find any defects, or at least not serious ones. Testing is just “done”. This job is then not very motivating, investment in learning is scarce, people try to find a different job, and testing does not improve over time.

WhatIsTesting.com

33

What a Tester Should Know, even After Midnight

Can we Do Testing in a Better Way?

problems and defects, a mindset that questions everything is likely to result in better testing.

Glenford Myers, in 1979, defined testing differently: The aim of Testing is to find defects. He used this definition because it motivates people. Having a negative focus, trying to break the product, leads to finding more and more serious defects than just checking whether the product “works”.

Part of the tester’s task is to report incidents. This is not easy because it requires describing the problem in such a way that the other side accepts it as important enough to do something about it. It means not only collecting more information about the problem but also to think about how to “sell” the bug report to the responsible person.

Most people do as they are told. If they are told to find as many defects as possible, they will try

Test in a better way: - Clearly define the purpose of testing. - Continuously try to learn more. - A mindset that questions everything is likely to result in better testing. - Report the bug with a view to “sell” it to responsible person. to do so. If they are told to get the job done (and explicitly or inherently passing the message that defects delay the progress), people will try not to find defects, or they will overlook many. Thus the first problem is to clearly define the purpose of testing, and make the purpose perfectly clear to the testers.

There is definitely more to it. A tester should always be on the outlook for more challenges in field of testing. This paper is only the beginning of what you may look for.

The Purpose of Testing There are a lot of possible goals for testing. One main, but possibly boring, purpose is to measure the quality of a product. Testing is then considered just a usual measuring device. Just usual measuring is not much fun, but a necessary job, which must be done well. However there are questions a tester should ask, in order to measure optimally. The main question is which quality characteristics are most important to know, and how precisely the measurement must be done. Another definition of testing is trying to create confidence in the product. Confidence, however, is best built by trying to destroy it (and not succeeding in doing so). It is like scientific work: Someone proposes a new theory, and all the educated specialists in the area try all they can to show that it is wrong.

Second problem relates to change. Software products are growing more and more complex. The focus of the requirements is changing, for example emphasizing more security, interoperability and usability. This leads to changes in the requirements on the testing job. Thus, a tester should continuously try to learn more. The next problem is the mindset of testers. As one of the purposes of testing is finding

34

WhatIsTesting.com

Defects clump together: they are social! Defects often have a common cause! Try to find common causes, and then follow them! Where you find defects, dig deeper!

by Hans Schaefer, Cairman : Norwegian Software Testing Qualifications Board

After trying this for some time, unsuccessfully, the new theory is slowly adopted. This view supports Myers’ definition of software testing: Find defects! The approach is a pessimist approach. The pessimist believes «this probably does not work» and tries to show it. Every defect found is then a success. People function by motivation. The definition of testing as actively searching for bugs is motivating, and people find more bugs this way. It works in two ways: One is by designing more destructive or just more test cases. The other one is by having a closer look at the results, analyzing details a not-so-motivated tester would overlook. In the latter case this may mean to analyze results that are not directly visible at the screen, but are deep inside some

A tester should try to find defects! Defects may appear at places where you do not see them easily, i.e. not on screen output!

input and misuse. There are three kinds of tests: The good, the bad and the ugly tests: Good means everything is fine, all inputs are right. Bad means inputs are wrong. The ugly tests is where all wrong things happen simultaneously: Someone restarts the server while you do your reservation and at the same time sets the machine date wrong, … Another way to look at risk is to asses the possible damage of the failure. This may be difficult to analyze but one can make a beginning by asking “what is the worst thing that can happen if this function, feature or request fails?” Better tests forces developers to do better work, inform management about risks, and lead to lower cost of software creation. Testers bring bad News, but that is their job. Nobody loves speed checks on the motorway! But speed checks make our roads safer, and we all benefit.

Continuous Learning Continuous learning is required in nearly any job. But for testers it is absolutely essential.

Testing is risk mitigation. files, databases, buffers or at other places in the network. Defects tend to cluster together. We should take a deeper look in areas where we find defects. The trouble is that this could be a self fulfilling prophecy. We are likely to find more defects because of our focus in that are. One more definition of testing is «measuring risk». This means that testing is a kind of risk mitigation, part of risk management. Testers should have an idea about product risk, as well as risk management. A very basic method for this is looking at the usage profile, and at possible damage. A tester should ask which kind of user would do what and how often. How will different users use the system? How will this vary over time? And it is very important to take into account wrong

What determines risk? What happens if some input is wrong?

In most cases, testing is done somehow systematically, using some black box approach. In addition, test design may follow heuristics. Any black box approach may leave important areas un-covered. Any heuristic is incomplete, as it is dependent on personal experience (or on learnt experience from others. And white box testing does not uncover errors of omission. It all comes down to: If there is some aspect the tester doesn’t know, it is not tested. Hence the tester should know as much as possible. But how?

WhatIsTesting.com

35

What a Tester Should Know, even After Midnight

A tester needs programming experience. There are lots of programming bugs, even after unit testing by programmers. The tester should have an idea of problems posed by a particular programming language. Most testers who have tested software written in C or C++ know that they should check for memory leaks or memory overwriting. Those who used early versions of MFC (Microsoft Foundation Classes) know that the program would crash for any date greater than a particular date “We need to check what that date was and with which version of MFC” The tester needs design experience. Much design is about contracts and communication amongst various modules. If the tester has no idea about architectures and their inherent problems, she will have trouble planning integration tests. The tester needs domain experience. System testing is about implementing the right solution, doing what the domain requires. Can you test a railway interlocking system? (Eh, what does interlocking mean, by the way...?). At least some domain experience is a must to test the system and communicate with different stakeholders.

your colleagues, and go to testing conferences!

A Critical Mindset Don’t believe anything! As a tester, don’t assume anything. It may be wrong! Designer, specifiers, and programmers assume a lot. It may be difficult to ask because you may look stupid asking. Someone who could answer may be far away or not easily available. You wouldn’t even realize that there may be another interpretation. Sometimes people you ask also don’t know, or you get some sarcastic answers. Using the pessimist view, you may as well assume that any assumption is probably wrong. There are ways to overcome the trouble that you may seem stupid. Learn how to deal with people, learn how to interview, learn how to be selfconfident. Ask someone else. Read, review, sleep over it, and try to find different interpretations. You may need a lawyer’s mindset. If you don’t get an answer, have it on your issues Don’t assume! Ask!

The trouble is that this may not be enough. Testing for today’s stakeholders may definitely be not enough. There are totally new ways a system may be used or interfaced with other systems and a tester should try to anticipate at least part of this! A tester should always try to find new ways to look at the object under test, new approaches, and new viewpoints. And finally: testers should try to use the newest approaches and technology. You have to learn them. Read testing books, look for and learn tools, study journals, participate in discussion groups, special interest groups, discuss with

If nobody else asks the right question, you might do so! Think about new possibilities, unknown problems, and the stuff you learn. Think «out of the box»! list. But don’t just assume anything! Don’t take things for granted! And especially don’t believe that the system is probably right

Defects Learn more, about everything! Programming, architecture, new domains, users, tools, anything!

36

Nobody loves the messenger with bad News! As a tester, most of what you report is bad news. The bad News is the bugs, or issues to call them in a neutral way. Textbooks generally handle this area well. There is issue reporting, registration,

WhatIsTesting.com

by Hans Schaefer, Cairman : Norwegian Software Testing Qualifications Board

handling, removal, retesting, regression testing. We know all this. But there is something extra to it, and that is not there in the books: 1 - An issue is only interesting for a tester if it is accepted as a defect and repaired. 2 - There are defects, which are the result of running many test cases in a row in some very special order. The first problem is one of salesmanship and discipline. As a tester, one has a sales job. Nobody is interested in spending any money on repairing defects. They will only be repaired if they are important enough. Thus, as a tester you have to report an issue in such a way that the developer understands that it must be repaired. The damage must look big, the probability of it occurring must look big, and the issue must be easy to repeat. Thus the tester should not just write an issue on the issues list. The tester should think: Are there any other related scenarios, which would make this problem worse? Are there any more stakeholders interested in this? Is this really the whole problem or is there more to it? It may also mean to invent and run some more test cases. Cem Kaner has presented some excellent material on this cause which is available at his website at http://www.kaner.com/pdfs/ BugAdvocacy.pdf Finally, there are the human issues, about diplomacy, politeness etc. A tester should make sure not to hurt anyone personally when reporting an issue.

For every issue (or bug), research more about it! Make sure you report it as a risk, and as the whole risk! Defect reporting is a sales job! Be diplomatic when reporting issues!

The second problem is worse: Sometimes we experience failures, and we cannot recreate them. These issues are called intermittent bugs. They are especially difficult if they introduce system crashes. Upon restarting the system, any corrupted data in the memory may be deleted, destroying the evidence. In many cases, intermittent bugs are the result of longterm corruption of some resource or memory. An example is memory leaks. Some function in the program does not return unused memory when finishing. But because there is a lot of available memory, this can go on for a long time, until the memory is depleted. Other resources may also be depleted. As an example, the Mars Explorer ceased working after 18 days due to too many files accumulated. In many real time embedded systems, the tasks are restarted at certain intervals, in order to cancel out possible corruption of resources. The trouble is that ANY shared resource can be corrupted. It comes down to checking the not very easily visible outputs such as files, memory pointers, buffers, databases, messages to remote systems, registry and many other resources. It could even be the result of a race condition, which depends on the exact timing of some parallel tasks. And intermittent bugs normally require a whole sequence of test cases, not just one input and output. However, if intermittent bugs occur, it is a good idea to be able to rerun the same sequence of test cases, maybe even with the same timing, and do more checking than before. James Bach has a good guide to investigating such problems available at http://blackbox.cs.fit.edu/blog/ james/archives/000197.html. One final problem: You may be wrong yourself. Humans err. Testers are humans. This means you overlook problems; you misunderstand outputs, and some of the problems you think you have found are actually not problems. Be self-critical: Sometimes it is your fault. Mistrust your memory. It is restricted. This means it is better to take notes, to log what you did, what the system did, what

WhatIsTesting.com

37

What a Tester Should Know, even After Midnight

test will be used. If nothing can be said, the test should be directed at any use, testing robustness. This means especially test cases for wrong inputs are interesting.

Analyze even intermittent problems! Log everything you do in testing! Log everything you see, and look at more remote details!

One technique is the basis of most black box testing: Equivalence partitioning. It helps to save

A tester must be able to state what coverage a test has achieved!

Make it possible to rerun your tests, with more logging and analysis tools switched on! is installed etc. You can trust notes more than your memory.

The Late Night Tester’s Toolbox How should a tester work? What should a tester always keep in mind when working? One main trouble is to test “everything”. This is far too much. It can never be achieved. But the tester should have some ideas about what is tested and what not, or what is tested more or less. That means test coverage. There are three very basic concepts of coverage, and they can be applied to any notation for

test effort, and it can be applied to derive other test techniques. As a tester, you should know it, but also be aware that it has caveats: Black box testing may leave out important aspects. You should also be aware that combination testing is interesting. Lee Copeland (Copeland 2004) has published a good introduction. Another big problem is test environment that has not been validated. The test environment should be prepared and tested early. Waiting for the test environment to work can kill any testing Follow the usage profile if possible!

You may be wrong – don’t trust yourself 100%!

If not possible, test for robustness against wrong input.

Take notes of what you do!

example, a control flow diagram, a data flow diagram, a state transition diagram, a call graph, system architecture or a use case. • Basic coverage is executing every box. • The next level is testing every connection.

effort. Also remember that the defect may not be in the object under test, but in the test data or the output analysis. Be self-critical! And finally, there is test automation. Running tests using automated tools helps regression testing. But test automation is more than that:

• This should be the minimum when testing. If there is more time, the next level is combining things, for example pairs of connections. Next, a test should follow the usage profile. This is difficult, especially in module and subsystem testing. But as a tester, one should at least try to get some idea about how the object under

38

WhatIsTesting.com

Equivalence partitioning is a good basic technique! Remember combination testing!

by Hans Schaefer, Cairman : Norwegian Software Testing Qualifications Board

Tools may read specifications and automatically generate test cases. Tools may automatically create or restore test environments. Tools may be used to manage the testing effort and the test material. Remember that automaton can be applied to any part of the software testing life cycle and not just test execution.

Test the test environment – well before test execution! Check you test data!

Selected References • Bach 2005: James Bach. A blog note about possible causes of intermittent bugs: http:// blackbox.cs.fit.edu/blog/james/ • Beizer 95: Boris Beizer, Black Box Testing, John Wiley, 1995 • Better Software Magazine, www. bettersoftware.com. www.stickyminds.com Very practical! • BS7925: British Standard: www. testingstandards.co.uk/bs_7925-1.htm

• ISEB: Information Systems Examinations Board of British Computer Society. http://www.bcs.org/BCS/Products/ Qualifications/ISEB/ has run a certification scheme for software testers since 1999. • ISTQB: www.istqb.org International Software Testing Qualifications Board. Develops and runs an international software tester certification scheme. • ISTQB Glossary: www.istqb.org/fileadmin/ media/glossary-current.pdf • Kaner 99: C. Kaner, J. Falk, H. Q. Nguyen, “Testing Computer Software (3rd ed.), John Wiley, 1999. • Kaner bugadvoc: A presentation about how a tester should report issues. http://www. kaner.com/pdfs/BugAdvocacy.pdf • Myers 79: Glenford Myers: The Art of Software Testing, John Wiley, 1979. • Schaefer 2004; Hans Schaefer, “What Software People should have known about Software Testing 10 years ago - What they definitely should know today. Why they still don’t know it and don’t use it”, EuroSTAR 2004

• Copeland 2004: Lee Copeland, A Practitioner’s Guide to Software Test Design, Artech House, 2004. • Crispin: Lisa Crispin, Tip House, Testing Extreme Programming, Addison-Wesley, 2002, also http://home.att.net/~lisa. crispin/XPTesterBOR.htm

Automate testing tasks! Be aware that there is more to automation than using test robots!

• Gilb 2003: “Testers Rights: What Test should demand from others, and why?”., Keynote at EuroSTAR 2003 • GTB: German Testing Board: www.germantesting-board.info The German Testing Board developed an earlier version of the nowadays-existing ISTQB certification. • IEEE Standards: See www.ieee.org

WhatIsTesting.com

39

Take Control Of Your Quality

4

Virtusa

uality Assurance Services

Enterprises know well that quality can make or break customer loyalty and brand reputation. However, the growing complexity, constantly changing market needs, and time to market pressures make software quality an elusive challenge. Virtusa QA lifecycle services allow you to take control of your QA processes, projects, people, and priorities, while strengthening your QA capabilities, minimizing risk and reducing cost.

QA Assessment

QA Offerings

- QA Outsourcing Evaluation - Tools Evaluation - Quick Wins Workshops

QA Ass ess men t

QA Process optimization - Process Improvement Framework - Health Check Assessment & Scorecard

QA Pro ces s Opt imiz atio n

Testing & Test Automation Tes tin g & Tes t Au tom ati on

- Functional Automated Testing Manual Testing - Performance - Certification - Internationalization - Usability

QA Ou tso ur cin g

QA Outsourcing - SQA Outsourcing and Management - Test Team Establishment/Augmentation - ‘Quick Win’ Test Project Consultancy

Mission-Critical Software Product QA Heritage Leverage Virtusa’s eight years of experience in testing and certifying hundreds of commercial products and mission-critical solutions to rapidly improve QA effcienceis. Reduce cost and time to market by leveraging Virtusa’s global delivery model, world-class QA professionals, and stateof-the-art labs, tools and technologies.

Offering Various domain & Tools Expertise Retail, 5% Publishing, 15%

Health care, 15%

State Of The Art SQA Labs

Shipping, 5% Financial, 30%

Insurance, 15%

Telecom, 15%

About Virtusa Virtusa is a global provider of custom IT solutions and outsourcing services to Global 2000 enterprises and software vendors in the financial services, insurance, media and communications, high technology and other industries. Our deep domain strength, custom solutions and services, and a unique platforming methodology enable our clients to reduce costs, improve business performance and accelerate revenue generation. We deliver these results by integrating a consultative approach with our global delivery model.

© 2006 Virtusa Corporation. All rights reserved.

2000 West Park Drive, Westborough, MA 01581 Phone: 508 389 7300 Fax: 508 366 9901 www.virtusa.com Contact: [email protected]

Learn How To Test

by Erik van Veenendaal, Improve Quality Services BV : The Netherlands

Finally

Usability Testing? will it finally happen?

A

new testing magazine that has chosen web testing as its central theme for the first issue! I assume that this also means attention is provided to one of the most critical success factors for websites: usability and thus usability testing. In fact usability has been identified in various surveys as one of the main reasons for project success in any IT project! For a number of years I’ve been lecturing on this topic at test conferences. At EuroSTAR, I have received the best tutorial award for Discount Usability Testing and our company, Improve Quality Services, regularly runs courses on Usability Testing. Yet, I don’t have the impression that all this pioneering work has had much impact on the testing community. During the last EuroSTAR conference, an usability expert from South Africa stated that “usability (testing) will only start to be performed and receive the right level of attention, when testing for functionality is under control”. An interesting thought, which as far as I’m concerned, is very much true. Perhaps we were just not ready for it, but are we ready now…? It remains strange that when no less than 40% of the

software code that is being developed worldwide is directly or indirectly related to the userinterface, we still only dedicate a minimum percentage of our test effort to usability testing. Yet, there are many so-called best practices available. Last week I was involved in a usability test that very clearly showed that thorough attention for usability throughout the development process (requirements, standards, expert reviews, usability testing) can deliver great results. The users were enthusiastic regarding the new system that supported their tasks in an efficient, effective, and user-friendly manner. This is also possible in real-life ITprojects! As far as I’m concerned every (senior) tester that operates at system level should, in addition to the conventional functional test design techniques (boundary value analysis, equivalence partitioning, decision tables etc.), also have the level of knowledge and skills that allows them to apply a number of relatively easy to use usability test techniques. The first things that come to my mind in this context are the Heuristic Evaluation technique developed by Jacob Nielsen (www.useit.

WhatIsTesting.com

41

Finally Usability Testing? Will it finally happen?

com) and the SUMI questionnaire method (www. improveqs.nl). Both techniques are relatively easy to learn, do not require more than one or two days of testing effort and therefore have a great cost/benefit ratio. Recently, the British Computer Society (BCS) Special Interest Group in Software Testing developed a testing standard in the area of usability testing that is certainly worthwhile to get acquainted with (www. testingstandards.co.uk). Sufficient possibilities exist, therefore, to make a good, low-cost but thorough start with usability testing; preferably in co-operation with developers. Prevention is always better than cure.

42

During a recent conference, a number of enthusiastic former participants of my usability courses told me they were now using the techniques mentioned above in their test projects with good results. Usability testing, will it finally happen? For queries or comments you can contact Erik van Veenendaal ([email protected]) A paper on discount usability testing can be downloaded from www.improveqs.nl

WhatIsTesting.com

Canarys Automation Pvt. Ltd, # 135, 7th Main, 4th Block Jayanagar, Bangalore – 560011 Ph: +91-80-26539915, +91-80-26642922 [email protected]; [email protected] Canarys is glad to inform that we are Partners to Radview Software, providing Sales & Support Services to customers in India. Please find a brief on the Black Box Testing Solutions offered by Radview:

Only RadView provides software-testing tools that feature JavaScript test-agenda creation for performance testing, load testing, and functional testing of website applications. Throughout the web application development lifecycle, the seamless integration between RadView's performance testing / load testing products (WebLOAD and WebRM) and functional testing product (WebFT) provides you with the necessary tools to make certain your web application is scalable and robust.

Award-winning WebLOAD can help companies deploy high performing e-business applications by modeling and anticipating the real-life demands on their applications. By accurately simulating Internet user behavior and predicting capacity requirements, WebLOAD reports bottlenecks, constraints, and weak links within an application – before they cost you downtime, lost sales, or more importantly, lost customers.

WebFT is the industry's first Web-centric testing solution that efficiently verifies the functional accuracy of Web applications and ensures applications perform as expected. Speed implementation time — use of open standards scripting language reduces the amount of staff training required Improve quality — resulting from higher quality Web applications Reduce test script development costs — leverage single transferable scripts for fast, easy reuse of test scenarios between development teams Speed Web application deployment — streamline the testing process through the use of single transferable scripts. For more information please contact: Canarys Automation Pvt. Ltd, # 135, 7th Main, 4th Block Jayanagar, Bangalore – 560011 Ph: +91-80-26539915, +91-80-26642922

Some Things I’ve Learned in Software Testing

Some Things

I’ve Learned in

Software Testing S

everal years ago, when I started out as a software tester, my goals for learning software testing were clear. I needed to learn how a software company operated, learn about the software I was assigned to test, and learn how to test software. After a while, I enjoyed finding and reporting difficult-to-reproduce bugs; I felt pleased that we were finding and fixing issues that could negatively affect customers. How did I go from a nervous student to becoming a confident tester? At first, I read everything I could about testing, the product we were working on, and the systems we used. When I couldn’t find anything to read, I asked coworkers to teach me things. I also learned a lot through mentoring from more senior developers and testers.

In the end, I learned the most by doing. (I still do.) When I first started out in testing, my view of what a bug looked like was quite narrow. Like many users, I felt

44

that any error messages that appeared were due to mistakes I made using the software. Then, a senior tester encouraged me to follow my instincts: he told me that if something bothered me, I should log it as a potential problem instead of letting it go by. As I tested, I learned that when I had a nagging feeling that something was wrong, there usually was a problem in the software. Sometimes it was a usability issue, or an awkward spot in the application that needed more investigation, but whatever the issue, it was almost always worth taking the time to investigate further. Even now, it is tempting not to listen to that voice inside that tells me something is wrong with the software, but every time I don’t pay attention, I miss at least one bug that I regret later. Occasionally, my instinct is wrong, but more often than not, that sense of feeling uncomfortable with the software helps me uncover important information.

WhatIsTesting.com

I also learn a lot from my

by Jonathan Kohl, Software Consultant, Kohl Concepts Inc : Calgary, Canada

mistakes, especially when I try something, fail and try something else. Do you learn from your mistakes and from trying new techniques? Who can you talk to in the organization that can help you learn more about testing?

There is More Than One Way to Test Software It didn’t take long for my experience to show me that much of popular software testing information is really folklore. It was interesting to read, but I soon discovered that when I tried to apply popular testing wisdom (frequently called “Quality Assurance”) on real projects, my efforts failed. One practice involved writing all the test cases first (prescripting test cases) based on the project documentation, and then testing using these procedural test scripts later. Trying this practice caused me a lot of problems. I found that testers would run out of time to actually test because they were so busy writing about what they were going to test. As things changed over the life of the project, we got bogged down maintaining documentation. And, when we started testing, these pre-scripted test cases really narrowed our focus, often making us miss entire classes of tests. In the software start-up I was working for, it was important for each member of the team to provide value to the goals of the business. A development manager then complained to me that we spent so much time developing and maintaining test cases that we didn’t do nearly enough actual testing. Once I took that to heart, the business was pleased with the results I provided by using exploratory testing to focus on what was important to the business and to the customer. Previously, I had focused on writing lots of documentation (test plans and test cases) before testing. At first it felt a bit wrong, like we weren’t following “the rules,” but the development team and our customers were happier with our results. That carried more weight with me.

I also read statements like: “Quality Assurance is much more than software testing.” In this line of thinking, it is common to suggest that QA should be responsible for the development process, that quality is only as good as the process it follows. However, in my testing experience working for small companies whose survival depended on satisfying paying customers, the process took a back seat to the product. We strove for product excellence over process excellence. If a process helped us deliver a product that met business goals, we followed it. If a process got in the way of those business goals, we scrapped it. Everything we did needed to be aligned with the goals of the business if we were to survive. This key lesson shaped my thinking as a software tester. My desire to learn more about software testing grew because I learned the most and provided the best business results from actually using and testing software. I wanted to find more techniques compatible with what I was learning about software testing. So I reread the first book I’d ever read on software testing: Testing Computer Software by Cem Kaner, Jack Falk and Hung Nguyen. Again it resonated with me, and I began to read everything I could find that was written by Cem Kaner. Through him, I discovered James Bach and later Brian Marick. That was when I found my niche. These were people who thought about testing the way I did. My own experiences and ideas were echoed in their work. Their writing was true to the point and backed by experience, but most importantly, when I tried what they recommended, it worked. These three testing thinkers, along with Bret Pettichord, went on to found the Context-Driven Testing School1. The Context-Driven Testing School has identified different schools of thought in software testing. Bret Pettichord has outlined four of them in his Four Schools of Software Testing2 presentation: • Analytical School • Factory School

WhatIsTesting.com

45

Some Things I’ve Learned in Software Testing

• Quality School • Context-Driven School The Analytical School draws heavily on mathematics, uses a lot of tools and techniques for code coverage and uses metrics to try to measure conformance to a specification. This school proposes that software reliability can be calculated. Some testers in this school use complex mathematical models for quality prediction. The Factory School generally expresses that that testing needs to be predictable, repeatable and software is validated from written requirements. This school of thought is often interpreted so that testing becomes a clerical task. (Note: many Agile and Lean testing ideas are variations of this school of thought. This shouldn’t be surprising. Many of the ideas in this movement come straight from manufacturing where automation, repeatable processes and eliminating waste are important factors.) The Quality School encourages conformance to a particular process. Often, the tester’s job is to police the process, and make sure that everyone is following it. The belief is that if we follow the process properly, our product quality goals will succeed as well. The Context-Driven School believes testing is a skilled activity, and the usefulness of a practice is dependant on the context it is used in. Not all companies developing software are the same, and it is irresponsible to try to standardize processes for all software development. Different companies have very different needs and those needs are far too diverse to standardize to one true way of testing. Testers in this school use whatever testing tools or techniques are the most effective to help them reach their testing goals. Since Four Schools of Software Testing was published, Cem Kaner has identified TestDriven Development as another school of

46

testing thought3. Bret Pettichord identifies this as the “Agile” school of thought. There may be other schools as well. What is important as a tester is to understand there are different ways of thinking about and communicating testing ideas, and recognize which ones you identify with. That will help you get a focus on what to learn, and will help you understand where others are coming from. Some people find the identification of schools of thought in testing objectionable, and this work by the contextualists is somewhat controversial. When I first discovered it I was reminded of different schools of thought in other disciplines that I studied in university. For example, in philosophy classes, when I learned of another school of thought in logic, I wanted to learn as much as I could to see if it altered my thinking. If I found ideas in a particular school of thought that resonated with me, I added them to my own repertoire. The schools of thought in testing idea shouldn’t be something that bothers us as testers; instead, it can be used as a model for learning. If I get stuck studying the same old techniques and processes, I can use this model to identify other schools of thought that I am less familiar with, and begin researching them. As a tester, what works for you, in a given situation? Have some techniques worked well on some projects, but not so well on others? Is there a particular school of thought that you haven’t studied before that might help you on what you are testing right now?

Software Testing is Multidisciplinary I once worked on an article on tester roles with my colleague Michael Kelly, who is also a software tester. It became clear to us how many different disciplines software testers draw from. Different testers described the following disciplines as influencing their work:

WhatIsTesting.com

by Jonathan Kohl, Software Consultant, Kohl Concepts Inc : Calgary, Canada

general systems thinking, history, software development, construction, law, psychology, family therapy, adult education theory, writing, writing and performing music, experimental science, among others. Older, more established disciplines have often already dealt the same kinds of problems we are trying to deal with in software development. Applying successful principles from other disciplines can be a rewarding source of learning. I often explain software testing ideas by using an intersection of ideas from different fields. Three that I think of frequently are philosophy, business and technology. Testing, like philosophy, involves asking questions. Not only do I question the assumptions made in the product I’m testing, in our development efforts, and in the process we adhere to, but I constantly challenge my own assumptions about testing. Studying philosophy and practicing critical thinking skills especially help me when I’m generating test ideas. Business considerations help me focus the work I’m doing. Am I adding value on a team? Is what I’m doing helping or hindering the goals of the organization? Is my testing focusing on what is important to the goals of the company, or am I blindly following a process? Understanding how businesses work, how they market themselves, and how they stay functioning is important to testing. Technology and testing are related because we use tools, and interact with designers, developers and users in order to test. We rely on computer technology not only in software development, but in most businesses situations. As testers, we must have a certain level of skill to be able to use technology. Since technology moves so quickly, it is difficult to keep up and it seems like there are infinite technological areas to explore as testers. We might specialize in testing a particular technology, or we may depend on certain tools for our testing. Philosophy, business and technology are areas

that influence me, and provide enormous possibilities for learning. My background in music and sports, and my education also influences the way I approach learning and testing. What are your influences in life, and how much do you draw on these other influences when testing software? What other disciplines remind you of your work in software testing?

Testers Can Learn From Other Roles I’ve worked in roles other than software tester, and each of these taught me something important about software testing. Working as a programmer, I learned how to look at applications differently. Understanding the application from a source code perspective helped me develop a catalogue of potential areas in a particular programming language that are more susceptible to bugs. It also provides a better understanding of where to find useful information for reporting a bug. For example, web applications usually have a server log that may have debugging information useful for reporting problems. As a tester wearing a developer hat, I was the recipient of bug reports, rather than a generator. It helped me understand how a poor bug report can frustrate developers. In a technical writing role, I learned about design and how design (UI or architectural) can affect the quality of a product. I noticed a correlation between complicated designs and the amount of bugs found. I learned that if I had trouble writing user documentation for a particular feature, it would likely be a source of bugs. Writing down requirements and design ideas from the point of view of a user exposed complexity. Working in sales taught me about the big picture, about how a software tester can serve the business goals of an organization. A company does not necessarily survive because they are such-and-such CMM level, or follow the 12 practices of Extreme Programming, or because

WhatIsTesting.com

47

Some Things I’ve Learned in Software Testing

they use .NET or Java, or other technical or procedural issues we find so important in development. When I worked in sales, the harsh reality was that a company survives by fulfilling an important equation: Revenues – Expenses = Profit This equation must be greater than zero in order for the company to survive. I learned customers cared that the product helped them solve a particular problem. They didn’t care what technology was used to solve it. They cared that the product was easy-to-use, that it helped them in their work, and that it was affordable. It was interesting to see that they would put up with application failures, but that they were not as tolerant of usability issues. When you deal with customers, you get a different view of the product. Testers serve customers, and it is in our best interests to be sure that they are satisfied and impressed with our product. I learned that it is much more important to be testing what is important to the customer rather than trying to fulfill some generally-accepted testing practice. It is absolutely paramount to do software testing from a user’s perspective. Testers need to understand the company, what its goals are, and to figure out how testing work maps to those goals. Understanding how an application works from a technical perspective, and being able to describe that in a document helps me test more effectively. It helps me ask questions that people I work with find useful. Working directly with potential and existing customers has helped me focus on their needs in my testing. Understanding the goals of the business helps me prioritize my mission in testing on a particular task. What other jobs have you had other than in software development and testing? What have you learned on those jobs that you could apply to testing? Are there specific skills you developed in other jobs that give you an advantage as a tester?

48

Practicing Software Testing is Important We can also learn by practicing software testing skills. For example, I regularly practice my testing skills, and ask my testing friends to help me develop my skills. One of these friends is James Bach, who taught me how to be more consistent with my testing approach. He found that when he gave me a number of practice testing problems, my test approach was inconsistent. James helped me develop my testing ideas into a consistent system when solving different kinds of testing problems. Now when I am faced with a testing problem, I have a strategy and a game-plan instead of an adhoc approach. Practicing testing with someone else helped identify an area of testing weakness for me to address and improve. I’ve also worked with programmers and asked for feedback on how to provide better information when logging bugs. Early in my career when I was first testing web applications, I wasn’t investigating beyond the error I saw on the screen. One of my programmer colleagues taught me how to find more information in log files. I practiced developing my web testing investigation skills by: • Finding relevant server logs on different web application architectures • Reading error messages in the server logs • Changing the level of logging to get more information on the server • Saving relevant information and talking about possible causes with administrators and developers Because I practiced, when I next tested the web application, gathering relevant information from server logs to use for a bug report was second nature. It took much less time to do after I had practiced those skills. This is just one very simple example, but it helped my testing effectiveness. There are many skills we can develop, and many techniques we can practice. What specific skills you rely

WhatIsTesting.com

by Jonathan Kohl, Software Consultant, Kohl Concepts Inc : Calgary, Canada

on as a tester? Can you name them? Can you identify technical skills, investigation and problem-solving skills and softer skills related to communication, and working with others?

Conclusion At the end of each section of this article, I have posed questions for you, the reader. If you would like to learn more about software testing, I urge you to answer these questions as best you can. Take out a pencil and paper, and write down your answers to the questions. Read them over, and see if there are any new areas for study for you to pursue in testing. You should have some new ideas about software testing you’d like to learn more about. Do you want to learn more about testing techniques? How about different schools of thought or definitions on testing? Are you interested in developing your technical or programming skills? Would you like to learn to use a test automation tool? How about learning more about the end users of your software? With software testing, there are many avenues for learning, and the more you learn, the more you realize how much more there is to learn. I constantly learn more about software testing and I hope I have spurred your thinking on how you can learn more about software testing too. Even though I am no longer a beginner I still do the following:

Learning software testing is a rewarding, career-long journey. We all learn differently, and I encourage you to try different ideas and see what works for you. Don’t be afraid to fail when you try something new; we often learn more from our failures than successes. Quality guru W. Edwards Deming said: “Drive out fear.” Don’t be afraid to try something new, if you have an open mind, you are guaranteed to learn something from it.

About the Author Jonathan Kohl is the founder and principal software testing consultant with Kohl Concepts Inc., based in Calgary, Alberta, Canada. A noted testing thinker, Jonathan is recognized as a leader in the exploratory testing community. He is a popular author and speaker who believes that testing is a challenging intellectual craft. Jonathan’s blog on software development and testing issues is one of the most well-read testing blogs in the industry. Jonathan is a regular contributor to Better Software magazine, as an author and technical editor. Check out other writings at www.kohl.ca.

Notes 1

• Read about software testing and other disciplines • Attend and contribute to workshops and conferences on software testing • Identify skills and techniques I am using when testing, and explain them to colleagues • Participate in mailing lists, forums and other communities of practice related to software testing

2

3

Kaner, Bach, Pettichord, Marick. The Seven Basic Principles of the ContextDriven School http://www.context-driventesting.com/ Pettichord, Bret.(2003-2007.) Four Schools of Software Testing http://www.io.com/ ~wazmo/papers/four_schools.pdf Kohl, Jonathan. (2005) TDD - A Fifth School of Software Testing http://www. kohl.ca/blog/archives/000087.html

• Work with others skilled practitioners who constantly challenge me to do better • Practice, practice, practice

WhatIsTesting.com

49

Sierra Atlantic Testing Competency Center Our Testing Competency With over 12 years of experience, Sierra Atlantic is an established offshoring services company using CMM Level 5, ISO 9001:2002, and BS7799 certified process. Using our testing competency center SATIN (Sierra Atlantic Testing Intelligence), we create speedy, low cost and high-quality software. This in turn is delivered through our well-defined engagement models and process methodologies and incorporates the industry’s best practices.

State-of-the-Art Facilities Our state-of-the-art Interoperability Lab provides compatibility / interoperability testing services to software product companies. The scope of this service includes: x Hardware / OS platforms x Enterprise Applications x Middleware platforms x Application Servers x Web Servers x Databases x Browsers With over 250 person years of experience in software testing services, our resources have indepth understanding of various testing methodologies and technologies. The following graph illustrates this: 7%

8%

7% 10% 28%

Only manual testing WinRunner Test Director QTP LoadRunner Silk Test Rational Robot

23% 17%

The following instances help demonstrate our competency. A leading Enterprise Application product vendor Sierra Atlantic’s Interoperability Lab enabled the testing of a leading Enterprise Application vendor’s product on platforms such as Sun Solaris, HP-UX and IBM-AIX. This testing allowed the customer to remotely log in and execute tests using its staff without having to build its own test environments, an expensive and time-consuming process. A leading portal product vendor This client of ours required its product to be tested and certified across a variety of web servers and platforms. The product also required to be tested for compliance under Rule 508. Sierra Atlantic’s Interoperability Lab created multiple functional and compliance test cases and an automation strategy based on Mercury test automation tools. As a result, the client was able to cut in-house costs, and the testing rendered its product totally compatible and fully interoperable with other leading enterprise applications and infrastructure products.

About Sierra Atlantic Founded in 1993, Sierra Atlantic is a leader in offshoring enterprise applications, helping our customers optimize their investments in Oracle, PeopleSoft, SAP and Siebel. We offer complete lifecycle application management solutions -- development, implementation, integration, upgrade and support -- using our NShore™ methodology. We integrate these point solutions into Application Networks® that enable seamless business processes within and outside the enterprise.

by Mrityunjay

:

QA is More Challenging than

Development

I

come from a background in product development and have been in QA for about a year or so, that too in a senior management capacity and not necessarily an engineer. It saddens me when I interview some good QA engineers and they seem almost apologetic about the fact that they are into testing, and more so when they are into manual testing and not automation. This probably stems from the fact that writing code is considered as the best (and smartest) aspect of software development and anyone not doing that is considered less smart. I started thinking about whether this is really the case, that writing code is the smartest part of the software development lifecycle and what I came out was the reverse; that writing code is probably the easier part of the development, it is testing and Quality Assurance (QA) which is tougher. This is based on my discussions with the QA engineers and leads, looking at the

product functionality and issues first hand, and delivering a complex product to the market. I am very convinced that in a real world, it is easier to don a developer hat than a QA hat. In this article, I attempt to show in a very simplistic way what I mean by this statement. Of course I say real world, because in an ideal world, a Dev will be much more like a QA and QA will be much more like a Dev, and they have joint ownership of quality of the product. The company I work in doesn’t claim to be the ideal company, but I find it much closer to it than many other companies I have seen. Let me start with an example. We are given a functionality to implement: List all prime numbers between two given numbers X and Y. Now here is what a developer has to deal with: ● Find/implement an algorithm to check if a given number is a prime number. ● Implement a logic which cycles through

WhatIsTesting.com

51

QA is More Challenging than Development

Test Entry

Design

Test Entry

F I

Functional Requirement

G Test Entry

U Design

R Test Entry

Product Requirement

E -

Test Entry

1

Design

Test Entry

Functional Requirement Test Entry

Design

Test Entry

every number between X and Y, and apply the algorithm in #1 above.

risky since some of the options are being left out and hence some issues may still remain.

● Unit test this with some combinations of X and Y and make sure output makes sense.

A QA engineer who is a white box tester may look at the code and decide there are 232 options only because the developer used a long variable. This is nowhere better than infinity. Some choices have to be exercised, probably based on decision points in the code. Again a risky affair, as QA is likely to miss many, including non-function, parts, like dependency on CPU and memory, impact of running on say

A QA engineer who relies on black box testing and hasn’t looked at the implementation will know that there are infinite inputs to this implementation and so he/she will try to pick some intelligent inputs which are more likely to produce issues (think equivalent partitioning, boundary values, etc). However, this is decidedly

52

WhatIsTesting.com

by Mrityunjay

:

Japanese OS, etc. Note some of the differences in the roles in a developer and QA here: ● A developer decides on an algorithm and unambiguously sticks to it, implements and delivers to QA. For a QA, life is full of ambiguities and risk-taking. A week-atheart QA is not a good QA. ● A QA has to second-guess Dev to figure out what are the likely error scenarios if he/she wants to avoid running an infiniteinput problem. This is analogous to 20questions game, where you think of a word and someone has to guess it by asking only 20 questions. This is tough. ● Think of the implication if a developer or a QA goofs up. If a Dev goofs up, it is supposed to be caught by QA, causing bugs, design changes, etc, a costly affair indeed. However, if a QA does so, customers hit those defects, and this is a much higher cost to the company, in terms of money, credibility and market share. Which role is easier? In a complex product, locating a bug is like searching for a needle in the haysack; you can apply heuristics, but it still is a hard and challenging work, scary too, when you know that a missed needle can cause immense customer frustration, and management backlash. Let’s look at it from another perspective. The waterfall model of product development is a tree structure. A product requirement spawns many functional requirements, each functional requirement spawns many design and algorithm pieces, which in turn spawn many test cases (see fig 1). Thus, even in such a simplified scenario (where next phase only doubles the entities), a product requirement generates 8 test entities for verification. No wonder then that the QA has to play the game of probabilities: trying to find maximum number of high impact defects

in the given time and money through various heuristics and techniques, none of which are scientifically proven to discover all the defects. This is decidedly tougher job than development, since the choices and the implications of choices are much better known and understood when a developer chooses a specific algorithm and its implementation. Given the fact that a functional requirement is broadly defined, it is easier for a developer to meet those requirements and verify they are met, than for a QA to verify developers’ output, since they are very precisely defined (algorithms and code) and hence so many more ways of proving failure exist. To summarize my arguments above, there are two basic points I have made: ● It is easier to drop a needle in a haysack (or bug in a product) than to find it (discover defect before release). The latter requires you to be much smarter, customer-focused, with good problem-solving abilities. ● QA problem is an undefined problem and hence solving it takes very different skills and lots of smartness in addition to hard work. However, don’t get me wrong: these arguments take the example of a typical developer role and a typical QA role and not of individuals who play them. As I have mentioned before, a good developer indeed dons the QA hat many times, verifies the algorithms he/she has produced and only then allows it to be given to QA. So do not use these arguments against any individual; I have been a developer, and I would not love to hear these arguments because I thought I donned QA hats many times to produce good quality code. Same goes for many of my developer friends out there. Now that we know doing QA takes a lot of smartness and skill, what does it take to be a great QA? Well, the trick is to think like a developer sometimes, and that will be the topic for a forthcoming issue.

WhatIsTesting.com

53

Is scripted testing bad?

25

20

15

10

5

Is

0

scripted testing

W

bad?

hen I started my testing career over a decade ago, my manager gave me two directives:

● Raise thirty bugs every day ● 80% of your bugs should be coming from test cases that you would have written prior to the test execution While I was able to accomplish the task of raising thirty bugs every day, I have not been able to achieve the second target even after twelve years of trying really hard. To figure out what was going wrong I analyzed the bugs, created checklists, got my test cases reviewed, learnt about various test design techniques and used them. Sure, I was able to improve the percentage of bugs being found but I never could achieve a figure anywhere close to 80%. This made me question the assumption that most of the bugs can be caught by pre-written test cases. Many of

54

the bugs I caught were the result of conditions that occur in the real world but are impossible to think about while designing test cases. And there were many bugs that I did not find rather the bugs found me. With these findings my respect for writing test cases took a beating. Exploratory testing had begun getting talked about when I had formed my own philosophy and technique of mixing test cases and adhoc testing and it struck a chord immediately. Exploratory testing described what I was doing all my life. Now I had a name for it. However, there was another trend that I noticed - scripted testing, which I had been doing all my life, started getting painted as a bad practice. To set the context right, let me describe what is generally meant by scripted and exploratory testing. “Exploratory testing is simultaneous learning, test design, and test execution” www.satisfice.com/articles/et-article.pdf

WhatIsTesting.com

by Vipul Kocher, CEO, PureTesting, NOIDA, India

“Scripted testing means that learning and test design happens prior to test execution.” http://en.wikipedia.org/wiki/Software_testing Before you read on the rest of the article, please pause for a moment and answer honestly if you consider scripted testing a bad practice. If you are careful about the contexts you are likely to have answered “It depends… scripted testing might be useful in some situations” and that is the truth. Unfortunately, contexts in which scripted testing is useful do not often get elaborated resulting in a message “scripted testing is bad.” Today scripted testing is taking a beating from exploratory testing and to take a stance against it is seen as a sign of enlightenment. The stance may vary from “scripted testing is bad” to “amount of scripted testing to be done depends on context” but in all cases the underlying message that gets conveyed is “scripted testing is bad.” It is unfortunate that there are very few advocates of scripted testing who are actively writing in favor of scripted testing or explaining the contexts in which scripted testing is good.

Is scripted testing really bad? Before we can answer the question “is scripted testing bad” we first need to agree on the definitions of scripted testing and also understand the contexts in which scripted testing is good or bad.

Scripted and Exploratory Testing: definition is important Without going into definitions of scripted and exploratory testing the only questions I would like to ask are: ● Who has created the definitions of scripted testing? ● Who has defined exploratory testing? If the definition of scripted testing comes from the exploratory camp or vice-versa, the definition would naturally portray one as more negative than the other. The definitions put scripted

Can there be scriptedexploratory testing continuum: meaning thereby, scripted and exploratory testing can be mixed in any percentage ? testing at one extreme and exploratory testing at the other extreme. Then, there is this idea of scripted-exploratory testing continuum which means that scripted and exploratory testing can be mixed in any percentage Unfortunately there exists no definition for this type of testing. But isn’t this nameless, definition-less continuum what most testers, at least the good ones, have been using? The process being ‘create scripted tests, use exploratory methods to create more tests, run them, learn from them, create more tests and run purely exploratory tests...’ To me it would be more natural if somebody provided a definition for this mix of “scripted” and “exploratory” testing. Let me call this type of testing as Natural testing. With this definition, I can call myself a “natural tester” and not necessarily a “scripted” or “exploratory” tester because in real life I am neither. This is the position most of the testers I interact with, take. They are neither pure exploratory nor pure scripted testers. We start our testing whenever our organizational goals permit us to which can be at the requirements stage or late in the cycle when development is about to be completed. We write test cases which help us understand the system and in the process of writing test cases we uncover bugs, raise uncomfortable questions and help improve the product. When we execute tests, we use our pre-written tests, come up with more test and scenarios to explore the product; we document some of these discovered tests and discard the rest. We mix testing using test cases (scripted testing) with guided ad-hoc (exploratory) testing. This is “natural testing” to

WhatIsTesting.com

55

Is scripted testing bad?

pre-designed test cases.

In exploratory testing test design, execution and learning happen at the same time whereas scripted testing uses pre-designed test cases. us and we fall in neither the exploratory nor the scripted camp.

Painting scripted testing as a bad practice Some of the arguments against scripted testing include: ● Scripted testing does not promote thinking about the problem in different ways. Scripted test cases mechanize the act of testing, taking intelligence away from testing. Statements like above make people believe that scripted testing promotes stilted thinking. If you define scripted testing as “following the pre-written test cases VERBATIM” then the statement would make sense. If this is how scripted testing was to be defined, I would not like to be found siding with scripted testing. Of course, there is no need to define scripted testing in this manner because that is not how most testers test. Unless there is a client demand, most testers do not follow the same test script again and again. They use test script only as a guide and not the gospel that has to be followed verbatim. Testers vary their data, their actions, and the sequence of actions. ● Scripted testing wastes time. Time is wasted in writing test cases whereas it could be spent testing. Also requirements change often, resulting in time wastage when test cases have to be reworked. The difference between scripted and exploratory testing is said to be that in exploratory testing test design, execution and learning happen at the same time whereas scripted testing uses

56

There are numerous advantages of “scripted” testing over “exploratory” testing, some of them being: • The act of writing test cases promotes logical and more complete thinking. • Written test cases can be reviewed. This is not a small advantage. The review is likely to result in better test cases and consequently better testing. • When test cases are written along with the requirements being written, many requirement bugs can be found and removed even before any code is written. • Test cases can be shared with developers resulting in removal of bugs even before they can be “created”. • Written test cases can be used as an aid for training new testers. • Even if requirements change, the impact of requirement change as related to testing can be addressed in a better way with written test cases. In my experience of testing, many complex products including some which are used by millions of users, review of written test cases has resulted in better understanding of testing and consequently better execution of testing by all testers involved in the review and not only the person who reviewed the test cases. Of course, there are situations where one does not have the time to write test cases. But enough people write about these contexts. My purpose to write this article was to defend what most people today would not defend. I do not favor exploratory testing over scripted testing or vice-versa. I favor using whichever style gives the most benefit in a given situation, rather than clinging to one end or the other of the continuum.

WhatIsTesting.com

Putting the “Test” back in Test Plan

Putting the

“Test” back in Test Plan

U

ntil fairly recently, many people who called themselves Software Testers came by their jobs by accident rather than by design. This placed many of us in the difficult position of having to learn how to do our job while we were trying to do that job. One of my biggest challenges was trying to figure out what goes into a Test Plan. In the first company that I worked for, the answer was straightforward enough. A Test Plan was simply a commonly-grouped set of test cases that was executed by a Tester. For example, there was a Print Test Plan, a GUI Test Plan, a Math Test Plan, and so on. No problem. Doubt in that definition began to creep into my mind when I started to network with other Testers in town and found out that Test Plans meant different things to different people. When I got my first Team Lead role in the next company I worked for, I discovered that the Test Plan I was being asked to produce was something completely different. It was supposed to be a planning document of some kind.

58

What to do when Confusion sets in I suddenly found myself in the uncomfortable position of not really knowing something that I thought I already understood. What do I do now? Where do I start? I began by asking people I worked with. I networked a bit and asked other testers for advice, but most of the time no one would say much because they thought it was confidential to share information like that. I even went to a day-long workshop on “Managing Systems Testing Projects”. While I found the workshop to be very insightful, I didn’t believe everything the instructor had said because a lot of it went against what I had learned so far on my own. For example, a Test Plan was not supposed to have any test cases in it. This just increased my doubt and anxiety. Back at my desk, faced with an empty Microsoft Word document in front of me, I searched the Internet for Test Plan templates and examples. I needed more opinions and references than

WhatIsTesting.com

by Paul Carvalho

Acknowledging that prior experience and knowledge based upon someone else’s incomplete knowledge is the biggest hurdle just my word against my workshop notes. It was at that time that I stumbled upon a few mailing lists focussed on Software Testing and Quality Assurance. People on those lists were very friendly and helpful and in no time I had several examples to help me develop my first test plans. The biggest hurdle I had to overcome was acknowledging that my prior experience and knowledge were based upon someone else’s incomplete knowledge. Until that moment, I thought that I knew what a test plan was. Then how was it that the definition changed when I changed companies? I had to unlearn the ways of the Dark Side.

Test Plan 1.0 Many of the examples and templates that I had were very similar to each other with some minor variations between them. In general, I found that a Test Plan contained elements like the following: ● Overview and Objectives ● Testing Scope and Strategy (Features To Be Tested, Not Tested, etc.)

I became quite proficient at creating these Test Plan documents and customising them for the needs for each project. I even became familiar with some industry standards for Test Documentation1. Then I started to notice an interesting pattern emerge with the more projects I worked on. For one thing, no one from any other department ever read any of these documents, even when they asked me to produce one. For another thing, no matter how I modified this document, I never really found it useful in the day-today management of the test projects I worked on. Finally, creating and maintaining the test documentation always seemed to take a significant amount of time and effort away from the actual testing activity itself. Naturally I tried to find ways to improve the efficiency of creating and maintaining these documents, but after a while I just started to question the usefulness of the document entirely. One day I realised that I had fallen into the trap of forgetting to question why I even had to create one. This time when I asked the question the response disturbed me: “Because it is expected that you create this documentation.” Wait a minute! Hold the phone! Who expects me to create this documentation? Why do I have to create this documentation? Where’s the value in that? I’m the one doing the job here and I’m seeing that I’m wasting my time creating and maintaining documents and procedures that don’t actually help me do my job any better. Where’s the sense in that?

● Approach, Entry and Exit Criteria, etc. ● Responsibilities and Staff ● Schedule and Deliverables (Test Reports, etc.) ● Resource Requirements (Test Environment, Tools, etc.) ● Risks and Contingencies

WhatIsTesting.com

A Brief Glimpse into Project Management By sheer coincidence, it was at that moment that I attended a Project Management workshop. I had requested it because I was struggling to manage several different test groups and projects simultaneously. An interesting thing happened after attending that workshop: I realised that I

59

Putting the “Test” back in Test Plan

Development is no different

Nature abhors a vacuum, and Software Development is no different never wanted to create a Test Plan again.

When you stop creating Test Plans, you run into all kinds of resistance, especially from Management. There are some good reasons to create extra documentation and there are many bad reasons. To be clear, “Because it’s expected” is not a good reason.

During the course I learned about the Project Management Institute2, we covered the Project Management Body of Knowledge (PMBOK), and worked through several examples in order to understand how to apply the correct knowledge, tools and techniques in order to successfully manage a project to completion. I highly recommend such a course for any Test Manager. One of the most significant aspects of the course for me was learning about the elements of a Project Work Plan:

One valid reason for producing some documentation is to help you learn from what you did on a test project. Being faced with all of the missed bugs now being reported by Customers through the Support department after a product ships is a harsh wake-up call. If you don’t have a record of the who, what, where, when, how, or why you did or didn’t test certain things, it is more difficult for you to learn how to improve and reduce the number of escaped bugs in the next project.

● The Project Team

It is important to note that a Project Plan doesn’t completely replace a Test Plan though. While the project plan may outline all the players, the general work breakdown structure and so on, there is at least one thing not captured to any useful detail by a Project Manager. One of the single most important things that a Test Team needs to own on any project is the Test Strategy.

● Project Charter ● Scope Management Plan ● Risk Management Plan ● Communications/Reportin Plan ● Work Breakdown Structure ● Master Budget ● Task Responsibility ● Critical Path Schedule Wow. That was it! I finally understood what it was that a Test Plan must have originally tried to accomplish. At some time, before Project Management became a formally recognised role in a Software Development organisation, it must have been commonplace for each of the teams to be required to produce their own little mini project work plans. Since that need doesn’t exist anymore, it was time to get with the program and stop wasting my time creating test documentation that clearly no one needed anymore. Nature

60

abhors

a

vacuum,

and

Software

It took me a few years to become reasonably good at developing and implementing an effective Test Strategy. I think it is a key skill for any Tester or Test Manager. My learning path involved “going back to square one” so that I could really know what I needed to be effective. Nowadays there are some good references and workshops that can help jumpstart you to reasonable success.

Test Plan 2.0 – A Fresh Start In the end, it comes down to this: you will never be able to test everything; you will never find all the bugs; and you will never have as much time, people, or resources as you really want. Starting with that in mind, your Test Strategy needs to help you work with what you have, in the time you have, to do the best job you possibly can.

WhatIsTesting.com

by Paul Carvalho

Why do certain test practices or approaches seem to be successful sometimes but not always? For me, the best way to accomplish this kind of “plan” was to use a Risk-Based Testing approach3. A few authors in particular had the most influence on me during my on-thejob education: James Bach4, Cem Kaner5, and Ross Collard. It was at that time that I also came to learn about the Context-Driven School of Software Testing.6 Context-Driven Testing was an interesting paradigm shift that helped me to understand the last piece of the test management puzzle: why do certain test practices or approaches seem to be successful sometimes but not always? Because that’s the way it is. Success depends on a great number of factors that change with almost every project. To be a successful Test Manager requires that you are a keen observer of these influencing factors and can adapt your team’s dynamics and approaches to suit the needs of your current situation. Darwin had it right when he said that things in nature adapt or die. We should too. Armed with this new knowledge, I am now quite comfortable developing Test Strategies tailored to suit the needs of each project that I work on. I take no offence when I am asked to produce a “Test Plan”. Instead of providing a document that someone else thinks I need, I create something that I find useful – a usable, working Test Strategy. Instead of worrying about templates, cover pages, and document control systems, I worry about Quality Criteria, Technical Risks, and Test Techniques. A typical Test Strategy document of mine is about one or two pages in length, and I don’t worry about formatting or templates anymore. It sits on a network drive that is accessible to the whole development team, and we update

our strategy as the risks and priorities change throughout the development phase. This is Agile Development. Things change and we need to adapt our testing strategy along with the new information as it arises, whenever it arises. The next time you are asked to provide a “Test Plan”, remember to focus more on the “Test” than the “Plan”. If you find you are putting too much effort on the latter, you should take a serious look at the value that the documentation effort is providing your organisation. Time is money, and the last time I checked, I’m getting paid to test.

References 1. An example of an Industry Standard for Test Documentation is IEEE Standard 829 - see http://standards.ieee.org/ 2. To learn more about the Project Management Institute, you can find them on the web at http://www.pmi.org/ 3. For an excellent article to get you started on Risk-Based Testing, read James Bach’s article “Heuristic Risk-Based Testing”, available on his web site at http://www. satisfice.com/articles/hrbt.pdf 4. James Bach: to find out more about his work and for excellent resources on Test Methodology, explore his web site at http://www.satisfice.com/ 5. Cem Kaner: author, professor, and advocate of great software testing practices, he maintains several useful web sites. His personal site is http://www.kaner. com/ but most of his work and Testing educational materials can be found for free at http://www.testingeducation.org/ 6. To find out more about Context-Driven School of Software Testing, check out http://www.context-driven-testing.com/ or pick up the book “Lessons Learned in Software Testing, A Context-Driven Approach” by Kaner, Bach and Pettichord.

WhatIsTesting.com

61

Accelrys has over twenty years of innovation and technology leadership in delivering solutions that transform discovery and development research. The company provides market-leading modeling, simulation and informatics software, data pipelining and platform technology, and solutions consulting and support services. A unique combination of expertise in both life and materials science lets Accelrys serve a diverse range of clients, including some of

San Diego, CA USA

the world’s leading pharmaceutical, biotechnology, chemical, nanotechnology, government and academic research organizations.

Cambridge, UK Bangalore, India

Accelrys is a global company. Our office in Bangalore, India undertakes engineering and quality control tasks.

Burlington, MA USA Tokyo, Japan Paris, France

For more information, visit our website:

EXPRESS YOURSELF “WhatIsTesting” magazine is written for software and application development mangers, project managers, team leaders and Test & QA managers. The magazine is not written for the entry level tester running test scripts all day, but rather has a larger more strategic view of the entire application life cycle. The reader is a decision maker within an IT department. Articles in the magazine should provide useful information to help him/her understand trends and emerging technologies, come to grip with new and timeless challenges, adopt best practices and ultimately make better decisions to improve software quality.

Presentation Articles should be in the range of 1500 to 4000 words, and be written for practitioners in the field; with a style that is down to earth, practical, interesting and original. Avoid jargon, speculative theory and abstract concepts; instead, focus on practical information that can help our readers today or in the near term. The purpose of an article is to transfer useful knowledge to the magazine’s readers, rather than to promote the author’s accomplishments or the products, services and technologies offered by the author’s employer. Don’t start the article with long fancy anecdote, a “first-the-earth-cooled” historical timeline or an academic-style abstract that says, “This article will do such and such”. Just get right into the subject matter. Also, don’t include footnotes or references. Refer to other sources, such as book, magazine articles or Web sites, within the main body of the article. Be sure to define terms and acronyms that may not be familiar to the reader; incase of doubt, define. Include, annotated source code, only when it helps explain the concepts in the article. Also, add copious figure, tables, listings, screen captures and other graphics to make the article easier to understand. Please avoid bulleted points within our prose; it is hard to read such lists. If you feel that a list can best be expressed using bullets, consider pulling it into a separate tale or textbox. Do not assume that the reader has specialized experience or knowledge about particular methodologies, platforms or tools. If the article is written about a specific methodology, platform or tool, present that dependency in the initial proposal. In the manuscript, include a short introduction or overview of prerequisite background concepts, if appropriate.

Article Reviews All proposals and manuscript submissions should be sent to the magazine by the article’s author. If there are multiple authors, one author will be designated as the lead author. The editors may send article, proposals and finalized manuscripts for peer review. In some cases, the editors might ask perform a peer technical review of the manuscript. Once the necessary review has been completed’ the article will be sent back to the article’s author for final approval.

Welcome to TEST 2008 Test2008 is the first conference being organized by PureConferences in India. Our conference will provide a platform for international and national test professionals to interact and participate. Speakers from around 10 countries, such as USA, UK, France, Sweden, Canada, Italy, Netherlands, including India will deliverkeynotes, tutorials, and papers during the conference. We intend to involve academia and institutions of learning with our conference. We also intend to make Test2008 a ‘green’ conference as far as possible. Additionally, we will institute two test scholarships for educating promising students unable to finance their studies. Animated discussions among testers frequently threw up the requirement of agility in testing and the challenges associated with it. Creating a balance between conflicting and changing demands of quality by customers and timely releases by organizations is a real challenge. Agility in Testing is the theme of Test2008--Test Excellence through Speed and Technology. Agility represents nimbleness, resourcefulness, and adaptability. In the world of testing,

agility is synonymous with the testing team’s resourcefulness and ability to respond quickly to changing contexts. Organizations need to catch the ‘window of opportunity’ when releasing a product to remain profitable. At the same time, we, as customers only want to use ‘quality’ products. We are hoping to have intense discussions around the theme to make it a learning experience for all participants. The theme is broad enough and will cover diverse topics. Keynotes, Tutorials, and Paper presentations by renowned speakers known internationally and in our country will be the highlight of the conference. Please mark theTest2008 conference program in your calendar. Our conference will offer partners theopportunities to associate themselves with this endeavor and showcase their products and participate in the conference. If you want to present a paper or tutorial, please fill in theSpeaker Submission Form. To attend the conference, please Register.

We welcome all delegates to our conference.