Golden Rule

Golden Rules

My Golden Rules are not an exhaustive list and I’m sure all readers could provide others from their own experiences.

for Quality Testing Golden Rule Number 1 Testing is a Risk Management Exercise. One of the first principles I learned as a tester – if there is no risk nobody will be worried. If there’s no worry, particularly from senior management or the Business, why expend valuable resources searching for defects that are not important or viable to find. Ensure all the Requirements are riskassessed to the best method that works for you or your organisation. Prioritise all the testing to target the most risk with the least test effort. When there are only minor risks outstanding, an informed decision can be made on when to stop testing.

Golden Rule Number 2 Training in Testing Reduces LongTerm Costs It is not un-common for less than 1% of an organisation’s training budget to be dedicated to training in testing, although with the introduction of ISEB Foundation Certificates this percentage is on the increase. Formal training in Test Management, Test Design, Static and Dynamic techniques, to name but a few of the many testing activities, will put systematic and structured method into the preparation, design and execution of tests.

22

Testing is famous for not quite going to plan, starting too late and being squeezed by fixed end dates.

Here is my top 20 (not in order of importance) that may help you navigate your Test Strategy.

If testing is an average 40% of project budget, why is training investment so low?

Golden Rule Number 3 The Testing Culture in an Organisation can Only Mature at the rate of its Capability If you wish to improve the effectiveness of the testing, don’t try too much too soon. The scale of what can be achieved will vary from one organisation to another and will be dependent on many variables. The first objective must be to establish all the perceived problems. Usually integration and de-duplication within and across different areas of responsibility will create a balanced economy of scale. I have worked on a number of process improvement programmes, where the benefits have been measured in the multi-million pound mark for testing improvements alone.

Golden Rule Number 4 The Greater the Number of Changes to Requirements, the More Rework is Required. How many times have you been in a situation where the requirements are not just changing but new ones are being created, even when you’re in test execution? Try and influence business and senior managers to work to the “Bus to Bus Stop” rule.

Journal of Software Testing Professionals

by Andy Redwood All the requirements get on the bus and the doors close. Nothing gets on and nothing gets off as the bus travel from A to B (ringfenced scope). Requirements may change seats on the bus (change control). If any new requirements are identified, they get on the next bus coming along behind (good release management). If this rule is broken (and it sometimes is for very good reason), there will be rework associated with test preparation. The associated cost, which can impact the original Business Case, increases the later new requirements are added, or when major changes in scope occur.

Golden Rule Number 5 Testing Estimates Should be Based on Business Risk. It’s very tempting to give your best guess when estimating the testing effort. If you’re lucky, or you’ve been collecting testing measures and metrics, you may have previous project information to guide you. If you adopt Golden Rule Number 1, you should be able to weight the risk with a “time taken to test.” This is more difficult on the first pass but each proceeding project will allow you to predict with ever increasing accuracy. A word of warning, this does not mean that the same tasks always take the same

http://www.testinginstitute.com

September 2002

time. You will need to evaluate any new “hotspots.” However it is quite common to hear Manager: “How long does it take to test?” Acceptance Testers: “We think about six weeks”. Manager: “How do you know?” Acceptance Tester: “Because it does”.

Please don’t estimate by habit! Golden Rule Number 6

As part of the test design the test analyst should continually scan back and forth looking for requirements that have no conditions and of equal importance, look for test conditions that do not map to a requirement. In this situation you may for example come across missing business processes or have an item that requires a business decision.

Workshops firm up known testing requirements very early in the life cycle. They also allow unknown activities to surface early, creating contingency within the programme. My measures show that on average, of the last 100 test strategy workshops I have facilitated an average thirty-six previously unknown issues have been raised. Some of these would not otherwise have been spotted until system test execution.

Tests should be designed to challenge high-risk attributes at the earliest opportunity.

Golden Rule Number 9 Always Test the Test Environment.

Drafting a Test Strategy from a Facilitated Workshop is a Quick Win.

This can be a difficult mindset for some people. There is a conflict with reporting a high number of errors and making good progress to plan. However, a high number of found errors in test must remain an important test objective, in order to ensure a high quality and robust system in Production.

“The log-ons were all wrong.” “The interface link was down”. “We were pointing at the wrong files.” Just a few of the remarks you may experience on day one of the testing. These are usually due to a lack of “pipe-cleaning” of the test environment prior to starting the real testing. Most of these tests (as with most testing activity) can be check-listed.

Golden Rule Number 10 Tests Should be Executed to Prove that the Item under Test Doesn’t Work, Rather than that it does Work.

Golden Rule Number 11 It is Better to Test the Errors out in Early Project Phases as Opposed to Having a Failure in Production. It has been suggested that 50% of all errors found in system testing originate before coding. It can cost hundreds of times more to fix a problem in production than in the requirements phase. A focus on static testing techniques would be advantages, for example, walkthroughs, reviews, workshops and formal inspections will raise early incidents.

Golden Rule Number 7 Linking Development Documents to Test Documents Improves Understanding.. Figure 1 is an old picture (thanks to Paul Smith), but it demonstrates the relationship between development and testing documents. It is the documentation that we as testers must test back to. In many cases this relationship is implied and not explicit.

Golden Rule Number 8 All Test Conditions Should have a Parent Requirement. Traceability between requirements and test conditions within a test repository (or inventory) will ensure all requirements are Figure 1. tested and aid test progress reporting.

September 2002

http://www.testinginstitute.com

Journal of Software Testing Professionals

23

Unfortunately in some organisations it is still radical for testing to be perceived as an across the life-cycle group of activities. In general, the more senior the management level, the harder it gets to persuade people that testing can begin as soon as the requirements are defined.

embark on a path of continuous process improvement with reduced costs.

Golden Rule Number 14 All Incidents Should be Traceable to a Test Condition and a Requirement.

Golden Rule Number 12

This is fundamental if you are to know that:

Each Test Phase Should Attempt to Test Something Different with as Little Duplication as Possible.

(a) you have mitigated all the risks (b) you know you can stop testing.

Duplication costs. Some duplication is unavoidable.

This activity has been made easier over the years by test management tools.

Testing is more efficient and effective if the testing undertaken is defined for each test phase, with minimal duplication, within the test strategy. Details for each test phase will appear in phase specific test plan documents, each linked to corresponding development deliverables.

It is also possible to evaluate data on types of incidents raised against requirements (or test conditions).

I think it needs to be questioned if a particular test phase is actually required. The basis for this decision could be risk and viability. In my travels in and out of organisations I see test phases executed only because they appear in the life cycle or the programme management method as an activity. The objective for undertaking the phase, in some cases, has already been proven in previous phases.

Where you have raised few incidents against certain requirements or conditions, you may conclude that these are likely to be stable parts of the system. In my experience areas of least testing, point at the high-risk areas in the production system to monitor in the first few weeks post implementation.

Test Assets and Project Histories Must be Considered for Re-use.

An Incident is not a Error or a Fault, but is Anything that was not Expected. Incidents can occur in any programme phase. Incidents need not be testing specific. An incident that results from human error or a component fault will be logged as a problem.

• reduce analysis, design and build costs • allow better understanding of the existing functionality • allow more accurate project estimates • enable earlier test preparation

It is important to evaluate all incidents, categorise them under strategically defined incident types and allow sufficient time to conduct root cause analysis. In this way you will gather meaningful measures, establish trends, calculate metrics and

Testware is worth an average 40% of every project budget; why not re-use this on iterative projects to reduce the test preparation time?

24

• provide the status of changes since the previous release • can be constructed at all test levels to enable maximum flexibility for testing large or small projects • form a baseline from which to commence preparation of new tests.

Golden Rule Number 16 If you can’t Measure, you will not Know if what you Achieve is Good or Bad. Measures lead to metrics lead to trends. Metrics form management information and allow informed decisions. Measures make perceptions factual.

Golden Rule Number 17 Measures Point to Root Causes. Fix the Root Cause and You Improve the Quality. Fixing the root cause in processes and procedures so that they never recur, will reduce costs and improve quality of delivery.

Golden Rule Number 15

Project histories give factual information and documentation of the previous release. Having access to all documentation sets, test logs, measures and metrics from the previous project will:

Golden Rule Number 13

Test packs run as regression tests:

Journal of Software Testing Professionals

Care is required, as it is easy to mistake analysis for the root cause for a witchhunt. The root cause could be anywhere and is likely not to be in the phase the incident was identified. The important thing is to fix it and learn from it, not apportion any blame.

Golden Rule Number 18 Tools Can Only Automate Existing Process. Tools Require Support. Existing process needs to be in a state that allows it to be automated. This can be done as the programme evolves, but in my experience automation should run as a parallel project to the main testing activity on a project.

http://www.testinginstitute.com

September 2002

Build the tests, run them manually and then follow up behind with a small team who will automate the testing for the future. Less risk, more control. My old friend Terry Stevens use to have a catch phrase, “the cost of the tool is not the cost of the tool.” This is still true. Tools are licensed software and require some expertise to obtain full benefit. Those of you thinking of embarking on automation projects can now make use of the many user groups, limited (sometimes free) on-site vendor support or help from the many consultancies with experience in this field.

Golden Rule Number 19 If You Don’t Ask a Third Party Supplier the Question, You May Never be Told the Answer. I’ll try not to get on a soapbox about this even though it is one of the most frustrating things that a Test Manager may have to deal with. Here are some of the most common questions you might wish to ask. I’d be happy to supply a more detailed set (email [email protected]). • How did you test this package before you handed it over to us?

Golden Rule Number 20 Sign off Criteria Should Never be a Surprise to Management When criteria are agreed in advance via the testing strategy or lower level test plans and reviewed at set points in the life cycle, sign off into production is the norm, not the exception. Nothing escapes into production with high-risk caveats or with insufficient testing. Build yourself a list of all interfacing managers that need to always be informed as to the status and progress of testing across all projects. Recommend via the staged update reports if sign off is ‘ok to go’ or what the deviation is from the standard approach detailed in the test strategy. Managers will be able to make proactive decisions as the project progresses rather than have to wait until it has finished to react.

About the Author Andy Redwood is a Principal Consultant with Cresta Group, working with clients to help them improve the quality, efficiency and effectiveness of their testing resources. Andy is a frequent public speaker at European Conferences and is an active member within the British Computer Society and ISEB (Information Systems Examinations Board).

• Do you have any test documentation I could look at? • What errors do you know to be outstanding that are in this release? • Does this release include changes for any other client and if so, how does it affect us? • What’s the turnaround time for fixing our priority 1 problems?

September 2002

http://www.testinginstitute.com

CALL FOR SUBMISSIONS JSTP is looking for articles reflecting real life experiences with any of the following areas: • Testing (test process, test management, testing techniques, performance test ing, security testing, test automation, etc.) • Requirements • Bug tracking and reporting • Incident management • Configuration management • Release management • Risk assessment • Test process measurement and improvement All submissions for publication in JSTP must adhere to the following guidelines: 1. Author’s Bio. A 25 word or less author’s biography shall accompany each submission. 2. All graphics must be in high resolution format and submitted in a separate graphic file such as a TIF. 3. Original Content. Unless otherwise agreed, all article submissions shall contain original content not previously published. Exceptions may apply only if written permission is granted by the holder of any previous copyright. 4. Grant of Copyright. Unless otherwise agreed, the copyright for articles submitted shall be granted to JSTP for publication and reprints. 5. Format. Unless otherwise agreed, submissions shall be provided in MS Wordtm format with page numbers (x of y) in the footer. 6. Size. The length shall not exceed 15 pages of MS Wordtm single-spaced, including all charts and graphical illustrations. 7. Multiple authors. Submission of multiauthor papers is taken as an indication that all authors agree to its publication. 8. Style. The text of the paper must be in clear, concise and grammatically correct English. Abbreviations and acronyms must be clearly defined. 9. Attribution. Any quotes or excerpts from any other person or publication shall be duly attributed either in the text or in a footnote. 10. Introduction. The first paragraph(s) of the article should provide a concise summary of the article’s topic, including the benefit(s) to the reader. 11. Sub-Headings. Where appropriate, subheadings should be added to indicated main ideas and to assist the reader in locating specific information. 12. Summary. The final paragraph(s) of the article should provide a concise summary of the article. 13. References. References in the text should include the authors’ names together with the year. 14. Bibliography. In the bibliography, list references in alphabetical order with the titles of journals cited in full.

Journal of Software Testing Professionals

25