To Be Certified Or Not

TO BE (Certified) OR NOT TO BE, That is the Question By Brian Hambling

S

oftware testing is at last emerging as a significant discipline within I.T. Always the poor relation to analysis and design, the processes of verification and validation are now seen to be fundamental to the development process as a whole. They should be applied from the beginning of the development life cycle, and be a core element of risk management for I.T. projects. Emerging technologies and business applications of I.T. are breaking new ground. E-commerce, for example, places I.T. at the heart of the business process and failure of computer systems becomes a major business risk. This is equally if less dramatically true of all the other forms of enterprise application integration. Reliable computer systems that add real value to the business are now essential to success, and failure of any kind is becoming increasingly unacceptable. Software testing is the key to managing these business risks. This growing sense of the importance of software testing has its disadvantages, however. A growing interest in testing will attract new blood into the discipline, but how do we ensure that these new recruits are competent and professional about their work? Most software testers have learned their skills through experience with a limited amount of ad hoc training and there have been few standards to guide them; university courses on software quality assurance and testing are still relatively March 2000

uncommon; testing practice has had to evolve to keep pace with software development practice and the literature struggles to keep pace with emerging testing techniques and methods. What software testing clearly needs is a properly defined discipline supported by professional standards and a process of ensuring that testers are aware of the professional standards expected of them.

The Need for Certification How do we develop and grow good testing professionals? In the past, few have sought software testing as a career; they may have come to it as a second choice, evolved into it from a career in software development, or been pushed into it by a short term crisis. Most software testing professionals learned the hard way, and it has typically taken somewhere between 5 and 10 years to build a sound testing career. This process of organic development will no longer work, because:

♦ Demand is too high for good testers to trickle through in sufficient numbers; ♦ The testing challenge is growing too rapidly for testers to learn all that they need to know by experience alone; ♦

Testers need a professional culture Journal of Software Testing Professionals

that will not happen by chance, because testers are spread thinly across a vast territory;

♦ Testers need a development path that is not dependent solely on the good will of any one employer. Development of software testing professionals must be managed so that opportunities are created early in a new professional’s career and so that a coherent and continuous development path exists even for those who cannot or who do not wish to remain with a single employer. Thus there is a need for some kind of development mechanism that is at least partially independent of employers, easily and universally available, broad in technical scope, and underwritten by other software testing professionals through a professional body.

Certification Schemes for Software Testing Professionals Knowledge-Based Aproaches to Certification Knowledge-based certification is based on a required level of knowledge rather than on a specific program of education. Certificates are awarded to candidates who can show that they have acquired http://www.softdim.com/iist

23

the necessary level of knowledge in a number of specified areas. The Quality Assurance Institute (QAI) QAI has two important certification programs in the area of software quality assurance – Certified Software Test Engineer (CSTE) and Certified Quality Analyst (CQA). The CQA qualification will not be considered in this article, though it may be of relevance to those wishing to broaden a career in testing to encompass the wider quality arena. Certified Software Test Engineer (CSTE) In QAI’s words: “The CSTE is expected not only to possess the skills required to pass the CSTE examination, but also to be a change agent……You cannot rely on your current knowledge to meet tomorrow’s job demands.” The knowledge expected of a test professional is detailed in a Common Body of Knowledge (CBOK), consisting of 5 categories, encompassing 16 knowledge domains, which are further subdivided into key topical areas. The categories and domains are:

♦ General skills: Communications, Professional Development, Quality Principles and Concepts, Processes for Software Development, Operation and Maintenance.

dates should keep themselves updated in line with its own stated aims: “The exam expects that familiarity with current industry topics by way of experience, periodicals and textbooks is continually being maintained…. The examination instrument is continually reviewed and revised to implement these factors into the testing process.” Given that the CSTE examination is a relatively new enhancement to the program, there is as yet little evidence of this regular updating of the examination; previously candidates were required only to submit evidence of work experience. The move to an examination is a welcome trend, indicating that QAI has recognized the need for some common baseline of knowledge that is tested. American Society for Quality (ASQ) ASQ’s certification program encompasses 7 different certificates, one of which is for software quality engineers – the Certified Software Quality Engineer (CSQE). The CSQE requirements include education and experience, proof of professionalism and a multiple-choice examination. Up to 8 years of relevant experience is required, 3 of which must be in a decision-making position - the actual level of experience required is determined by the candidate’s level of education. Proof of professionalism requires membership of a professional body or the endorsement of at least 2 members of such a body. The Body of Knowledge for the CSQE

♦ Test Skills/Approaches: Testing examination is made up of the following Principles and Concepts, Verification and Validation Methods, Test Management, Environment and Standards.

♦ Test Planning: Risk Analysis, Test

categories (the breakdown for the relevant testing category only is shown to indicate the depth of coverage within the certificate as a whole) :

Tactics, Planning Process.

♦ Executing the Test Plan: Test De- ♦ General Knowledge, Conduct and sign, Performing Tests, Defect Tracking and Management.

♦ Test Result Analysis, Reporting and Improvement: Quantitative Measurement, Test Reporting, Improving the Testing. QAI believes that certification candi24

http://www.softdim.com/iist

Ethics ♦ Software Quality Management ♦ Software Processes ♦ Software Project Management ♦ Software Metrics, Measurement and Analytical Methods ♦ Software Inspection, Testing, Verification and Validation: Inspection, Journal of Software Testing Professionals

Testing, Verification and Validation. ♦ Software Audits ♦ Software Configuration Management Since software testing is mentioned in only one of the eight categories listed, the CSQE is not an appropriate certificate for software testing professionals, at least not while they wish to focus their career solely or predominantly on testing. These basic requirements are supplemented by a list of minimum expectations for Certified Software Quality Engineers. Key Characteristics of the KnowledgeBased Approach to Certification The CBOK requirement for CSTE is well defined and comprehensive, requiring a good understanding of the theory and practice of testing. The basis of the certificate is the achievement of a level of knowledge sufficient to pass the CSTE examination, which QAI claims will be continually updated. After certification, successful candidates must provide evidence of continuing professional education to retain their certificate. There is, however, no syllabus for the examination. As a result, the extent of study needed from the required reading is unclear. The resources available to the CSTE Board may be such that it is more aware of new technologies and advanced testing practices than a potential candidate. This makes benchmarking difficult, and it would, therefore, not be straightforward for a potential employer to assess the value of the qualification, unless they were already familiar with the CSTE. There is no indication of the level at which a CSTE can be expected to operate, which again makes assessment of the value of the certificate difficult. There is no link between the CSTE and experience to enable an employer to use the CSTE as a stepping-stone in a career development program. March 2000

The QAI CQA and the ASQ CSQE schemes are aimed at the quality professional rather than the testing specialist. The CSQE requirements for experience and proof of professionalism clearly position this certification program at the more senior end of the spectrum. Those new to software testing would be excluded from such a program, but those wishing to extend their experience and expertise later in their career might find this a suitable certificate to aim for.

formal instruction in the body of knowledge approved by the IIST advisory board. This formal training can be acquired by taking IIST courses, courses approved by IIST and running at PSQT conferences, or (for up to 2 days of the required training only) courses run by an alternative training provider and within the required curriculum.

The knowledge-based certification programs require applicants to acquire knowledge sufficient to pass an examination that is continuously updated to reflect changes in the discipline. The method of knowledge acquisition is not defined.

♦ Principles of Software Testing and Test Case Design Techniques: Levels of Testing, Code-Based Testing, Requirement-Based Testing, Test Design Specification.

Education-Based Certification Programs

♦ Testing Process: Defining the Test

An education-based certification is one based on the successful completion of one or more courses, each with an examination to test knowledge acquired. International Institute for Software Testing (IIST) The IIST has been created to focus attention on the problem of encouraging new software testers from other specialisms and to ensure a sound baseline of knowledge and experience. There is currently just a single certification program - the Certified Software Testing Professional (CSTP). Certified Software Testing Professional CSTP exists to encourage development of a broad range of testing professionals, including managers and auditors as well as testing practitioners. It promotes a broad, life cycle view of software testing. CSTP requires both formal training and job experience. The job experience requirement is a minimum of one year of relevant experience. The formal training requirement consists of ten days of March 2000

The required curriculum for certification as a CSTP is:

♦ Test Management: Planning, Scheduling, Resources, Risk Management. Process, Measuring the Testing Process, Improving the Testing Process, Testing Mainframe Applications, Testing Client/ Server Applications, Testing Internet and Web Applications, Testing ObjectOriented Applications, Testing Embedded Systems.

protection, systems analysis and design, information security and software testing. There is currently only one qualification in software testing - the Foundation Certificate, though a Practitioner Certificate and a Practitioner Diploma are planned. Demand for the Foundation Certificate courses has been high; at the time of writing over 600 people have so far achieved the Foundation Certificate. The Practitioner Certificate syllabus is in preparation for courses to begin early in 2001. No specific plans have yet been published for the Diploma. Courses are delivered by accredited course providers, of which there are currently seven, working from an approved syllabus. ISEB Foundation Certificate in Software Testing The Foundation Certificate course lasts for 3 days and covers:

♦ Principles of Testing: Terminology, Why Testing is Necessary, Fun-

♦ Test Execution: Test Scripting, damental Test Process, Psychology of Reporting, Environment, Defect Tracking.

♦ Test Automation: Tool Evaluation and Selection, Scripting.

Testing, Re-Testing and Regression Testing, Expected Results and Prioritization.

♦ Requirement Definition and Refine- ♦ Testing Throughout the Lifement: Writing Testable Requirements, Requirement Validation, Requirement Traceability.

♦ Static Testing (Inspections, Reviews, and Walkthroughs): Defining the Process, Defining Standards, Measuring the Process, Improving the Process, Defect Data Analysis. The Information Systems Examination Board (ISEB) of the British Computer Society (BCS)

Cycle: Models for Testing, Economics of Testing, High Level Test Planning, Acceptance Testing, Integration Testing in the Large, Functional and Non-Functional System Testing, Integration Testing in the Small, Component Testing, Maintenance Testing.

♦ Dynamic Testing Techniques: Black and White Box Techniques, Error Guessing.

♦ Static testing: Reviews and the Test Process, Types of Review, Static Analysis.

The ISEB provides industry-recognized qualifications that measure competence, ability and performance in many areas of I.T. Qualifications are available in areas such project management, telecommunications, service management, data

♦ Test Management: Organization

Journal of Software Testing Professionals

http://www.softdim.com/iist

and Configuration Management, Test Estimation, Monitoring and Control, Incident Management, Standards for Testing. 25

♦ Tool Support for Testing: Types of CAST Tool (Computer-Aided Software Testing), Tool Selection and Implementation. Key Characteristics of the EducationBased Approach to Certification The CSTP requires formal training through approved courses. Ten days of training in 1 or 2 day modules must be completed, each of which is separately examined by a short written examination; a score of 80% is necessary in each of the written examinations for award of the certificate. Courses are run mainly by the IIST, though some courses are prepared and taught by specialists not employed permanently by the IIST. Examinations are set by the course deliverers and moderated by the IIST. The experience requirement is sufficiently undemanding to allow those new to testing to qualify reasonably quickly and therefore encourages an early move to certification for those new to testing. The ISEB Foundation Certificate syllabus is published via a web site, where accredited trainers are also identified. Approved training courses are reviewed against the published syllabus and the one-hour examination, in multiple-choice format, is set and administered by the ISEB independently of the trainers. The education-based approach to certification uses approved courses and course providers to ensure that the required knowledge is imparted by appropriately experienced professionals and to an approved syllabus. One key advantage of this approach is that the fundamentals can be properly established, using trusted members of the profession to impart an agreed set of basic principles, before applicants move on to more advanced subjects. Both IIST and ISEB build on basic courses with more advanced training, providing a well defined route to senior practitioner status.

26

http://www.softdim.com/iist

Conclusion The software testing discipline has reached a level of maturity where certification is both necessary and desirable; necessary to underpin the required levels of professionalism in the industry, and desirable to provide software testing professionals with a qualification that reflects their knowledge, skill and experience. The number of certificates so far issued by the certification bodies indicates a strong demand, though it is likely that only a tiny

The challenge now is to encourage industry to recognize and support the certification programs

proportion of those practicing as software testers have yet been certified. Alternative certification approaches provide potential certificate holders with mechanisms to suit their individual needs and preferences. The knowledge-based approaches favor those who cannot find time or opportunity to attend training events and who are capable of sustained private study before taking the examination. The absence of a standard syllabus adds to the difficulty of preparing for the certificate examination, and the approach tends to favor the more experienced professional seeking recognition. The knowledge-based

Journal of Software Testing Professionals

approach also carries with it the risk that applicants will acquire knowledge in a way that inhibits further development. The education-based approach, on the other hand, provides for very broad introductory training with relatively small-scale examinations and tends to attract the beginner to testing and those who wish to consolidate on early experience. The CSTP courses cover a broad range of topics, some general and some specific to particular application types, while the ISEB Foundation Certificate is a single 3-day course with a 40-question multiple-choice examination. ISEB plans to introduce a more advanced course in early 2001, requiring approximately 8 days of training at a more advanced level for software testing practitioners. There is clearly scope for the CSTP and ISEB courses to be used together, for example ISEB Foundation as a component of the CSTP requirements, or CSTP as a precursor to the ISEB Practitioner course. The educationbased approach is well suited to corporate schemes, where testing professionals can acquire a common language and culture via a certification scheme applied across an organization. It also ensures that certificate holders have a genuinely common underlying language and discipline, and therefore propogates good testing practices in a consistent way. The rapid development of certification programs, coupled with initiatives such as Special Interest Groups and career progression models (such as the Industry Structure Model of the BCS) provides a flexible but sound underpinning for the software testing discipline. The challenge now is to encourage industry to recognize and support the certification programs to ensure their continued growth and influence in building a true software testing profession.

¾Brian Hambling can be reached at ImagoQA, 52-54 High Holborn, London WC1V 6RL, UK +44 207 421 8109 or [email protected].

March 2000