Cste Body Of Knowledge Course 2006

CSTE Body of Knowledge Knowledge Category 1

Software Testing Principles and Concepts The “basics” of software testing are represented by the vocabulary of testing, testing approaches, methods and techniques as well as the materials used by testers in performing their test activities. Specifically, this knowledge category will address: •

•

•

• • •

• •

Testing Techniques - Understand the various approaches used in testing, including static (e.g., desk checking), white-box (logic driven), black-box (requirements driven), load testing, coverage testing and regression testing. Also included are the methods for designing and conducting tests. Levels of Testing - Identify the levels of testing such as unit, performance, string, integration, systems recovery, acceptance, parallel, performance, and interface testing. Testing Different Types of Software - The changes in the approach to testing when testing different development approaches such as batch processing, client server, web based, object oriented systems and wireless systems. Independent Testing - Testing by individuals other than those involved in the development of the product or system. Vocabulary - The technical terms used to describe various testing techniques, tools, principles, concepts, and activities. The Multiple Roles of Software Testers - The test objectives that can be incorporated into the mission of software testers. This would include the testing to determine whether requirements are met, testing effectiveness and efficiency, testing user needs versus software specifications and testing software attributes such as maintainability, ease of use and reliability. Testers Workbench - An overview of the process that testers use in performing a specific test activity such as developing a test plan and preparing test data. The “V” Concept of Testing - The “V” concept relates the build components of the development phases to the test components that occur during the test phases.

CSTE Body of Knowledge Knowledge Category 2

Building the Test Environment The test environment is comprised of all the conditions, circumstances, and influences surrounding and affecting the testing of software. The environment includes the organization’s policies, procedures, culture, attitudes, rewards, test processes, test tools, methods for developing and improving test processes, management’s support of software testing, as well as any test labs developed for the purpose of testing software and multiple operating environments.

This category also includes assuring the test environment fairly represents the production environment to enable realistic testing to occur. Specifically this knowledge category will address: Knowledge of Test Process Selection and Analysis 1. Concepts of Test Processes – the concepts of policies, standards and procedures and their integration into test process. 2. Test Process Selection – selecting test processes that lead to efficient and effective testing activities and products. 3. Acquisition or Development of a Test Bed/Test Lab/Test Processes – designing, developing, and acquiring a test environment that simulates “the real world,” including capability to create and maintain test data. 4. Test Quality Control – test quality control to assure that the test process has been performed correctly. 5. Analysis of the Test Process – the test process should be analyzed to ensure: a. The effectiveness and efficiency of test processes. b. The test objectives are applicable, reasonable, adequate, feasible, and affordable. c. The test program meets the test objectives. d. The correct test program is being applied to the project. e. The test methodology, including the processes, infrastructure, tools, methods, and planned work products and reviews, is adequate to ensure that the test program is conducted correctly. f. The test work products are adequate to meet the test objectives. g. Test progress, performance, processes, and process adherence are assessed to determine the adequacy of the test program. h. Adequate, not excessive, testing is performed. 6. Continuous Improvement – identifying and making improvements to the test process using formal process improvement processes. 7. Adapting the Test Environment to Different Software Development Methodologies – the test environment must be established to properly test the methodologies used to build software systems such as waterfall, web-based, object oriented, agile, etc. 8. Competency of the Software Testers – management must provide the training necessary to assure that their software testers are competent in the processes and tools included in the test environment. Test Tools 1. Tool Development and/or Acquisition – understand the processes for acquiring and using test tools, methods, and understand the skills needed for test development, execution, tracking, and analysis tools. (Both manual and automated tools including test management tools).

2. Tool Usage – understanding of how tools are used for: a. b. c. d.

automated regression testing tools defect management tools performance/load testing tools manual tools such as checklists, test scripts, and decision tables; traceability tools e. code coverage f. test case management tools g. common tools to aid in testing such as an excel spreadsheet. Management Support for Effective Software Testing 1. Management must create a “tone” that encourages software testers to do their work in an efficient and effective manner. This is accomplished through test policies, management support of those policies, open communication between management and testers, and enforcing compliance to policies and processes. 2. Test processes must align with organizational goals, user business objectives, release cycles and different developmental methodologies.

CSTE Body of Knowledge Knowledge Category 3

Managing the Test Project Software testing is a project with almost all the same attributes as a software development project. Software testing involves project planning, project staffing, scheduling and budgeting, communicating, assigning and monitoring work and ensuring that changes to the project plan are incorporated into the test plan. Specifically this knowledge category will address: Test Administration and Organizational Structure 1. Test planning, scheduling and budgeting. 2. Alignment – Assurance the test processes are aligned with organizational goals, user business objectives, release cycles and different development methodologies. 3. Test Performance – monitoring test performance for adherence to the plan, schedule and budget, reallocating resources as required, and averting undesirable trends. 4. Staffing – acquiring, training, and retaining a competent test staff. 5. Management of Staff – keeping staff appropriately informed, and effectively utilizing the test staff.

6. Organizational differences between traditional management utilizing a hierarchical structure versus quality management using a flattened organization structure. Personal and Organizational Effectiveness 1. Communication Skills a. Written Communication – providing written confirmation and explanation of a variance from expectations. Being able to describe on paper a sequence of events to reproduce the defect. The ability to analyze information, so that all pertinent information is recorded and communicated to the proper person. b. Oral Communication – understand how to communicate problems and/or defects in a non-offensive manner that will not incite ill feelings or defensiveness on the part of the developers. The ability to articulate a sequence of events in an organized and understandable manner. Includes effective participation in team activities. c. Listening Skills – actively listening to what is said; asking for clarification when needed, and providing feedback statements to acknowledge understanding; documenting conclusions. d. Interviewing Skills – developing and asking questions for the purpose of collecting data for analysis or evaluation; includes documenting conclusions. e. Analyzing Skills – determining how to use the information received. 2. Personal Effectiveness Skills a. Negotiation – working together with one or more parties to develop options that will satisfy all parties. b. Conflict Resolution – bringing a situation into focus and satisfactorily concluding a disagreement or difference of opinion between parties. c. Influence and Motivation – using techniques and methods in order to invoke a desired effect on another person. Influencing others to act in a certain goal-oriented activity. d. Judgment – applying beliefs, standards, guidelines, policies, procedures, and values to a decision. e. Facilitation – helping a group to achieve its goals by providing objective guidance. 3. Project Relationships – software testers need to develop an effective working relationship with project management, software customers and users, as well as other stakeholders having invested interest in the success of the software project. 4. Recognition – recognition is showing appreciation to individuals and teams for work accomplished. This also means publicly giving credit where due and promoting other’s credibility.

5. Motivation – encouraging individuals to do the right thing and do it effectively and efficiently. 6. Mentoring – working with testers to assure they master the needed skills. 7. Management and Quality Principles – understanding the principles needed to build a world class testing organization. Leadership 1. Meeting Chairing – organizing and conducting meetings to provide maximum productivity over the shortest time period. 2. Facilitation – helping the progress of an event or activity. Formal facilitation includes well-defined roles, an objective facilitator, a structured meeting, and decision-making by consensus, and defined goals to be achieved. 3. Team Building – aiding a group in defining a common goal and working together to improve team effectiveness.

CSTE Body of Knowledge Knowledge Category 4

Test Planning Testers need the skills to plan tests, including the selection of techniques and methods to be used to validate the product against its approved requirements and design. Test planning assesses the business and technical risks of the software application, and then develops a plan to determine if the software minimizes those risks. Test planners must understand the development methods and environment to effectively plan for testing, including regression testing. Specifically this knowledge category will address: Prerequisites to Test Planning 1. Risk Analysis and Risk Management a. Identifying Software Risks – knowledge of the most common risks associated with software development and the platform on which you are working. b. Identifying Testing Risks – knowledge of the most common risks associated with software testing for the platform you are working on, tools beings used, and test methods being applied. c. Identifying Premature Release Risk – understand how to determine the risk associated with releasing unsatisfactory, untested software products. d. Risk contributors – ability to identify contributors to risk

e. Identifying Business Risks – knowledge of the most common risks associated with the business using the software. f. Risk Methods – understanding of the strategies and approaches for identifying risks or problems associated with implementing and operating information technology, products, and processes; assessing their likelihood, and initiating strategies to test for those risks. 2. Managing Risks a. Risk Magnitude – ability to calculate and rank the severity of a risk quantitatively. b. Risk Reduction Methods – the strategies and approaches that can be used to minimize the magnitude of a risk. c. Contingency Planning – plans to reduce the magnitude of a known risk should the risk event occur. Test Planning Entrance Criteria 1. Pre-Planning Activities a. Success Criteria/Acceptance Criteria – the criteria, established by the business at the inception of a project, that must be validated through testing to provide user management with the information needed to make an acceptance decision. b. Test Objectives – understanding of the objectives to be accomplished through testing. c. Assumptions – establishing those conditions that must exist for testing to be comprehensive and on schedule; for example, software must be available for testing on a given date, hardware configurations available for testing must include XYZ, etc. d. Issues – identifying specific situations/products/processes which, unless mitigated, will impact forward progress. e. Constraints – limiting factors to success. f. Entrance Criteria/Exit Criteria – the criteria that must be met prior to moving to the next level of testing, or into production, and how to realistically enforce this or minimally how to reduce risk to testing organization when external pressure (from other organizations) causes you to move to the next level without meeting exit/entrance criteria. 2. Test Planning a. Test Scope – what is to be tested b. Test Plan – the deliverables to meet the test’s objectives; the activities to produce the test deliverables; and the schedule and resources to complete the activities.

c. Requirements/Traceability – defines the tests needed and relates those tests to the requirements to be validated. d. Estimating – determines the amount of resources and timeframes required to accomplish the planned activities. e. Scheduling – establishes milestones for completing the testing effort and their dependencies on meeting the rest of the schedule. f. Staffing – selecting the size and competency of staff needed to achieve the test plan objectives. g. Approach – methods, tools, coverage and techniques used to accomplish test objectives. h. Test Check Procedures (i.e., test quality control) – set of procedures based on the test plan and test design, incorporating test cases that ensure that tests are performed correctly and completely. i. Maximizing Test Effectiveness – methods to assure test resources will be used most effectively. 3. Maintaining the Most Current Test Plan a. Software Configuration Management (SCM) – SCM is the organization of the components of a software system, including documentation, so that they fit together in a working order. It includes change management and version control. b. Change Management – modifies and controls the test plan in relationship to actual progress and scope of the system development. c. Version (control) – the methods to control, monitor, and achieve change.

CSTE Body of Knowledge Knowledge Category 5

Executing the Test Plan The skills needed to execute tests, design test cases; use test tools; and monitor testing to ensure correctness and completeness. Specifically this knowledge category will address: Test Design and Test Data/Scripts Preparation 1. Specifications – assure test data scripts meet the objectives included in the test plan.

2. Cases – development of test cases, including techniques and approaches for validation of the product. Determination of the expected result for each test case. 3. Test Design – considerations including tests (including functional, negative, performance, load/stress); Test Design Strategies (e.g. small modular tests, scenario based tests); Test Design Attributes (repeatable, reusable level of detail: trade-offs in specificity vs. test case maintenance, how to organize; e.g. by feature, by test type, by application architectural area (client/server). 4. Scripts – development of the on-line steps to be performed in testing, focusing on the purpose and preparation of procedures; emphasizing entrance and exit criteria. 5. Data – development of test inputs, use of data generation tools. Determination of the data set or sub-sets needed to ensure a comprehensive test of the system. The ability to determine data that suits boundary value analysis and stress testing requirements. 6. Test Coverage – achieving of the coverage objectives in the test plan to specific system components. 7. Platforms – identify the minimum configuration and platforms on which the test must function. 8. Test Cycle Strategy a. Determination of the number of test cycles to be conducted during the test execution phase of testing. b. Determination of what type of testing will occur during each test cycle. Performing Tests 1. Execute Tests – perform the activities necessary to execute tests in accordance with the test plan and test design (including setting up tests, preparing data base(s), obtaining technical support, and scheduling resources). 2. Compare Actual versus Expected Results – determine if the actual results met expectations (note: comparisons may be automated). 3. Documenting Test Results – recording test results in a desired form. Information to be recorded must be defined. Results can include incidents not related to testing that can impact software quality, such as time required to process a business transaction or ease of use. 4. Use of Test Results – how the results of testing are to be used, and who has access to the test results.

5. Record Discrepancies – documenting defects as they happen including supporting evidence. Defect Tracking (Note: defect tracking begins by recording a variance from expectations; and will not be considered a true defect until the originator acknowledges the variance as an incorrect condition.) 1. Defect Recording – defect recording is used to describe and quantify deviations from requirements/expectations. 2. Defect Reporting – reports the status of defects; including severity and location. 3. Defect Tracking – monitoring defects from the time of recording until satisfactory resolution has been determined and implemented. Testing Software Changes 1. Static Testing – Evaluating changed code and associated documentation at the end of the change process to ensure correct implementation of the change. 2. Regression Testing – testing the whole product to ensure that unchanged functionality performs as it did prior to implementing a change. 3. Verification – Reviewing requirements, design, and associated documentation to ensure they are updated correctly as a result of a change.

CSTE

Body

of

Knowledge

Knowledge Category 6

Test Status, Analysis and Reporting The testers need to demonstrate the ability to develop testing status reports. These reports should show the status of the testing based on the test plan. Reporting should document what tests have been performed and the status of those tests. To properly report status, the testers should review and conduct statistical analysis on the test results and discovered defects. The lessons learned from the test effort should be used to improve the next iteration of the test process. Metrics of Testing Metrics specific to testing include data collected on testing, defect tracking, and software performance. Use quantitative measures and metrics to manage the planning, execution, and reporting of software testing, should focus on whether test objectives and goals are being reached. Test Status Reports

Reports the status of testing as specified in the test plan and would include information on: 1. Test Plan Coverage – percent of test plan completed. 2. Code Coverage – monitoring the execution of software and reporting on the degree of coverage at the statement, branch, or path level. 3. Requirement Coverage – monitoring and reporting on the number of requirements tested, and whether or not they are correctly implemented. 4. Test Status Metrics: a. Metrics Unique to Test – includes metrics such as Defect Removal Efficiency, Defect Density, and Mean Time to Last Failure. b. Complexity Measurements – quantitative values accumulated by a predetermined method, which measure the complexity of a software product. c. Project Metrics – status of project including milestones, budget and schedule variance and project scope changes. d. Size Measurements – methods primarily developed for measuring the software size of information systems, such as lines of code, and function points. These can also be used to measure software testing productivity. Sizing is important in normalizing data for comparison to other projects. e. Defect Metrics – values associated with numbers or types of defects, usually related to system size, such as “defects/1000 lines of code” or “defects/100 function points”; severity of defects, uncorrected defects, etc. f. Product Measures – measures of a product’s attributes such as performance, reliability, failure, usability. Final Test Reports 1. Reporting Tools – use of word processing, database, defect tracking, and graphic tools to prepare test reports. 2. Test Report Standards – defining the components that should be included in a test report. 3. Statistical Analysis – ability to draw statistically valid conclusions from quantitative test results.

CSTE Knowledge Category 7

Body

of

Knowledge

User Acceptance Testing The objective of software development is to develop the software that meets the true needs of the user, not just the system specifications. To accomplish this, testers should work with the users early in a project to clearly define the criteria that would make the software acceptable in meeting the user needs. As much as possible, once the acceptance criterion has been established, they should integrate those criteria into all aspects of development. This same process can be used by software testers when users are unavailable for test; when diverse users use the same software; and for beta testing software. Concepts of Acceptance Testing 1. Acceptance testing is a formal testing process conducted under the direction of the software users to determine if the operational software system meets their needs and is usable by their staff. 2. Understand the difference between system test and acceptance test. Roles and Responsibilities The software testers need to work with users in developing an effective acceptance plan, and to ensure the plan is properly integrated into the overall test plan. If users are not available the software testers may become responsible for acceptance testing. Acceptance Test Planning Process The acceptance test plan should include the same type of analysis used to develop the system test plan with emphasis on: 1. Defining the acceptance criteria 2. Develop an acceptance test plan for execution by user personnel 3. Test data is use case oriented Acceptance Test Execution 1. Execute the acceptance test plan 2. Develop an acceptance decision based on the results of acceptance testing. 3. Sign off by users upon successful completion of the acceptance test plan.

CSTE Knowledge Category 8

Body

of

Knowledge

Testing Software Developed by Outside Organizations Many organizations do not have the resources to develop the type and/or volume of software needed to effectively manage their business. The solution is to obtain or contract for software developed by another organization. Software can be acquired by purchasing off the shelf software (COTS) or contracting for all or parts of the software development to be done by outside organizations, often referred to as outsourcing. Software testers need to be involved in the process of testing software acquired from outsourcers. Specifically, this category addresses: The difference in testing software developed in-house versus software developed by outside organizations. Differences between testing software developed in-house and software developed by outside organizations: 1. COTS Software – testers normally do not have access to the methods in which the software was developed or the people who developed it. 2. Contractors/Outsourced – the contractual provisions will determine whether testers can perform verification activities during development; and the ability of testers to access the developers. Selection Process for Acquired Software: 1. Selecting COTS Software. This involves first determining the needed requirements; second, the available software that might meet the requirements, and then third, evaluating those software packages against the selection criteria. Testers can perform or should participate in this process. Note that the acquisition of test tools follows this same process. 2. Selecting organizations to build all or part of the needed software. Testers should be involved in these activities, specifically to: a. Assure that requirements are testable. b. Review the adequacy of the test plan to be performed by the outsourcing organization. c. Oversee acceptance testing. d. Issue a report on the adequacy of the software to meet the contractual specifications. e. Assure compatibility of software standards, communications, change control etc. between the two organizations. Testing Acquired Software

Uses the same approach as used for in-house software, but may need to be modified based on documentation available from the developer. Testers Involvement in Testing Changes for Purchased/Contracted Software The objectives of involving testers in testing changes include: 1. Testing the changed portion of the software. 2. Perform regression testing. 3. Compare the documentation to the actual execution of the software. 4. Issue a report regarding the status of the new version of the software.

CSTE

Body

of

Knowledge

Knowledge Category 9

Testing Software Controls and the adequacy of Security Procedures The software system of internal control includes the totality of the means developed to ensure the integrity of the software system and the products created by the software. Controls are employed to control the processing components of software, assure that software processing is in accordance with the organization's policies and procedures, and according to applicable laws and regulations. Software systems are divided into two parts, the part that performs the processing and the part that controls processing. The control part includes a system of controls as well as the means employed to assure processing cannot be penetrated by outside sources. This category addresses all the components of the software system of internal control and security procedures. Principles and Concepts of a Software System of Internal Control and Security 1. Vocabulary of Internal Control and Security – the vocabulary of internal control and security which includes terms such as risk, threat, control, exposure, vulnerability and penetration. 2. Internal Control and Security Models – includes internal control and security models. The current model that is most accepted is the COSO model. (Committee of Sponsoring Organizations, COSO, is comprised of five major U.S. accounting associations.) Testing the System of Internal Controls The test process for testing the system of internal controls in software is: 1. Perform risk analysis – determine the risks faced by the transactions/events processed by the software.

2. Determine the controls for each of the processing segments for transactions processing including: a. transaction origination b. transaction entry c. transaction processing d. data base control e. transaction results 3. Determine whether the identified controls are adequate to reduce the risks to an acceptable level. 4. When all components of the control system are present and functioning effectively, the internal control process can be deemed “effective.” Testing the Adequacy of Security for a Software System Testers need to evaluate the security for an individual software system. The tests should include: 1. Evaluate the adequacy of management’s security environment. 2. Security Risk Assessment – determining the types of risk requiring security controls. 3. Identify the most probable points where the software would be penetrated. 4. Determine the controls at those points of penetration. 5. Test/assess whether those controls are adequate to reduce the security risks to an acceptable level. These tests should include: a. Security awareness of the software stakeholders b. Adequacy of management’s security environment.

CSTE

Body

Knowledge Category 10

Testing New Technologies

of

Knowledge

Testers require skills in their organization’s current technology, as well as a general understanding of the new information technology that might be acquired by their organization. This knowledge category addresses: An Understanding of the New Testing Challenges with These Technologies: 1. New application architecture including: a. web based applications b. PDA’s 2. New application business models including: a. e-commerce b. e-business 3. New communication methods including: a. wireless 4. New testing tools including: a. test automation software Evaluating New Technologies to Fit into the Organization’s Policies and Procedures Assessing the adequacy of the controls within the technology and the changes to existing policies and procedures that will be needed before the new technology can be implemented effectively. This would include: 1. Testing new technology to evaluate actual performance versus supplier’s stated performance. 2. Determine whether current policies and procedures are adequate to control the operation of the new technology and modify to bring in currency. 3. Assess the need to acquire new staff skills to effectively implement the new technology.

Bibliographic References IMPORTANT: It is each candidate's responsibility to stay current in the field and to be aware of published works and materials available for professional study and development. Software Certifications recommends that candidates for certification

continually research and stay aware of current literature and trends in the field. There are many valuable references that have not been listed here. These references are offered for informational purposes only.