Building The Test Environment

Skill Category 2

Building the Test Environment

1

Skill Category 2 • • • • • • • • • • • • • • • • •

Building the Test Environment .............................................................. 133 Management Support ............................................................................. 133 Management Tone ..................................................................... 134 Integrity and Ethical Values ..................................................................... 135 Commitment to Competence ..................................................................... 137 Management’s Philosophy and Operating Style ............................................................. 137 Organizational Structure ...................................................................... 138 Test Work Processes .............................................................................. 140 The Importance of Work Processes ...................................................................... 140 Responsibility for Building Work Processes .................................................................... 142 Developing Work Processes ...................................................................... 149 Tester’s Workbench ...................................................................... 150 Analysis and Improvement of the Test Process .............................................................. 153 Test Tools .............................................................................. 167 Tool Development and Acquisition ...................................................................... 167 Tool Usage ...................................................................... 179 Testers Competency .............................................................................. 180

2

The test Environment is comprised of the following conditions within the organization: • • • • • • • • • • • •

policies procedures culture attitudes rewards test processes test tools methods for developing and improving test processes management’s support of software testing test labs developed for the purpose of testing software multiple operating environments assuring the test environment fairly represents the production environment to enable realistic testing to occur 3

Management Support = Good for testers. Following will be discussed: • Management “tone”: Management sets the tone by providing testers the resources and management time needed to do their job effectively. • Test process alignment: Management creates a test function that is aligned with the business needs of the organization.

4

Testers need to know 3 things about the control environment: 1. The control environment is established by the highest levels of management and works downward through the organization. 2. The test function cannot create the organization’s control environment, but can influence how that environment is implemented within the test function. 3. The control environment will influence the way in which testers perform the work which may be ethical or unethical. 5

Integrity and Ethical Values Excerpt: “Integrity is a prerequisite for ethical behavior in all aspects of an enterprise’s activities. A strong corporate ethical climate at all levels is vital to the wellbeing of the corporation, all of its constituencies, and the public at large.”

6

Incentives and Temptations

Individuals may engage in dishonest, illegal or unethical acts simply because their organizations give them strong incentives or temptations to do so.

7

Incentives cited for engaging in fraudulent unethical behavior: •

Pressure to meet unrealistic performance targets, particularly for short-term results • High performance-dependent rewards.

Temptations for employees to engage in improper acts: •

Nonexistent or ineffective controls, such as poor segregation of duties in sensitive areas that offer temptations to steal or to conceal poor performance. • High decentralization that leaves top management unaware of actions taken at lower organizational levels and thereby reduces the chances of getting caught. • A weak internal audit function that does not have the ability to detect and report improper behavior. • Penalties for improper behavior that are insignificant or unpublicized and thus lose their value as deterrents. 8

Providing and Communicating Moral Guidance The most effective way of transmitting a message of ethical behavior throughout the organization is by example.

9

Commitment to Competence • Competence should reflect the knowledge and skills needed to accomplish tasks that define the individual’s job. • Management needs to specify the competence levels for particular jobs and to translate those levels into requisite knowledge and skills. 10

Management’s Philosophy and Operating Style •

Management’s philosophy and operating style affect the way testing is managed, including the kinds of business risks accepted. – Informal: An informally managed company may control operations largely by face-to-face contact with key managers. – Formal: A more formally managed exception reports. • Other elements of management’s philosophy and operating style include: – attitudes toward financial reporting – conservative or aggressive selection from available alternative accounting principles – conscientiousness and conservatism with which accounting estimates are developed – attitudes toward data processing and accounting functions and personnel. 11

Organizational Structure •

An entity’s organizational structure provides the framework within which its activities for achieving entity-wide objectives are planned, executed, controlled and monitored.

•

Activities that relate to the value chain: – inbound (requirements) activities – operations or production – outbound (software) – deployment and maintenance

•

Support Functions: – administration, – human resources – Technology development 12

Assignment of Authority and Responsibility • Management has an important function with the assignment of authority and responsibility for operating activities, and establishment of reporting relationships and authorization protocols. • Management function deals with: – policies describing appropriate business practices – knowledge and experience of key personnel – resources provided for carrying out duties 13

Human Resource Policies and Practices Messages to employees: •

• • • •

Standards for hiring the most qualified individuals, with emphasis on educational background, prior work experience, past accomplishments and evidence of integrity and ethical behavior, demonstrate an organization’s commitment to competent and trustworthy people. Recruiting practices that include formal, in-depth employment interviews and informative and insightful presentations on the organization’s history, culture and operating style send a message that the organization is committed to its people. Training policies that communicate prospective roles and responsibilities and include practices such as training schools and seminars, simulated case studies and role-play exercises, illustrate expected levels of performance and behavior. Rotation of personnel and promotions driven by periodic performance appraisals demonstrate the entity’s commitment to the advancement of qualified personnel to higher levels of responsibility. Competitive compensation programs that include bonus incentives serve to motivate and reinforce outstanding performance. Disciplinary actions send a message that violations of expected behavior will not be tolerated.

14

Test Work Processes • Work processes are the policies, standards and procedures in a quality IT environment. Test work processes are those work processes specific to the testing activity. – Policy: Managerial desires and intents concerning either process (intended objectives) or products (desired attributes). – Standards: The measure used to evaluate products and identify nonconformance. The basis upon which adherence to policies is measured. – Procedure: The step-by-step method that ensures that standards are met.

15

The Importance of Work Processes • It is important to a quality IT environment to establish, adhere to, and maintain work processes in the testing activity. It is also critical that the work processes represent sound policies, standards and procedures.

16

Major purposes and advantages to having work processes • • • • •

Improves communication Enables knowledge transfer Improves productivity Assists with mastering new technology Reduces defects and cost

17

Responsibility for Building Work Processes •

IT management is responsible for issuing IT policy. Policies define direction and by definition are general rules or principles. It is the standards, which will add the specificity needed for implementation of policies. • Policies are needed in: – – – – – – – – –

Building Systems Testing Systems Maintaining Systems Operating Systems Quality of Systems Security of Systems Allocation of Resources Planning for Systems Training Personnel

18

Responsibility for Standards and Procedures The workers who use the procedures and are required to comply with the standards should be responsible for the development of those standards and procedures. Management sets the direction and the workers define that direction. This division permits each to do what they are best qualified to do. 19

The key concepts of a process engineering program are: • Management provides an organizational structure for the workers to develop their own standards and procedures. • The program is driven by management policies. • Absolute compliance to standards and procedures is required. • A mechanism is provided for the continual maintenance of standards and procedures to make them more effective. 20

Test Process Selection Selecting, developing, and acquiring work processes is an overall IT organization responsibility. Software testers need to both understand how the activity operates AND participate when test processes, and related processes, are selected and put into practice.

21

Test Process Selection (cont’d) IT groups should develop a plan for implementing and operating a process engineering program. This would require a policy for standards, and a charter or job description for the function. These need to be customized for each organization. The specific components that need to be addressed include: – – – – –

Building a Process Engineering Organization Developing a Standard and Procedure for Standards Planning for Standards Writing, Storing, and Retrieving Standards and Procedures Enforcing Standards 22

Building a Process Engineering Organization The structure that is put in place to develop and update policies, standards, and procedures must involve both staff and management. The process engineering program should be directed by management, but implemented by staff. 23

Building a Process Engineering Organization (cont’d) Some guidelines on establishing an organizational structure for process engineering are: – Establish a Process Engineering Committee compromised of the most senior IT managers possible. – Represent all IT organizational areas on the Process Engineering Committee. – Appoint an individual as the Process Engineering Manager. (Note that this can be a part-time assignment.) – Appoint Ad Hoc Committees (i.e., special task forces) to develop individual standards and procedures. – Let the Standards Ad Hoc Committees develop the technical standard. 24

Recommended Standards Organizational Structure The recommended organizational structure is comprised of the following three components: – Process Engineering Manager • A full-time or part-time individual responsible for managing the standards program.

– Process Engineering Committee • A policy-setting board, which runs the standards program.

– Ad Hoc Committee • Small groups which develop single standards at a time. 25

Organizational chart for process engineering

26

Role of Process Engineering Manager • Promote the concept of process engineering. • Be the driving force behind processes. • Administer the standards program defined by the Process Engineering Committee. • Be a resource to the Process Engineering Committee and Ad Hoc Committees. • Ensure involved parties are adequately trained. 27

Role of the Process Engineering Committee • Accept topics for processes. • Set priority for implementation of processes. • Obtain the resources necessary to develop the process. • Approve or reject developed processes.

28

Role of the Ad Hoc Committee • Gain representatives from all involved areas. • Ensure that the committee has between three and eight members in size. • Create the standard and procedure. • Coordinate reviews of the standard with involved parties. • Periodically review and update the standards and procedures previously developed by • the Ad Hoc Committee. 29

Selecting Process Engineering Committee Members

• • • • •

The makeup of the Process Engineering Committee is important. Some guidelines in selecting members are: Select the highest-level manager who will accept the position. Assign individuals who are supportive and enthusiastic over standards. Make long-term assignments to the Process Engineering Committee. Select individuals who are respected by their peers and subordinates. Ensure the appropriate areas of interest are involved. 30

Selecting Process Engineering Committee Members (cont’d) Some guidelines to ensure the appropriate areas of interest are included in developing standards are: • Is every organizational function within IT involved? • Are the activities that interface to the IT function involved; for example, key users and customers? • Are activities having a vested interest involved, such as auditors? • Are all the IT “businesses” represented such as the project leaders, systems programmers, data library, security, help desk, and so forth? 31

Developing Work Processes Prior to creating any processes, the Process Engineering Committee must develop a standard and procedure for developing standards and procedures. The standards and procedures developed by the standards program are products. Those products must be standardized just as the other products produced in an IT function.

32

Defining the Attributes of a Standard for a Standard The standard for a standard and a standard for a procedure, define the format, style, and attributes of a document called a standard and a document called a procedure. This standard for a standard and standard for a procedure become the prototype that will be used by the Ad Hoc Committees in developing standards and accompanying procedure(s). This standard should define the following: – A testing policy – A test workbench to support the policy (i.e., standards and processes) 33

Establishing a Testing Policy • A testing policy is management’s objective for testing. It is the objective to be accomplished. • Good testing does not just happen, it must be planned, and a testing policy should be the cornerstone of that plan. Sample Testing Policy

34

Tester’s Workbench •

The tester’s workbench shows that input products drive the workbench, which uses procedures and standards to produce output products.

35

Professional Test Standards Test professionals should be familiar with the standards published by organizations such as the International Standards Organization (ISO), U.S. Department of Commerce (National Institute of Standards and Technology), and IEEE. In addition, testers should know when these standards apply, and how to obtain the latest versions.

36

Professional Test Standards (cont’d) •

IEEE publishes a body of standards that address software engineering and maintenance. A portion of these standards addresses software testing and test reporting. Listed below are the current standards that apply to the test process. – – – – – –

829-1998 IEEE Standard for Software Test Documentation 830-1998 IEEE Recommended Practice for Software Requirements Specifications 1008-1987 (R1993) IEEE Standard for Software Unit Testing (ANSI) 1012-1998 IEEE Standard for Software Verification and Validation 1012a-1998 IEEE Standard for Software Verification and Validation – Supplement to 1012-1998 – Content Map to IEEE 12207.1 – 1028-1997 IEEE Standard for Software Reviews

•

Other professional organizations have similar standards. The CSTE candidate should have an overall familiarity with the purpose and contents of these standards.

37

Analysis and Improvement of the Test Process •

Test Process Analysis: Test quality control is performed during the execution of the process. Analysis is performed after the test process is completed. – Effectiveness and efficiency of test processes – The test objectives are applicable, reasonable, adequate, feasible, and affordable – Testers do not have the needed competencies to meet the test objectives – The test program meets the test objectives – The correct test program is being applied to the project – The test methodology is used correctly – The task work products are adequate to meet the test objectives – Analysis of the results of testing to determine the adequacy of testing – Adequate, not excessive, testing is performed

38

Continuous Improvement • •

A testing process assessment is one excellent way to determine the status of your current test process. Process assessment is applicable in the following circumstances: – Understanding the state of processes for improvement. – Determining the suitability of processes for a particular requirement or class of requirements. – Determining the suitability of another organization’s processes for a particular contract or class of contracts.

•

The framework for process assessment: – Encourages self-assessment. – Takes into account the context in which the assessed process operates. – Produces a set of process ratings (a process profile) rather than a pass or fail result. – Addresses the adequacy of the management of the assessed processes through generic practices. – Is appropriate across all application categories and sizes of organization.

39

Test Process Improvement Model •

A model for test process improvement has these eight steps: 1. Examine the Organization’s Needs and Business Goals 2. Conduct Assessment 3. Initiate Process Improvement 4. Analyze Assessment Output and Derive Action Plan 5. Implement Improvements 6. Confirm Improvements 7. Sustain Improvement Gains 8. Monitor Performance 40

Test Process Alignment •

In establishing the test environment management must assure that the mission/goals of the test function are aligned to the mission/goals of the organization.

41

Adapting the Test Process to Different Software Development Methodologies • Understanding the software development process used in his/her organization is important in conducting software testing for the following three reasons: – Understanding the developmental timetable and deliverables. – Integrating software – Understanding how software is developed 42

Six major categories of developmental methodologies • • • • • •

Waterfall D-shaped Prototype Spiral RAD Incremental

43

Testers testing software developed by a specific software development methodology need to:

1. Understand the methodology. 2. Understand the deliverables produced when using that methodology which is needed for test purposes. 3. Identify compatible and incompatible test activities associated with the developmental methodology. 4. Customize the software test methodology to effectively test the software based on the specific developmental methodology used to build the software. 44

Test Tools • It is difficult to perform testing economically without the aid of automated tools. • The most important aspect of software testing tools is the process used to acquire those tools. Most of this component of the environment will focus on acquiring or developing testing tools. 45

Tool Development and Acquisition •

Difficulties surrounding the introduction of tools can arise in three areas: – Organizational obstacles – Problems arising from the tools – Obstacles in the computer environment

•

The activities associated with the introduction of tools should include these activities: – Identifying the goals to be met by the tool (or by the technique supported by the tool), and assigning responsibility for the activities required to meet these goals. – Approving a detailed tool acquisition plan that defines the resource requirements for procurement and in-house activities. – Approving the procurement of tools and training, if this is not explicit in the approval of the acquisition plan. – Determining, after some period of tool use, whether the goals have been met. 46

Tool Development and Acquisition (cont’d) •

Test management is assigned responsibility for: – Identifying tool objectives. – Approving the acquisition plan (it may also require approval by funding management). – Defining selection criteria. – Making the final selection of the tool or the source.

•

The Test Manager is responsible for the following activities. Note that a selection group for the more important test tools may assist the Tool Manager. – Identifying candidate tools. – Applying the selection criteria (in informal procurement) or preparing the RFP (in formal procurement). – Preparing a ranked list of tools or sources. – Conducting any detailed evaluations or conference room pilots.

47

Tool Development and Acquisition (cont’d) • This distribution of responsibilities reduces the chances of selecting a tool that: – Does not meet the recognized needs of the organization – Is difficult to use – Requires excessive computer resources – Lacks adequate documentation

48

Event Sequencing • The following event sequence addresses only the introduction of existing tools. • The recommended event sequence allows for two procurement methods for bids: – 1. Informal procurement (by purchase order) – 2. Formal procurement (by a request)

49

Event Sequence Steps • •

• • • • • • • • • • • •

Goals Tool Objectives • Acquisition Activities for Informal Procurement • Acquisition Plan • Selection Criteria • Identify Candidate Tools • User Review of Candidates • Score Candidates • Select Tool B. Acquisition Activities for Formal Procurement • Acquisition Plan • Technical Requirements Document • User Review of Requirements • RFP Generation • Solicitation of Proposals • B6 – Technical Evaluation should be Consistent • Source Selection Procure Tool Evaluation Plan Implementation Plan Training Plan Tool Received Acceptance Test Orientation Modifications Training Use in the Operating Environment Evaluation Report Determine if Goals Are Met

50

Tool Usage The most commonly used tools can be grouped into these eight areas: 1. 2. 3. 4. 5. 6. 7. 8.

Automated Regression Testing Tools Defect Management Tools Performance/Load Testing Tools Manual Tools Traceability Tools Code Coverage Test Case Management Tools Common tools that are applicable to testing 51

Tool Usage (cont’d) • Most testing organizations agree that if the following three guidelines are adhered to tool usage will be more effective and efficient. – Guideline 1 – Testers should not be permitted to use tools for which they have not received formal training. – Guideline 2 – The use of test tools should be incorporated into test processes so that the use of tools is mandatory, not optional. – Guideline 3 – Testers should have access to an individual in their organization, or the organization that developed the tool, to answer questions or provide guidance on using the tool. 52

Testers Competency • Test competency is a direct responsibility of the individual and the organization that employs that individual. • Software testing organizations should develop a roadmap for testers to pursue to advance their competencies. The roadmap will have these two paths: – Skill Sets – Performance Skills 53

This figure is typical of how a software testing organization may measure an individual tester’s competency.

54