From 2006 Cste Common Body Of Knowledge Study Session Outline Sept. 9, 2006 John Peterson

From 2006 CSTE Common Body of Knowledge Study Session Outline Sept. 9, 2006 John Peterson Vocabulary Why Do We Test Software? The Multiple Roles of the Software Tester Life Cycle Testing

Quality Assurance versus Quality Control – Quality control is concerned with a specific product. – Quality assurance helps establish processes. – Quality assurance is sometimes called quality control over quality control because it evaluates whether quality control is working.

The Cost of Quality Cost of Quality is a term used to quantify the total cost of prevention and appraisal, and costs associated with the production of software. Prevention Costs - Up-front costs for benefits that will be derived later - Establishing procedures, training, tools and planning - Spent before the product is actually built Appraisal Costs - Review completed products against requirements - Includes the cost of inspections, testing, and reviews. - After the product is built but before it is shipped to the user Failure Costs - Defects that make it to the user or to production. - Repairing products to make them meet requirements. - Cost of operating faulty products and operating a Help Desk. The majority of costs associated with the Cost of Quality are associated with the identification and correction of defects.

Software Quality Factors The software quality factors are attributes of the software that, if they are wanted and not present, pose a risk to the success of the software.. Identifying the software quality factors is done by surveying the decision makers. Their answers prioritize testing. Software Quality Factor Examples Correctness Authorization File Integrity Audit Trail Continuity of Processing Service Levels

How Quality is Defined Quality is frequently defined as meeting the customer's requirements the first time and every time. Quality is also defined as conformance to a set of customer requirements that, if met, result in a product that is fit for its intended use. The common thread that runs through today's quality improvement efforts is the focus on the customer and, more importantly, customer satisfaction.

Why Do We Test Software? Developers are unable to build defect-free software.

Developers are not Good Testers The disadvantages of a person checking their own work using their own documentation are as follows: Misunderstandings will not be detected, because the checker will assume that what the other individual heard from him was correct. Improper use of the development process may not be detected because the individual may not understand the process. The individual may be “blinded” into accepting erroneous system specifications and coding because he falls into the same trap during testing that led to the introduction of the defect in the first place. Information services people are optimistic in their ability to do defect-free work and thus sometimes underestimate the need for extensive testing. Without a formal division between development and test, an individual may be tempted to improve the system structure and documentation, rather than allocate that time and effort to the test.

What is a Defect? A defect is an undesirable state. In order to know what a defect is we must first define a desirable state. What is Quality Software? IT evaluates software against requirements. User’s of software view of quality software means fit for use. Just because software meets IT requirements doesn’t mean a user will find it fit for use. When software doesn’t meet requirements, it is an IT gap When software isn’t fit from the user’s perspective, it is a user gap. To close the user’s gap, the IT quality function must understand user needs with customer surveys and customer testing. Testers need to ensure that software not only meets the requirements, but is in fact usable by the customer.

Why Does a Development Process Produce Defects? Ideally, the software development process should produce the same results each time the process is executed. The example.. If a process that produces one function-point-of-logic in 100 person hours the first time, that process should produce one function-point-of-logic in 100 hours when ran again. If the process takes 110 hours to produce one functionpoint-of-logic, then there is “variability” in the software development process.

Can anyone explain what a “function-point-of-logic” is? Variability is the “enemy” of quality – maturing a software development process should reduce variability.

The concept of measuring and reducing variability is commonly called statistical process control (SPC). Variation is measured with statistics like standard deviation. Standard deviation is basically how spread out the values in a data set are. Examples of Typical Sources of Variation Measurement Components Counting Sampling

Material Components Forms Suppliers

Machine Components Office equipment Computers Software

Method Components Procedures Policies

People Components Training Experience Attitude Aptitude

Environment Components Temperature Humidity Noise Level Lighting

What are some things that could be varying in figures 5 and 6? (What occurrence might have been counted and plotted?)

Special causes of variation are not typically present in the process. They occur because of special or unique circumstances. If special causes of variation exist, the process is unstable or unpredictable. Special causes must be eliminated to bring a process into a state of statistical control. A state of statistical control is established when all special causes of variation have been eliminated. Someone please help with this. I don’t think this really says to ignore special causes so the statistics look good. The book doesn’t give examples of special causes of variation. Can anyone provide examples?

I think that if we some answers to “variation in what?” by this point, this section will be more understandable. Bringing a process into a state of statistical control is not really improving the process; it is just bringing it back to its typical operation. Reducing variation due to common causes is process improvement and the real essence of continuous process improvement. As previously mentioned, variation due to special causes must be identified and removed to create a stable process. However, a stable process may not be an acceptable process. If its variation, due to common causes results in operation of the process beyond specifications, the process is called “incapable.” The process can be taken from incapable to capable by reducing variation due to common causes, retargeting the process, or both.

Software testing does not add a lot of value to the business if all they are doing is validating that incorrect requirements are implemented correctly.

Reducing the Frequency of Defects in Software Development Department of Defense (DOD) had Carnegie Mellon University’s Software Engineering Institute (SEI) study what makes software development more effective. The result was the capability maturity model. The Five Levels of Maturity As the model moves from Level 1 to Level 5, the variability in the process is significantly reduced. Thus, those at Level 5 have minimal variability in their software development process, while Level 1 organizations have significant variability. The cost differences to produce a function point of logic between a Level 1 and Level 5 organization may vary by 100 times. In other words, what a Level 1 organization may spend on building software, for example $1,000, may only cost $10 for a Level 5 organization.

Level 1 – Ad Hoc Ad hoc means unstructured, inconsistent levels of performance. At the ad hoc level, tasks are not performed the same way by different people or different groups. Level 2 – Control There are two major objectives to be achieved at Level 2. The first is to instill discipline in the culture of the information organization so that through the infrastructure, training, and leadership of management individuals will want to follow defined processes. The second objective is to reduce variability in the processes by defining them to a level that permits relatively constant outputs. Level 3 – Core Competency At this level, an information organization defines its core competencies and then builds an organization that is capable of performing those core competencies effectively and efficiently.

Level 4 – Predictable This level has two objectives. The first is to develop quantitative standards for the work processes based on performance of the Level 3 stabilized processes. The second objective is to provide management the dashboards and skill sets needed to manage quantitatively. The result is predictable work processes. Level 5 – Innovative At Level 5, the information organization wants to be a true leader in the industry. At this level, the organization is looking to measure itself against the industry through benchmarking, and then define innovative ways to achieve higher levels of performance. Innovative approaches can be achieved through benchmarking other industries, applying new technology in an innovative way, reengineering the way work is done, and by constantly studying the literature and using experts to identify potential innovations. This level is one in which continuous learning occurs, both in individuals and the organization.

The Multiple Roles of the Software Tester

The major factors that affect the tester’s role are: People relationships Scope of testing When should testing occur How should testing be performed Testing constraints

People Relationships Organizations are finally becoming convinced that testers truly can not test in quality at the end of the development process and that the traditional waterfall methodology has lead to many of the issues surrounding testing today. I’m rusty on the waterfall methodology. Anyone want to explain this statement? Some attitudes that have shaped a negative view of testing and testers are: Testers hold up implementation. Giving testers less time to test will reduce the chance that they will find defects. Letting the testers find problems is an appropriate way to debug. Defects found in production are the fault of the testers. Testers do not need training; only programmers need training. Essential testing skills include test planning, using test tools (automated and manual), executing tests, managing defects, risk analysis, test measurement, designing a test environment, and designing effective test cases.

People Relationships Top ten people challenges: Training in testing Relationship building with developers Using tools Getting managers to understand testing Communicating with users about testing Making the necessary time for testing Testing “over the wall” software Trying to hit a moving target Fighting a lose-lose situation Having to say “no”

Any thoughts about the red items?

Scope of Testing The scope of testing is the extensiveness of the test process. A narrow scope may be limited to determining whether or not the software specifications were correctly implemented. The scope broadens as more responsibilities are assigned to software testers. Among the broader scope of software testing are these responsibilities: 1. Determining if the user’s needs have been met regardless of specs 2. Finding defects early when they cost less 3. Removing defects prior to going into a production 4. Identifying weaknesses in development process to improve process and make the development process more mature. In defining the scope of software testing each IT organization must answer the question, “Why are we testing?”

When Should Testing Occur? When testing is constrained to a single phase and confined to the later stages of development, severe consequences can develop. An error discovered in the latter parts of the life cycle must be paid for four different times. The first cost is developing the program erroneously, which may include writing the wrong specifications, coding the system wrong, and documenting the system improperly. Second, the system must be tested to detect the error. Third, the wrong specifications and coding must be removed and the proper specifications, coding, and documentation added. Fourth, the system must be retested to determine that it is now correct. Verification must not be isolated to a single phase in the development process, but rather, incorporated into each phase of development.

When Should Testing Occur? The recommended test process involves testing in every phase of the life cycle. During the requirements phase, the emphasis is upon validation to determine that the defined requirements meet the needs of the organization. During the design and program phases, the emphasis is on verification to ensure that the design and programs accomplish the defined requirements. During the test and installation phases, the emphasis is on inspection to determine that the implemented system meets the system specification. During the maintenance phases, the system will be retested to determine that the changes work and that the unchanged portion continues to work.

When Should Testing Occur? Requirements The adequacy of the requirements must be thoroughly analyzed and initial test cases generated with the expected behavior Generating these tests and the expected behavior of the system clarifies the requirements and helps guarantee that they are testable. Design Test plan is produced. Test schedule with observable milestones is constructed Validation support tools should be acquired or developed Perform design inspections using design walkthroughs.

When Should Testing Occur? Program (Build/Construction) Actual testing occurs during the construction stage of development. As opposed to some kind of non-actual testing? Test Process Test sets, test results, and test reports should be catalogued and stored in a database. Installation Placing tested programs into production is an important phase normally executed within a narrow time span. Testing during this phase must ensure that the correct versions of the program are placed into production.

When Should Testing Occur? Maintenance As the system is used, it is modified either to correct errors or to augment the original system. After each modification the system must be retested. Such retesting activity is termed regression testing. If test cases from initial development have been catalogued and preserved, duplication of effort will be minimized.

How the Test Plan Should be Developed The applicable test objectives would be listed and ranked, and the phases of development would be listed as the phases in which testing must occur. Four steps must be followed to develop a customized test plan: 1. Select and rank test objectives. Customers or key users of the system in conjunction with the test team should define and rank the test objectives 2. Identify the system development phases. 3. Identify the business risks 4. Place risks in the matrix The risk team should determine the test phase in which the risk needs to be addressed by the test team and the test objective to which the risk is associated.

Testing Constraints Budget and Schedule Constraints May limit the ability complete the test plan Costs dramatically increase the later a defect is found The risk of under testing is directly translated into system defects present in the production environment. The risk of overtesting is the unnecessary use of valuable resources At some point, an over test condition begins and the cost of testing to uncover defects exceeds the losses from those defects.

Most problems associated with testing occur from one of the following causes: Failing to define testing objectives Testing at the wrong phase in the life cycle Using ineffective test techniques

Testing Constraints Incomplete Statement of Work If requirements are lacking or poorly written, then the test team must have a defined method for uncovering and defining test objectives. Changes in Technology Integration Technology is being more closely integrated into day-to-day business System Chains Problems in one can cascade into and affect others. The Domino Effect One problem condition, can cause hundreds or even thousands of similar errors within a few minutes. Reliance on Electronic Evidence The validity of transactions is dependent upon controls, and thus a control error may result in extensive losses. Multiple Users Systems belong to multiple users, making it difficult to identify a single organizational unit responsible for a system.

Testing Constraints Limited Tester Skills Testers should be competent in all ten knowledge categories to be effective. Does anyone know the list of ten skills?

Software Risks One of the most effective methods to reduce computer system strategic risk is testing. Risks deemed important to reduce become the areas for testing. Software Design Defects Designing software with incomplete or erroneous decision-making criteria Failing to program the software as intended Omitting needed edit checks for determining completeness of output data.

Data Defects Poor quality data can adversely affect the computer-directed actions. Finding Defects Testing focuses on discovering and eliminating defects or variances from what is expected. Testers need to identify these two types of defects: Variance from Specifications A defect from the perspective of the builder of the product. Variance from what is Desired A defect from a user (or customer) perspective.

Why Are Defects Hard To Find? There are at least two reasons defects go undetected: Not Looking Looking, But Not Seeing

Life Cycle Testing Life cycle testing involves continuous testing of the solution even after software plans are complete and the tested system is implemented. Testing should not use the same methodology that was used for developing the system. Testing does not stop once you’ve completely implemented your system; it must continue until you replace or update it again!