Software Qm And Pm

ARTICLE IN PRESS

The Journal of Systems and Software xxx (2006) xxx–xxx www.elsevier.com/locate/jss

An intelligent early warning system for software quality improvement and project management Xiaoqing (Frank) Liu *, Gautam Kane, Monu Bambroo Department of Computer Science, University of Missouri-Rolla, 1870 Miner Circle, Rolla 65401, United States Received 30 November 2004; received in revised form 23 January 2006; accepted 29 January 2006

Abstract There are three major problems with software projects: over budget, behind schedule, and poor quality. It is often too late to correct these problems by the time they are detected in failed software projects. In this paper, we discuss design, implementation, and evaluation of an experimental intelligent software early warning system based on fuzzy logic using an integrated set of software metrics. It consists of the following components: software metrics database, risk knowledge base, intelligent risk assessment, and risk tracing. It helps to assess risks associated with the three problems from perspectives of product, process, and organization in the early phases of the software development process. It is capable of aggregating various partial risk assessment results into an overall risk indicator even if they may be conflicting. In addition, it can be used to analyze a risk by identifying its root causes through its risk tracing utility. Ó 2006 Elsevier Inc. All rights reserved. Keywords: Early warning; Fuzzy logic; Metrics; Process improvement; Quality management

1. Introduction 1.1. Background Lots of efforts and money is wasted on unfruitful software development projects. One of the main reasons behind failed software projects is that it is often too late to correct the problems by the time they are detected. It clearly indicates the need for early warning about the potential risks. In practice, one of main causes behind the inability for early warning in a software project is its lack of important data about many software artifacts, software development process elements, and organization units, and lack of tools for analyzing available software metrics, and lack of tools for visualizing problems of development. Not only is the measurement of software entities difficult, but also the way to analyze these measurements. It is even

*

Corresponding author. Tel.: +1 5733414848; fax: +1 5733414501. E-mail addresses: fl[email protected] (Xiaoqing (Frank) Liu), [email protected] (G. Kane), [email protected] (M. Bambroo). 0164-1212/$ - see front matter Ó 2006 Elsevier Inc. All rights reserved. doi:10.1016/j.jss.2006.01.024

more difficult to assess risks based on these software metrics. Many software metrics have been proposed and used to measure the software entities at different stages of software development. How to make a systematic use of these metrics to identify future risks in software development remains challenging. Very few methods and tools are developed to address the issue while most of existing risk assessment methods or tools are focused on only a particular aspect of software development. In practice, software risk assessment is often done by experts subjectively. It is not only necessary to identify risks in early development phases, but also important to quantify the risks in order to prioritize them during assessment. Furthermore, it is important to be able to identify the causes of risks so that they can be eliminated. A few research projects for the assessment of individual risk factors directly based on a number of software metrics are still at their preliminary stages. Although they have demonstrated benefits, many challenging issues remain to be resolved. Barry Boehm’s work in software metrics, such as the COCOMO model and the spiral model has originated from

ARTICLE IN PRESS 2

Xiaoqing (Frank) Liu et al. / The Journal of Systems and Software xxx (2006) xxx–xxx

the research on software risk assessment (Boehm, 1981, 1989; Boehm et al., 2000; Boehm, 2005). The quantitative effort and cost estimation model has significant impact in software engineering. Several other research projects have also demonstrated interesting results. NASA’s approach of risk assessment directly based on software metrics has shown promising results (Rosenberg and Hyatt, 1996). NASA’s WISE Project Management System (NASA Software IV & V Facility, 1995) uses the Internet to provide a framework for issue management in software development. The enhanced measurement for early risk assessment of latent defects (EMERALD) (Hudepohl et al., 1996; Khoshgoftaar et al., 1998), developed in the Nortel, consists of decision support tools to help software designers and managers assess risks and improve software quality and reliability in the area of defect analysis. At various points in the development process EMERALD predicts which modules are likely to be fault-prone. In a research paper (Nogueira et al., 2000), a formal model is introduced to assess the risk and duration of software projects based on a few objective indicators that can be measured early in the process. The approach supports (a) automation of risk assessment, and (b) early estimation methods for evolutionary software processes. Project risks related to schedule and budget are addressed with the help of a risk assessment model based on software metrics. The indicators used are requirements volatility (RV), complexity (CX), and efficiency (EF). Individual risk factors are assessed directly based on software metrics. The overall risk associated with the entire process or system often is not obtained. In addition, conflicting results may be obtained based on multiple groups of metrics from multiple perspectives, and it is very difficult to reconcile them since they are crisp. Furthermore, all of the above works are directly based on quantitative measurements represented by numbers, which are sometimes not very accurate, reliable, or complete. The common sense and knowledge of human beings, which form the basis of any risk management, are ignored. On the other hand, fuzzy logic has found a lot of successful applications in risk assessment from financial markets, environment control, project control, to health care. It provides a rich set of mathematical tools for assessing risks associated with software project management and software quality control. Therefore, it is used for the construction of the early warning system presented in this paper. The comparison of neural network models, fuzzy logic models and regression models are made in a research project (MacDonell and Gray, 1997) to find the best predictive technique for effort estimation. It concluded that neurofuzzy hybrids may be used for better estimation. Researchers have been trying to incorporate the fuzzy logic for estimation in software development. The use of case based reasoning for software project management is explored in another recent research project (Vasudevan, 1994). The case indices are expressed using fuzzy sets. Risk assessment is done by employing fuzzy aggregation to evaluate the

cases. Application of fuzzy clustering for software quality prediction is proposed in a paper, where data set is modeled by fuzzy clusters and fuzzy inference technique is applied to predict fault-prone modules (Yuan et al., 2000). The drawbacks of using traditional methods for risk assessment like checklists, risk matrix are discussed and a fuzzy expert system for early operational risk assessment is developed in another related research project (Xu et al., 2002). 1.2. Overview We develop an intelligent early warning system using fuzzy logic based on an integrated set of software metrics from multiple perspectives to make sponsors, users, project managers and software developers aware of many potential risks as early as possible. It has the potential to improve software development and maintenance by a great extent. Various factors for risk assessment in software development can be measured quantitatively using software metrics. All the metrics represent measurements from different perspectives. For example, ‘‘requirement volatility index’’ is associated with a software product artifact, whereas ‘‘effort’’ is associated with a development process. Similarly ‘‘productivity’’ is associated with individuals as well as organization units. In addition we need to capture the relationships of metrics across perspectives. The software development is characterized by three perspectives: product, process, and organization. A set of fuzzy rules are developed based on individual metrics and their combinations from these three perspectives to identify risks. The system is customizable. The thresholds for many inference rules of risks based on metrics may be adjusted for different organizations. Hence it is not wise to have the thresholds hard coded in the system. As mentioned earlier, only a prediction of risk is not enough. The root cause of the risk should be located. Our system has a trace-down capability for the identified risk. The identified risk can be traced in all the three perspectives. 2. Design of the intelligent software early warning system The system architecture is shown in Fig. 1. It contains the following components: (1) Metric database; (2) Risk knowledge base; (3) Object model for software metrics; (4) Intelligent risk assessment engine; and (5) Risk tracing. 2.1. Software metrics database Software metrics are the measurements of software artifacts, software processes and people involved in the development from different aspects. It is difficult to measure software because there is no physical existence of software in the development phases. Software measurement is vague. Software metrics represent properties of software, which can be measured periodically to keep track of the development progress.

ARTICLE IN PRESS Xiaoqing (Frank) Liu et al. / The Journal of Systems and Software xxx (2006) xxx–xxx

Risk Tracing

Intelligent Risk Assessment

Product Process Organization

Project Object Model

Metric Database

Risk Knowledge Base

Fig. 1. System structure.

Many software metrics are used in our system. For example, the ‘‘requirement volatility’’ and ‘‘requirement inconsistency index’’ are used to assess requirement quality and development risks; the ‘‘cyclomatic complexity’’ is used to measure logic complexity of programs; and ‘‘schedule deviation’’ is used to assess risks with being behind schedule at all phases of the development. The metric values are stored in a database so that they can be retrieved whenever needed for the risk assessment. The attributes required for the metrics measurements are also stored in the database so that they can be updated periodically. Metrics database represents the current and past status of the software development. The accuracy and reliability of the risk assessment increase with more accurate software metrics. In this experimental system, values of software metrics are simulated data for hypothetic projects. When a full-scale industry-strength early warning system is developed based on this prototype in the future, a rigorous software metrics collection must be established to assure accuracy and reliability of software metrics in the industrial environment. If the software metrics are wrong, warning signals produced by and risk assessment done by the system will not be reliable. 2.1.1. Dimensional analytic model for metrics database The dimensional analytic model was introduced in a research paper (Liu and Viswanathan, 1999). This model involves software metrics from three perspectives: product, process, and organization. Usually the metrics from only the product perspective are considered in risk assessment, and the other perspectives are ignored. Dimensional analytic model is a comprehensive approach to consider metrics from all three perspectives simultaneously. The software metrics can be categorized and processed accordingly using this model. It becomes easier to collect metrics attributes, when the metrics are categorized into the three perspectives. For example ‘‘code size’’ is a product metric, ‘‘schedule deviation’’ is a process metric, and ‘‘productivity’’ is an organization metric.

3

The dimensional analytic model is shown in Fig. 2. Dimensional analytic model has two parts. The first part is shown in Fig. 2(a). It specifies three perspectives of the software development: product, process, and organization. Each perspective has different levels of components. For example, ‘‘system’’ is the biggest component in the product perspective. A system consists of several subsystems. Each subsystem consists of several modules. ‘‘Project’’ is the biggest component in the process perspective. A project has many phases, such as analysis, design, coding, and testing. Each phase consists of several tasks. Similarly, ‘‘company’’ is the biggest component in the organization perspective. There may be a number of groups/divisions in a company. Each group/division may consist of many software engineers. Certainly, the number of levels in each perspective may vary for different organizations. Software metrics can be collected for a component at any level of a perspective. For example, ‘‘code size’’ can be obtained for each module from the product perspective. ‘‘Schedule deviation’’ can be obtained for each task from the process perspective. Similarly, ‘‘productivity’’ can be obtained for each individual from the organization perspective. Some of the metrics can be used for multiple levels in a perspective. For example, ‘‘code size’’ is both a module-level and a system level metric from the product perspective. It is useful to trace the software risk in different perspectives and to each component in these perspectives. The second part of the dimensional analytic model is shown in Fig. 2(b). It shows the metrics values from a perspective over a period of time. It helps describe the state of a particular software artifact at a particular point of time. By relating the time factor to each of the perspectives and its attributes, it is possible to identify the trend for a particular attribute of a component in a particular perspective over a given period of time. It also helps track the risks over time for a component within a dimension. 2.1.2. Object model for software metrics database Risks should be identified based on objective data about software product, process and organization. Metrics database stores software metrics which are used for risk analysis and warning generation. The software metrics serve as the basis for intelligent risk assessment in software develop-

Product System Subsystem Module

Time Division

Project Phase Task

Company Group Individual

Process

Perspective • Product • Process • Organization

Organization

(a) Fig. 2. Dimensional analytic model.

(b)

Attributes

ARTICLE IN PRESS 4

Xiaoqing (Frank) Liu et al. / The Journal of Systems and Software xxx (2006) xxx–xxx

Developed Through

System

Company

Project

Subsystem

Group

Phase

Module

Individual

Task

Work Assignment

Fig. 3. Object model of software metrics database.

ment and maintenance. The metrics database contains three types of metrics: (1) product metrics, (2) process metrics and (3) organization metrics. An object model is used to organize software metrics and visualize software development as shown in Fig. 3. The object model shows a hierarchy of units in each perspective. It has two relationships. The first relationship, ‘‘developed through’’, establishes the relationship between the product and the process perspectives. It shows which systems are developed in which projects. The ‘‘work assignment’’ relationship shows which individual works on which module or subsystem or system and which task is assigned to that individual. Although the software metrics database is organized in terms of a generic software process in which a process consists of many phases and a phase contains many tasks, it does not suggest that a waterfall model must be followed. Any process which can be described using the above generic process can be managed using the system. In the future, we will investigate feasibility of representing other software development models, such as spiral and iterative software development, using the above generic software process so that the proposed system can be applied to software organizations using them.

The consequents can be quality risks, schedule risks or budget risks. There can be more than one consequent in one fuzzy rule. The antecedents and consequents have linguistic variables such as high, medium, low, etc., each having a membership function associated with it. The membership functions are trapezoidal, triangular, Gaussian, etc. Each membership function varies between 0 and 1, while 0 indicates that the metric value does not belong to a particular membership function and 1 indicates that the metric value completely belongs to a particular membership function. The range indicates the degree of membership of a software metric. The membership functions for different linguistic variables of software metric may overlap with each other. In an overlapping area, the metric value belongs to all overlapping linguistic variables with different degrees of membership. Multiple rules may be fired according to current system status. The range of each metric value is divided into fuzzy regions as illustrated in Fig. 4. The rules in our knowledge base which cover all sensible combinations of the fuzzy regions (sets) for different metrics form a rule set. Multiple rules can operate on the same set of metrics but with different linguistic variables. Many rule sets like this can be found in our knowledge base. Some examples of the rules are listed below: 2.2.1. Quality risk rule IF requirements volatility of system is HIGH AND requirements inconsistency of the system is HIGH THEN quality risk of the system is VERY HIGH The requirement volatility is calculated based on number of added requirements (NA), number of revised requirements (NR), and number of deleted requirements (ND) for a period of time using the following model: (NA + NR + ND)/(NT + NA), where NT is the total number of requirements at the start of the period. The units of time periods considered in the system include day, week, month, and year. The requirement inconsistency index is the percentage of inconsistent requirements in terms of total number of requirements over a period of time. ‘‘HIGH’’ is a fuzzy set whose membership function is defined in Fig. 4. The membership function is subjective, and is determined based on experience using criteria, such

2.2. Knowledge base The knowledge base contains the fuzzy rules. The fuzzy rules represent the expert knowledge used to analyze the software metrics and to predict risks. The fuzzy rules are in IF–THEN form. The IF part contains antecedents and the THEN part contains consequents. The antecedents are software metrics and consequents are risks. The antecedents may come from the same perspective (product, process, or organization) or from different perspectives.

LOW

MEDIUM

HIGH

1 0 0

0.2

0.35

0.5

0.6

0.7

0.8

1

Fig. 4. Membership functions of fuzzy set LOW, MEDIUM, and HIGH for requirements volatility.

ARTICLE IN PRESS Xiaoqing (Frank) Liu et al. / The Journal of Systems and Software xxx (2006) xxx–xxx

as risk tolerance level and threshold for issuing an early warning.

Product Metrics

Module Size

Process Metrics

Effort Deviation

Organiz ation Metrics

Productivity

5

Quality Risk

2.2.2. Schedule risk rule IF percentage of modules whose code volatility is over a threshold is HIGH AND total schedule deviation of all active or finished tasks in coding phase is HIGH THEN schedule risk associated with the system in coding phase is VERY HIGH Code volatility is defined similarly as requirements volatility except that it is based on number of source lines of code which are added, deleted, and revised. The percentage is represented as a decimal number. The threshold is subjective and is determined similarly like membership functions of fuzzy sets. Once again, ‘‘HIGH’’ and ‘‘VERY HIGH’’ are fuzzy sets. It should be pointed out that their membership functions are rule-specific since they are used in many rules. 2.2.3. Budget risk rule IF percentage of modules whose design volatility is over a threshold is HIGH AND percentage of modules whose design complexity is over a threshold is HIGH AND total budget deviation of all active or finished tasks is HIGH THEN budget risk associated with the system in design phase is VERY HIGH Design volatility is defined similarly as requirements volatility except that it is based on number of design items which are added, deleted, and revised. 2.3. Intelligent risk assessment Intelligent risk assessment is the core processing unit of our framework. The functional block which performs risk assessment is ‘‘fuzzy inference system’’ (FIS). This is algorithmic block of the system as shown in Fig. 5. The inputs to the FIS are from the software metrics database and the knowledge base. FIS uses the knowledge base to retrieve fuzzy rules for specific types of risks. It retrieves software metric values from the metric database at a specific point of time during the software development. Therefore, the inputs to the fuzzy inference engine are product metrics, process metrics, organization metrics, and the fuzzy rules. The outputs of the fuzzy inference engine are quality risks, schedule risks, and budget risks. The risk outputs are the quantified risk values in the range between zero and one. The fuzzy inference system establishes a nonlinear relationship between the software metrics from all perspectives and future risks. Various steps involved in fuzzy inference are described below.

Inference Engine

Cost Overrun Risk

Schedule Overrun Risk

Fig. 5. Fuzzy inference engine.

2.3.1. Fuzzification of software metrics Fuzzification is the process of determining the fuzzy region(s) which the software metric belongs to, and finding the degree of membership for each fuzzy region. The metric value may belong to more than one fuzzy region simultaneously with different degrees of membership. The crisp input value of a software metrics is converted into degrees of membership in related fuzzy regions. For example, in Fig. 6, the value of 0.7 for requirements volatility is transformed into a set {‘HIGH’, 0.5}. 2.3.2. Antecedents in fuzzy rules A fuzzy rule may have more than one antecedent. Each antecedent is fuzzified. The t-norm operator is applied to the fuzzified inputs. The t-norm operator is the fuzzy min operator. The output of this step is the minimum of the degrees of membership for the input software metrics in a fuzzy rule. For example, in Fig. 7, the resultant value is 0.5. 2.3.3. Fuzzy inference Similar to the antecedent, the consequent of a fuzzy rule also has a range of output values. In this system, the range

HIGH

0.5

Requirements Volatility is HIGH Requirements Volatility = 0.7

Fig. 6. Software metrics fuzzification.

ARTICLE IN PRESS 6

Xiaoqing (Frank) Liu et al. / The Journal of Systems and Software xxx (2006) xxx–xxx

HIGH

HIGH

0.8 0.5

t-norm operator

0.5

Requirements Volatility

Requirements Inconsistency

Quality Risk

(Rule 1) Requirements Volatility is HIGH

Requirements Inconsistency is HIGH

Requirements Volatility = 0.7

Requirements Inconsistency = 0.8

HIGH

MEDIUM

HIGH (Rule 2)

MEDIUM

HIGH

HIGH

Fig. 7. Antecedent evaluation. (Rule 3)

of risk values is between 0 and 1. Fig. 8 shows an example of fuzzy inference using the rule ‘‘If requirements volatility is HIGH and requirements inconsistency is HIGH Then quality risk of the system is VERY HIGH’’. The output from the previous step is 0.5 as explained. This value is projected on the membership function of ‘‘Very High’’ fuzzy region of quality risk. The t-norm operator is applied to the output membership function. The t-norm operator is a fuzzy min operator which cuts off the portion of the membership function of the output risk above the output value of 0.5. This step is shown in Fig. 8. This produces a trapezoidal shaped output membership function for very high quality risk. Each rule produces its own output risk region. 2.3.4. Aggregation Many rules may have the same consequent. Each rule produces the region of such a consequent separately. Aggregation is a process in which different output risk regions from the previous step are combined into a single fuzzy set. The input to the aggregation process is the list of truncated output functions from the fuzzy inference step. The output of the aggregation process is a single fuzzy set for the combined risk. Many aggregation operators, such as are t-norm, s-norm, and compromise operator, are available in fuzzy logic (Zimmermann, 1991). The tnorms operators are a class of fuzzy conjunction operators. Examples of t-norms operators includes $MIN$ and algebraic product. The s-norm operators are a class of fuzzy disjunction operators. Examples of s-norm operators include MAX and algebraic sum. In this system, s-norm operator MAX is used for aggregation. An example of

Antecedents HIGH HIGH

Requirements Volatility is HIGH

Requirements Inconsistency is HIGH

Requirements Volatility = 0.7

Requirements Inconsistency = 0.8

Consequent VERY HIGH

VERY HIGH

0.5

HIGH

VERY HIGH

HIGH

(Rule 4) MEDIUM

MEDIUM

Requirements Volatility = 0.7

MEDIUM

Requirements Inconsistency = 0.8

Aggregation s-norm (Max)

Output Quality Risk Fuzzy Region

Fig. 9. Aggregation.

the aggregation of quality risk using s-norm operator MAX is shown in Fig. 9. Averaging and compensatory compromise operators are often used to combine multiple risk factors in fuzzy risk assessment if they conflict with each other. The result of an average operator lies between the most optimistic lower bound and the most pessimistic upper bound. 2.3.5. Defuzzification Defuzzification is the process of converting a fuzzy region back into a crisp output value. The input for the defuzzification process is a fuzzy set (the aggregated output fuzzy region) and the output is a single crisp numeric value. The aggregated fuzzy region cannot be expressed as an output of FIS, because it is hard and inconvenient to infer the risk from the fuzzy region. While the input numeric metrics values need to be fuzzified for risk assessment, the output region must be converted back into a numeric value to the end user. There are many defuzzification methods available, such as centroid, mean of max, etc. Centroid defuzzification is used in this system prototype. Fig. 10 shows the defuzzification of a fuzzy region obtained in the aggregation step. The centroid of the fuzzy region is 0.6. Hence the output of the FIS is 0.6.

Quality Risk is VERY HIGH

Fig. 8. Implication method.

Quality Risk = 0.6 Fig. 10. Risk assessment through defuzzification.

ARTICLE IN PRESS Xiaoqing (Frank) Liu et al. / The Journal of Systems and Software xxx (2006) xxx–xxx

2.4. Risk tracing The knowledge base contains rules to estimate risks for the components at each level from all perspectives and in each phase. Since the risk for each component can be estimated separately, we can trace the risks associated with higher levels of components down to lower levels of components. For example, risks associated with the whole system from the product perspective can be traced down to

7

the subsystem level. The risks associated with each subsystem are traced down to the module level. Then the risks associated with each metric for each component are estimated one at a time. The risks estimated in such a technique represent the responsibility of one metric towards the risk of a complete component. This technique is used for comparative analysis of risk responsibility from different metrics. Illustrative example of risk tracing is given in Section 3.

Fig. 11. Main menu.

Fig. 12. Define product structure.

ARTICLE IN PRESS 8

Xiaoqing (Frank) Liu et al. / The Journal of Systems and Software xxx (2006) xxx–xxx

3. Implementation

3.1. Graphical user interface

The experimental early warning system is implemented in visual basic 6.0 and MS Access database. In the subsequent sections, we will discuss its graphical user interface and its functionalities.

The ‘‘structure’’ menu in Fig. 11 is used to set up the hierarchy of components in the product/process/organization perspectives. The ‘‘relationship’’ menu is used to establish the two types of relationships discussed in the object

Fig. 13. Define process structure.

Fig. 14. Define organization structure.

ARTICLE IN PRESS Xiaoqing (Frank) Liu et al. / The Journal of Systems and Software xxx (2006) xxx–xxx

model. The ‘‘customize’’ menu is used to customize fuzzy regions and fuzzy rules. The ‘‘collect metrics’’ menu is used to collect software metrics or all the parameters required to

9

calculate the metrics for all units in all perspectives. The ‘‘risk estimation’’ menu is used to estimate risk for any unit in any perspective at any particular point of time.

Fig. 15. Fuzzy region customization.

Fig. 16. Fuzzy rules customization.

ARTICLE IN PRESS 10

Xiaoqing (Frank) Liu et al. / The Journal of Systems and Software xxx (2006) xxx–xxx

3.2. Structure menu

3.3. Customization menu

The example of defining product structure is shown in Fig. 12. There are three frames at the top in which the components in each level can be added, modified or deleted. The form used to define the process structure is shown in Fig. 13. The process structure has three levels of units with ‘‘project’’ at the top. A project has several ‘‘phases’’ and each phase consists of several ‘‘tasks’’. These project, phase and task units can be added, updated and deleted. The form used to define the organization structure is shown in Fig. 14. This is also a three level structure with ‘‘company’’ at the top. A company has several ‘‘groups’’ and each group has several ‘‘individuals’’. The company, group and individual units can be added, updated and deleted.

Two types of customization are provided in our prototype system. The fuzzy regions need to be customized because the thresholds for different fuzzy regions may differ among different organizations. An example of fuzzy region setting for a module level metric ‘‘design volatility’’ is shown in Fig. 15. Each region, low, medium, and high, is associated with a trapezoidal membership function which is controlled by four key points, namely, P1, P2, P3, and P4. If these points are changed by a user, the pictorial view of the region changes accordingly on the canvas on the right side. Apart from customizing fuzzy regions, an interface is provided to customize fuzzy rules. An example of rule setting is shown in Fig. 16. A user can change the linguistic variables in all

Fig. 17. Metrics collection.

Fig. 18. Risk assessment.

ARTICLE IN PRESS Xiaoqing (Frank) Liu et al. / The Journal of Systems and Software xxx (2006) xxx–xxx

rules. Rules customization is necessary because the logic for risk assessment using different metrics may be different among organization. 3.4. Metrics acquisition This interface is used to collect parameters for metrics calculation. An example of the metric ‘‘cyclomatic complexity’’ is shown in Fig. 17. Similar graphical user interface are provided for all metrics for all units in all three perspectives. 3.5. Risk assessment This utility is used to assess three types of risks: quality, budget, and schedule for any unit from all three perspec-

11

tives, as shown in Fig. 18. These types of risks are estimated one at a time. Risks can be assessed for a system, a subsystem, and/or a module from the product perspective. Risks estimation can be restricted to a particular group or a particular individual. Risks can also be assessed for a project, a phase, and a task from the process perspective. A project may be at more than one phase at a particular point of time. In such cases, risks at each phase are calculated separately. Risks of the complete project should be greater than individual risks at each phase. Hence, the average or mean operator cannot be used. Otherwise, the result will be less than or equal to greatest risk value. Different formulas are required to estimate risks of a project in various phases. The risk of the complete project is calculated as:

Fig. 19. Risk analysis through risk tracing.

Fig. 20. Risk tracking.

ARTICLE IN PRESS Xiaoqing (Frank) Liu et al. / The Journal of Systems and Software xxx (2006) xxx–xxx

R = R + (1 R) * Ri R = Risk of complete project Ri = Risk of project in phase i

R = 0 (Initial); R = 0 + (1 0) * 0.32 = 0.32 (1st Iteration); R = 0.32 + (1 0.32) * 0.57 = 0.71 (2nd Iteration).

For example, Consider a project in the analysis and design phase:

The risk calculated using this formula is independent of the order of the phases being considered.

R1 = Risk in analysis phase = 0.32; R2 = Risk in design phase = 0.58.

3.6. Risk tracing Risk can be assessed for any level of unit from the product, process, and organization perspectives. Once a risk is identified, its root causes can be analyzed through the risk tracing capability by navigating along object hierarchy as illustrated in the object model. An example of tracing the quality risk of a system to the subsystem and module levels is shown in Fig. 19. The risk can also be traced to the metrics level by considering each of the metrics one at a time. This analysis helps pinpoint the units and the metrics of the units which need to be improved in order to reduce/eliminate the risk. 3.7. Risk monitoring The risk assessment at different time intervals can be used to monitor risk levels for all units of a product, process, and organization. An example of risk tracking is shown in Fig. 20. 4. System evaluation Our intelligent early warning system is evaluated by comparing the test results with the expected results, and sensitivity analysis is performed. Two test results are dis-

Fig. 21. Test I results.

1 0.9 0.8

Quality Risk

12

0.7

Module Design Volatility

0.6

Module Quality Risk Subsystem Quality Risk

0.5 0.4

System Quality Risk

0.3 0.2 0.1 0 Week 5

Week 6

Week 7

Week 8

Week 9

Week 10

Development Time Fig. 22. Test II results.

Week 11

ARTICLE IN PRESS Xiaoqing (Frank) Liu et al. / The Journal of Systems and Software xxx (2006) xxx–xxx

cussed here. In the first test, experimental data are entered and the results are evaluated against the expected results. The data is set up in such a way that the metrics values for four different subsystems are in high, high-medium, medium-low, and low regions, respectively. The expected risk values are very high, high, medium, and low, respectively. The results obtained are shown in Fig. 21. It demonstrates that the results obtained match the expected results. In the second test, only one metric design volatility is changed in each week of the development. The effects of this change on the estimated risk of the module, the subsystem and the system it belongs to are verified. The results obtained are show in Fig. 22. The risks associated with the module, the subsystem, and the system change accordingly as we change the design volatility of that module. Since all the other modules and the other metrics of that module are all kept constant, this change in the risk value reflects only the change in the design volatility of that particular module. The effect on the module level risk is greater than that on the subsystem level risk. The effect on the system level risk is even smaller because there are many subsystems and modules in the whole system, and everything, except for one module, remains the same. Once again, the test result matches our expected result and the system is sensitive to a change in software metrics. 5. Conclusions This research is a successful attempt to design a system to assess risks at the early phases of a software development. Three types of risks are assessed: quality risks, schedule risks, and budget risks. The vague nature of software metrics in software development and the lack of exact software measurements are handled by fuzzy logic. The system collects and analyses the metrics from three perspectives of the software development: product, process, and organization. The risks can be assessed for a project, a phase, and a task from the process perspective; and for a system, a subsystem, and a module from the product perspective. Our system has the ability to trace down a risk to the basic units from each perspective and to find out what causes the risk. Furthermore, the system is customizable for different organizations. Our system has successfully demonstrated that the effect on risk assessment from the changes in the software metrics is consistent with the expected results. A full-scale industry-strength early warning system can be developed based on this prototype by including more metrics and metrics data which are collected and used in practice, and by tuning the knowledge base appropriately. Our system can be evaluated against actual results and the rules can be tuned to minimize errors in real risk predictions.

13

Another future research topic is to investigate how to scale the system to smaller software enterprises which have very limited resources for quantitative software quality management and process improvement. Acknowledgement A technical writer and Mr. Yan Sun have helped to edit this paper to improve its presentation. References Boehm, B., 1981. Software Engineering Economics. Prentice Hall, Englewood Cliffs, NJ. Boehm, B., 1989. Risk Management. IEEE Computer Society Press, Los Alamitos, CA. Boehm, B., Horowitz, E., Madachy, E., Reifer, D., Clark, B., Steece, B., Brown, A., Chulani, S., Abts, C., 2000. Software Cost Estimation in COCOMO II. Prentice Hall, Englewood Cliffs, NJ. Boehm, B., 2005. Value-Based Software Engineering: Overview and Agenda, USC-CSE-2005-504. Hudepohl, J.P., Aud, S.J., Khoshgoftaar, T.M., Allen, E.B., Mayrand, J., 1996. Integrating metrics and models for software risk assessment. In: Proc. of 1996 International Symposium on Software Reliability Engineering, pp. 93–98. Khoshgoftaar, T.M., Allen, E.B., Jones, W.D., Hudepohl, J.P., 1998. Return on investment of software quality predictions. In: Proc. of 1998 IEEE Workshop on Application-Specific Software Engineering Technology, pp. 145–150. Liu, X.F., Viswanathan, R., 1999. A WWW based software metrics environment for software process management and software product quality improvement. In: Proc. of the 23rd Annual International Computer Software and Applications, Phoenix, Arizona, pp. 301–304. MacDonell, S.G., Gray, A.R.A., 1997. Comparison of modeling techniques for software development effort prediction. In: Proc. of 1997 International Conference on Neural Information Processing and Intelligent Information Systems Proceedings, pp. 869–872. NASA Software IV & V Facility, 1995. The WWW Integrated Software Metrics Environment. Available from: . Nogueira, J.C., Luqi, Bhattacharya S., 2000. A risk assessment model for software prototyping projects. In: Proc. Int’l Workshop on Rapid System Prototyping, 2000, pp. 28–33. Rosenberg, L., Hyatt, L., 1996. Software metrics program for risk assessment. Available from: , Technical report, Software Assurance Technology Center. Vasudevan, C., 1994. An experience-based approach to software project management’. In: Proceedings of the 1994 International Conference on Tools with Artificial Intelligence, pp. 624–630. Xu, Z., Khoshgoftaar, T.M., Allen, E.B., 2002. Early operational risk assessment of software using fuzzy expert systems. In: Proceedings of the 5th Biannual World Automation Congress, vol. 13, pp. 435–442. Yuan, X., Khoshgoftaar, T.M., Allen, E.B., Ganesan, K., 2000. An application of fuzzy clustering to software quality prediction. In: Proceedings of 2000 IEEE Symposium on Application-Specific Systems and Software Engineering Technology, pp. 85–90. Zimmermann, H.J., 1991. Fuzzy Set Theory and its Applications. Kluwer Academic, Boston.