Whole.pdf

Understanding Risks in Off-The-Shelf-Based Custom Software Projects

Dana Sulistiyo Kusumo

A thesis submitted in fulfilment of the requirements for the degree of Doctor of Philosophy

School of Computer Science and Engineering The University of New South Wales Sydney, Australia

April 2013

PLEASE TYPE

THE UNIVERSITY OF NEW SOUTH WALES Thesis/Dissertation Sheet

Surname or Family name: Kusumo First name: Dana Sulistiyo

Other name/s:

Abbreviation for degree as given in the University calendar: PhD School: Computer Science and Engineering

Faculty: Engineering

Title: Understanding Risks in Off-The-Shelf-Based Custom Software Projects Abstract 350 words maximum: (PLEASE TYPE) Different software development project stakeholders (developers and acquirers) can have different perceptions of risks and how they should be mitigated. But these differences are not well understood or managed. This general issue occurs in off-the-shelf (OTS)-based custom software projects, which use and integrate OTS software in developing specialized software for an individual customer. This research provides a better understanding of risks to developers and acquirers in OTS-based custom software projects. This research consisted of five phases: 1. A systematic mapping study to identify OTS-based software acquisition processes. 2. A further mapping study to identify and classify risks related to OTS-based custom development and acquisition. 3. A survey of developers and acquirers in Indonesia to investigate in practice the characteristics of OTS-based risks that are shared by developers and acquirers. 4. Analysis of the survey data using a risk assessment framework based on stakeholder analysis. This focused on identifying differences in risk control, impact and responsibility between both stakeholders. 5. A multi-case study (four cases) to provide a deeper and more contextual analysis of these differences. Through these investigations we were able to: 1. Identify six OTS-specific acquisition processes not previously recognized. 2. Identify and classify risks of OTS-based custom projects into seventeen categories. 3. Use the survey to investigate eleven risks of the seventeen relevant risk categories. This revealed that: (a) A greater number of risks occurred more frequently in acquisition rather than in development. (b) In general the stakeholders agreed on who can best control the risks (developer). (c) There were different perceptions on who is most impacted by risks. (d) Developers were considered to bear most responsibility for risk. 4. The case study identified seventeen detailed findings related to risk control and impact for the eleven risks studied. In general most acquisition risks derive from the same concerns as development risks. However technically related risks are found less often in acquisition and project management related risks less often in development. Declaration relating to disposition of project thesis/dissertation I hereby grant to the University of New South Wales or its agents the right to archive and to make available my thesis or dissertation in whole or in part in the University libraries in all forms of media, now or here after known, subject to the provisions of the Copyright Act 1968. I retain all property rights, such as patent rights. I also retain the right to use in future works (such as articles or books) all or part of this thesis or dissertation. I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstracts International (this is applicable to doctoral theses only).

Signature

……………………………………..……………… Witness

…….……………… Date

The University recognises that there may be exceptional circumstances requiring restrictions on copying or conditions on use. Requests for restriction for a period of up to 2 years must be made in writing. Requests for a longer period of restriction may be considered in exceptional circumstances and require the approval of the Dean of Graduate Research. FOR OFFICE USE ONLY

Date of completion of requirements for Award:

THIS SHEET IS TO BE GLUED TO THE INSIDE FRONT COVER OF THE THESIS

ii

Declaration of Originality ‘I hereby declare that this submission is my own work and to the best of my knowledge it contains no materials previously published or written by another person, or substantial proportions of material which have been accepted for the award of any other degree or diploma at UNSW or any other educational institution, except where due acknowledgement is made in the thesis. Any contribution made to the research by others, with whom I have worked at UNSW or elsewhere, is explicitly acknowledged in the thesis. I also declare that the intellectual content of this thesis is the product of my own work, except to the extent that assistance from others in the project's design and conception or in style, presentation and linguistic expression is acknowledged.’

iii

List of Publications The following papers were written and published during the candidature. In each case, the candidate’s contribution was to design and also conduct the study, perform the analysis, and draft the original manuscript. Co-authoring supervisors and a NICTA researcher contributed guidance on content and editorial changes on content, linguistic expression and presentation. D. S. Kusumo, L. Zhu, M. Staples, and H. Zhang, “A Systematic Mapping Study on Off-The-Shelf-based Software Acquisition,” in ACIS 2011 Proceedings, Sydney, 2011. (Chapter 3 is based on this paper) D. S. Kusumo, M. Staples, L. Zhu, H. Zhang, and R. Jeffery, “Risks of Off-The-Shelfbased Software Acquisition and Development: A Systematic Mapping Study and A Survey,” in 16th International Conference on Evaluation & Assessment in Software Engineering, Ciudad Real, 2012. (Chapter 4 is based on this paper) D. S. Kusumo, M. Staples, L. Zhu, and R. Jeffery, “Analyzing Differences in Risk Perceptions between Developers and Acquirers in OTS-based Custom Software Projects Using Stakeholder Analysis,” in 6th International Symposium on Empirical Software Engineering and Measurement (ESEM 2012), Lund, 2012. (Chapter 5 is based on this paper)

iv

Abstract Different software development project stakeholders (developers and acquirers) can have different perceptions of risks and how they should be mitigated.

But these

differences are not well understood or managed. This general issue occurs in off-theshelf (OTS)-based custom software projects, which use and integrate OTS software in developing specialized software for an individual customer. This research provides a better understanding of risks to developers and acquirers in OTS-based custom software projects. This research consisted of five phases: 1. A systematic mapping study to identify OTS-based software acquisition processes. 2. A further mapping study to identify and classify risks related to OTS-based custom development and acquisition. 3. A survey of developers and acquirers in Indonesia to investigate in practice the characteristics of OTS-based risks that are shared by developers and acquirers. 4. Analysis of the survey data using a risk assessment framework based on stakeholder analysis. This focused on identifying differences in risk control, impact and responsibility between both stakeholders. 5. A multi-case study (four cases) to provide a deeper and more contextual analysis of these differences. Through these investigations we were able to: 1. Identify six OTS-specific acquisition processes not previously recognized. 2. Identify and classify risks of OTS-based custom projects into seventeen categories. 3. Use the survey to investigate eleven risks of the seventeen relevant risk categories. This revealed that: (a) A greater number of risks occurred more frequently in acquisition rather than in development (b) In general the stakeholders agreed on who can best control the risks (developer). (c) There were different perceptions on who is most impacted by risks. (d) Developers were considered to bear most responsibility for risk. v

2. The case study identified seventeen detailed findings related to risk control and impact for the eleven risks studied. In general most acquisition risks derive from the same concerns as development risks. However technically related risks are found less often in acquisition and project management related risks less often in development.

vi

Acknowledgements I would like to thank the following people for their support, without whose help this thesis would never have been possible. Before all, I want to thank Almighty God for His love and guidance through this journey. I owe my deepest gratitude to my supervisors Dr. Liming Zhu, Dr. Mark Staples and Professor Ross Jeffery for their continuous support, and valuable suggestions and discussions. I also thank He Zhang for his contributions to ACIS and EASE papers. Similar thanks goes to my colleagues at the Software Systems Research Group of NICTA, Paul R., Shukor, Qinghua Lu, Sherry, Liang Zhao, Emam, Yin Kia, Bassem and Suronapee, for their fellowship and support. I am particularly grateful to those survey respondents and case study participants who made this research possible. Support was given by Higher Directorate General of Higher Education - Ministry of Education and Culture of the Republic of Indonesia, UNSW, NICTA and IT Telkom. Finally, I am indebted to my beloved wife Palupi, and my wonderful daughters Kanaya and Kiara, for their continuous support, understanding and love that encouraged me to complete my study.

vii

Contents List of Figures .................................................................................................................. xi List of Tables .................................................................................................................. xii 1

Introduction ........................................................................................................... 1 1.1 Background ........................................................................................................... 1 1.2 Research Problem ................................................................................................. 2 1.3 Research Questions ............................................................................................... 3 1.4 Research Methodology ......................................................................................... 4 1.5 Research Limitations............................................................................................. 7 1.6 Research Contributions ......................................................................................... 7 1.7 Thesis Outline ....................................................................................................... 9

2

Literature review ................................................................................................. 11 2.1 Off-The-Shelf-Based Custom Software Project Processes ................................. 11 2.2 Risk Management in OTS-Based Custom Software Projects ............................. 15 2.3 Stakeholder Analysis........................................................................................... 18 2.4 Research Methods in Software Engineering Research ....................................... 21 2.5 Summary ............................................................................................................. 22

3

Off-The-Shelf-based Software Acquisition Processes ........................................ 24 3.1 Introduction ......................................................................................................... 24 3.2 Method ................................................................................................................ 25 3.3 Definition of Research Questions ....................................................................... 25 3.4 Conduct Search for Primary Studies ................................................................... 25 3.5 Screening Papers for Inclusion and Exclusion (Relevant Papers) ...................... 26 3.6 Data Extraction and Mapping of Study............................................................... 27 3.7 Discussion ........................................................................................................... 31 3.8 Threats to Validity .............................................................................................. 33 3.9 Summary ............................................................................................................. 34

4

Risks of OTS-based and Custom Software Development and Acquisition ........ 35 4.1 Introduction ......................................................................................................... 35 4.2 Research Design .................................................................................................. 36 4.3 Systematic Mapping Study on Risks of Off-The-Shelf-Based Custom Software Development and Acquisition ..................................................................................... 37 viii

4.4 Survey on Risks of Off-The-Shelf-Based Software Development and Acquisition .................................................................................................................. 48 4.5 Discussion ........................................................................................................... 53 4.6 Threats to Validity .............................................................................................. 56 4.7 Summary ............................................................................................................. 57 5

Differences in Risk Perceptions .......................................................................... 60 5.1 Introduction ......................................................................................................... 60 5.2 Research Design .................................................................................................. 61 5.3 Collected Data ..................................................................................................... 67 5.4 Results ................................................................................................................. 67 5.5 Discussion ........................................................................................................... 72 5.6 Threats to Validity .............................................................................................. 73 5.7 Summary ............................................................................................................. 74

6 Comparison of Process-related Risk Perceptions between Developers and Acquirers ......................................................................................................................... 76 6.1 Introduction ......................................................................................................... 76 6.2 Proposed Framework .......................................................................................... 77 6.3 Case Study Design .............................................................................................. 80 6.4 Result .................................................................................................................. 93 6.5 Discussion ......................................................................................................... 109 6.6 Threats to Validity ............................................................................................ 127 6.7 Summary ........................................................................................................... 128 7

Discussion and Conclusion ............................................................................... 132 7.1 Research Contributions ..................................................................................... 133 7.2 Comparison with Related Work ........................................................................ 139 7.3 Implications for Research ................................................................................. 144 7.4 Implications for Practice ................................................................................... 145 7.5 Limitations ........................................................................................................ 145

Bibliography.................................................................................................................. 147 Appendix A: Glossary................................................................................................... 157 Appendix B: Mapped papers for OTS-based software acquisition mapping study ...... 158 Appendix C: Mapped papers for risks of OTS-based and custom software development and acquisition .............................................................................................................. 165 Appendix D: The framework inputs ............................................................................. 173 ix

Appendix E: Cross-case analysis for data collection 1 and 2, excluded inputs to the framework and evaluation criteria for risk assessment ................................................. 175

x

List of Figures Figure 1.1 OTS-based custom software developer and acquirer relationship .................. 1 Figure 1.2 Overview of the research methodology ........................................................... 6 Figure 2.1 Stakeholder power/interest matrix with stakeholder involvement ................ 19 Figure 5.1 A method for analysing differences in off-the-shelf risks between the developer and acquirer used in the survey ....................................................................... 63 Figure 5.2 Relationship model of respondent, risk, risk control and impact to stakeholder ......................................................................................................................................... 63 Figure 5.3 Risk control/impact matrix, adapted from power/interest matrix [79][33][80] ......................................................................................................................................... 65 Figure 5.4 The developer respondents’ responses mapped to step 1 and 2 of the template (35 respondents) .............................................................................................................. 68 Figure 5.5 The acquirer respondents’ responses mapped to step 1 and 2 of the template (34 respondents) .............................................................................................................. 69 Figure 6.1 The proposed framework ............................................................................... 78 Figure 6.2 Risk control/impact matrix with stakeholder involvement ............................ 79

xi

List of Tables Table 1.1 A summary of five steps of research methodology........................................... 5 Table 2.1 Software acquisition and OTS-based software acquisition process models found in the literature ...................................................................................................... 14 Table 2.2 Risk checklists in the literature extended from Keil et al. [66]....................... 16 Table 3.1 Search results using the search string ............................................................. 26 Table 3.2 Refined results after paper screening .............................................................. 27 Table 3.3 Frequency of identified processes according to software acquisition classification .................................................................................................................... 29 Table 3.4 Research paper classification summarised from the paper classification by Wieringa and Heerkens [95] ........................................................................................... 30 Table 4.1 Search strings used in this study .................................................................... 38 Table 4.2 Inclusion and exclusion criteria used in this study ......................................... 39 Table 4.3 Data extraction form description..................................................................... 40 Table 4.4 Search and selection results ............................................................................ 41 Table 4.5 Mapping study classification based on type of paper ..................................... 43 Table 4.6 OTS software development risks mapped in this study .................................. 44 Table 4.7 OTS software acquisition risks mapped in this study ..................................... 46 Table 4.8 Custom software development risks mapped in this study ............................. 47 Table 4.9 Custom software acquisition risks mapped in this study ................................ 48 Table 4.10 OTS-specific project risks relevant to the developer and acquirer ............... 50 Table 4.11 Risk occurrences of OTS-based software development and acquisition ...... 52 Table 5.1 11 OTS-specific project risks relevant to the developer and acquirer respondents ...................................................................................................................... 61 Table 5.2 A template to compare risk perceptions of OTS-based custom software project .............................................................................................................................. 64 Table 5.3 Analysing of different risk perceptions ............................................................ 70 Table 5.4 Summary of comparison of risks mapped into the risk control/impact matrix between the developer and acquirer respondents (summarized from Table 5.3) .............. 70 Table 6.1 Differences between the case study framework and the method used in Chapter 5 ......................................................................................................................... 77 xii

Table 6.2 Mapping between case study questions, research questions and related propositions ..................................................................................................................... 84 Table 6.3 Case study participants.................................................................................... 86 Table 6.4 Summary of case study participants’ project and participant information .... 90 Table 6.5 Framework outputs (stakeholder involvement and risk priorities) and agreement of risk responsibilities.................................................................................... 95 Table 6.6. Comparison of evaluation criteria for risk assessment for four participants 105 Table 6.7 Example of cross-case analysis for data collection 1 and 2, excluding framework inputs and evaluation criteria for risk assessment ...................................... 107 Table 6.8 Classification of key results regarding risk control and impact .................... 113 Table 6.9 Summary of case study evidence and related propositions/literature ........... 125

xiii

1 1.1

Introduction

Background

Software development projects increasingly use off-the-shelf (OTS) products, integrating them

into

the

systems

under

development.

Software-developing

organisations can avoid building every part of their product software ‘from scratch’ by reusing technologies available from third parties [1]. Acquiring ‘OTS-based software’, i.e. software that itself uses OTS software platforms or components, can be less expensive than acquiring fully custom-developed software. Custom software development is either performed in-house or contracted software development with specific requirements for an individual customer [2][3]. OTS-based custom software is developed using and integrating OTS software in developing specialised software for an individual customer [4]. This research focuses on OTS-based custom software that is developed by being contracted or outsourced to an external software developing organisation. OTS products are described in this research as “a commercially available or open source piece of software that other software projects can reuse and integrate into their own products” [5:91]. We also classify open source software (OSS) as OTS. For the purpose of this research, an OTS-based custom software project consists of OTS-based software development and acquisition processes. OTS-based custom software project processes are detailed in Chapter 2. The relationship between OTS-based custom software acquirer and developer is depicted in Figure 1.1.

OTS software

acquire

Developer

Acquirer

develop

Figure 1.1 OTS-based custom software developer and acquirer relationship Throughout this thesis, we use software acquisition term definitions from , the IEEE Standard 1062-1998 Edition: Recommended Practice for Software Acquisition [6] to define

customer/software

acquiring

organization 1

and

software

development

organisation. An acquirer is defined to be “A person or organisation that acquires or procures a system or software product (which may be part of a system) from a supplier” [6:3] and a developer is defined to be “A person or organisation that performs development activities (including requirements analysis, design, testing through acceptance) during the software life cycle process” [6:4]. 1.2

Research Problem

In a software project, risks affect all stakeholders [7][8][9][10]. A risk is defined as “the effect of uncertainty on objectives” [11:1]. Stakeholders are defined as anyone who are affected by or can influence the systems under development [10][12][13][14]. Software project risks arise from the beginning of the software acquisition process on the software acquirer’s side [8][7]. Most literature focuses on the risks from the software developer organisation’s perspective and little attention has been given to software acquirers [7][15]. However, to manage risks effectively, it is important for all software project stakeholders to understand the risks involved [16]. In addition, there is a need to better understand OTS specific risks related to technical development processes [17] as a consequence of using OTS products. In this thesis, risks are investigated for both acquirers and developers of OTS-based custom software projects. There are differences in risk perceptions between software developers and acquirers, since they are from different organisations and have different backgrounds, experiences, needs and expectations [18]. These differences in perception could also occur because of differences between each party’s goals and structures [19]. Perspective mismatch could lead to projects not meeting their expected value, quality and schedule [20]. Differences in risk perspectives could increase areas of conflict and obstruct a shared understanding of risks [16]. Most existing studies and guidance on software acquisition do not explicitly deal with the acquisition of OTS-based software. For example, the IEEE Standard 1062-1998 Edition: Recommended Practice for Software Acquisition [6] can be applied to software acquisition process regardless of the size and complexity of the software [6]. However, this recommended practice is more applicable for fully developed software and must be tailored to other types of software acquisition [6]. In this thesis we first investigate the processes of OTS-based software acquisition before investigating the differences in 2

process-related risk perceptions. Process-related risks are defined as “risks that are related to the overall process, process elements and their interaction” [21:14]. Different stakeholders perceive risk differently [16]; therefore, there are different perceptions of stakeholder’s risk responsibility. To manage risks effectively, it is important to identify risks [16], to involve stakeholders [22][8] aiming to take account of differences in risk perceptions and to identify stakeholders’ risk responsibility [22]. Even though Keil et al. [16] noted the importance of reconciling user and project manager perceptions of IT project risks and compared the two stakeholders’ perceptions, they did not provide detailed steps for reconciling risks. A recent study has highlighted the importance of reconciling perspective mismatch in managing processes of software projects [20]. In this study, Adolph et al. [20] introduced two stages of reconciling perspectives: converging and validating. However, they [20] offered no scheme for achieving perspective agreement. Therefore, it is necessary to reconcile stakeholder perceptions for managing processes and risks in software projects [16][20]. To manage OTS-based custom software projects successfully, the overall goal of this thesis is to understand how developers and acquirers manage differences in processrelated risk perceptions in OTS-based custom software projects. To obtain a better understanding of differences in risk perceptions between stakeholders, one common solution is to compare and discuss risk perceptions [22][23]. This approach was used by Svahnberg and Wohlin [24] to compare and discuss quality attributes and structures of software architecture between different software engineers. One approach that takes account of different stakeholders in a project is stakeholder analysis [25][12][26][14][13][10]. Therefore, this research has compared the results of stakeholder analysis for developers and acquirers in OTS-based custom software projects. 1.3

Research Questions

The goal of this research is to investigate differences in process-related risks of OTSbased custom software projects from software acquirers’ and developers’ perspectives. The motivation is to give guidelines to analyse differences in perception of these risks. The thesis has three main research questions. To understand differences in process-related risk perspectives between developers and acquirers in OTS-based custom software projects, OTS-based software acquisition 3

processes were first identified and better understood. Therefore, the research question RQ1 is defined as follows. RQ1: What are the processes of OTS-based software acquisition? It is important to identify and classify acquisition and development risks in OTS-based custom software projects before investigating differences in perceptions of these risks. We have investigated the occurrences of 11 OTS process-related risks that are relevant to both developers and acquirers. The research question is as follows. RQ2: What are the risks in OTS-based custom software projects? The answers to RQ2 were expected to give us an initial understanding of differences in risk perception between stakeholders. We then investigated differences in the perceptions of these risks using stakeholder analysis [25][12][26][14][13][10]. This thesis then empirically investigated perceptions of process-related risks between developers and acquirers in OTS-based custom software projects. Therefore, research question RQ3 is defined as: RQ3: How should software developers and acquirers analyse differences in perceptions of process-related risks in OTS-based custom software projects? 1.4

Research Methodology

The research methodology is described in five steps below, and shown in Figure 1.2. The mapping among the steps, research questions and related chapters is given in Table 1.1. Step 1: Planning the research First we identified problematic areas and the importance of the research questions. This included reviewing previous research in related areas, and identifying gaps. We then formulated research problems and research objectives. Step 2: Investigating processes in OTS-based software acquisition A systematic mapping study of evidence-based software engineering was used to identify and classify OTS-based software acquisition processes.

4

Step 3: Investigating risks in OTS-based custom software projects Then we investigated risks in OTS-based custom software projects. This began by identifying, classifying and comparing risks of OTS-based software development and acquisition using a second mapping study. Then a survey was performed to empirically investigate the occurrences of the 11 OTS-specific risks in practice. The survey partially replicated a study by Li et al. [27] on risk management for OTS-based software development. Step 4:

Investigating differences in process-related risk perceptions between

developers and acquirers using a risk framework Next we compared perceptions of the 11 risks. A framework was developed to analyse the survey results about differences between these perceived risks (see Chapter 5). The framework extends a prior stakeholder analysis framework [25][12][26][14][13][10]. Step 5: Comparing differences in process-related risk perceptions between developers and acquirers In the case study (Chapter 6), the framework was modified to compare, analyse and discuss levels of risk control and impact between developers and acquirers. As shown in Table 1.1, this research consists of three research methods: systematic mapping study, survey and case study. There are two different systematic mapping studies, detailed in Chapter 3 and 4. Further, the survey results were then used in two studies (Chapter 4 and 5). Two research methods, the survey and the multi-case study were used to answer RQ2 and RQ3. Table 1.1 A summary of five steps of research methodology Step Activity

Method (study)

1

Literature Review Literature Review

2 3

Search the literature Identify problematic areas and research importance Formulate research objectives and research questions Identify and classify OTS-based software acquisition Identify, classify and compare risks of OTS-based software development and acquisition 5

Research Chapter Question RQ1 to 1-7 RQ3

Literature Review Systematic Mapping Study Systematic Mapping Study

RQ1

3

RQ2

4

4

5

Compare the occurrences of risks Survey relevant to developers and acquirers Investigate differences in process- Survey related risk perceptions between developers and acquirers using a risk framework Compare differences in process- Multi-Case Study related risk perceptions between developers and acquirers

RQ2

4

RQ3

5

RQ3

6

Step 1: Planning the research Research gap

Research problems  Research importance  Research objectives

Step 2: Investigating processes in OTS-based software acquisition

Literature review

Step 3: Investigating risks in OTS-based custom software projects

Step 4: Investigating differences in process-related risk perceptions between developers and acquirers using a risk framework

Step 5: Comparing differences in process-related risk perceptions between developers and acquirers

Figure 1.2 Overview of the research methodology 6

1.5

Research Limitations

To answer RQ2 and RQ3, this research investigated differences in the perception of 11 process-related risks in OTS-based custom software projects relevant to developers and acquirers (described in Table 4.10 and in Section 4.4.1). Although focusing on these 11 risks could limit the study’s generalisability, we believe that it can increase our understanding of how to assess risk by analysing differences in risk perceptions between developers and acquirers. The context of OTS-based software is not narrow - it is now very common in software development projects1 due to its advantages. These include rich functionality, dedicated support from OTS software vendors, and less expensive development and maintenance [28]. This research investigated risk control and impact for developers and acquirers in OTSbased custom software projects. However, it did not investigate other risk attributes (such as the probability of loss and the brief description of risks [29]). This research also only focused on risk assessment, not risk-management planning, risk resolution, or risk monitoring [30]. In our survey, we asked the survey respondents which stakeholders were affected by risks and who could influence risks. Even though we did not investigate actual risks from developers’ and acquirers’ perspectives (without discussing their responses to their stakeholders), a study of risk perception is an important approach to understand differences of actual risks between developers and acquirers. The case study investigated actual risk perceptions. 1.6

Research Contributions

In relation to the research questions, we have made the following contributions. RQ1: What are the processes of OTS-based software acquisition? We identified six OTS-based software acquisition processes from the first mapping study: decision-making to make or buy software, architectural decision-making, OTS selection, managing relationships between developers and acquirers, managing relationships between OTS adoption and acquirers’ organisations, and managing relationships between OTS vendors and developers.

These processes should be

http://www.gartner.com/it/page.jsp?id=2131115 describes the prediction of using OTS software (such as cloud, mobility and open source component) in 2012-2015 1

7

considered by an acquirer when acquiring OTS-based software, in addition to the process in the existing software acquisition process standard [6]. RQ2: What are the risks in OTS-based custom software projects? The second mapping study provides a list of 17 risk categories of OTS-based and custom-based software, not only from developers’ perspective but also from acquirers’ perspective: planning, requirements, design, integration and testing, system lifecycle, maintenance, project closure, software, OTS products, cost, environment, people, systems engineering, vendor relationships, project management, contract and legal. RQ3: How should software developers and acquirers analyse differences in perceptions of process-related risks in OTS-based custom software projects? For RQ2 and RQ3 contributions, the survey and case study show that comparing and discussing risk perceptions about risk control and impact of 11 process-related risks relevant to developers and acquirers enhance our understanding of risks in OTS-based custom software projects (described in Table 4.10 and in Section 4.4.1). Our survey found that: 1) there are similar and different perceptions about risk occurrences in OTSbased custom software projects between the developer and acquirer perspectives (RQ2 contribution); and 2) there are similarities and differences in respondents’ (developers and acquirers) perceptions about which stakeholder is affected by and can control risks (RQ3 contribution). For RQ3 contribution, the case study resulted in 17 detailed findings related to levels of risk control and impact, and proposed risk priorities (are detailed in Chapter 6, and summarised in Chapter 7). This thesis has proposed a framework to analyse differences in risk perceptions and to assign risk responsibilities between a developer and acquirer (RQ3 contribution). The framework extends a stakeholder analysis framework [25][12][26][14][13][10] and can be used to compare, analyse and discuss level of risk control and impact between a developer and acquirer. The case study results show that all participants and their stakeholders agreed on proposed risk responsibilities and risk priorities. Furthermore, our case study finds that a responsibility should also be assigned to the stakeholder who has the competence and capacity needed to discharge the responsibility [34][35].

8

From a methodological perspective, another contribution of this research is to demonstrate that stakeholder analysis can be used for each stakeholder to compare, analyse and discuss their risk perceptions. 1.7

Thesis Outline

This thesis consists of seven chapters. This chapter introduces the background, research problem, research questions, research methodology and research contributions. The remaining chapters are as follows. Chapter 2 describes a literature review of processes for OTS-based custom software projects, risk management of OTS-based custom software projects, stakeholder analysis and research methods in software engineering research. Chapter 3 describes a systematic mapping study on OTS-based software acquisition. Chapter 4 describes a systematic mapping study and a survey on risks of OTS-based custom software projects. Chapter 5 reports a survey on risk perceptions of developers and acquirers in OTSbased custom software projects. Chapter 6 discusses a multi-case study to analyse differences in the perception of process-related risks between developers and acquirers in OTS-based custom software projects. Chapter 7 concludes this thesis by listing research contributions, comparing with related work and discussing future work. Appendix A lists key terminologies used in this thesis. Appendix B details mapped papers for software acquisition and OTS-based software acquisition processes based on type of paper (empirical, experience, conceptual framework and standard). Appendix C details mapped papers for risk categories of OTS-based and custom software development and acquisition based on type of paper (empirical, experience and conceptual framework). 9

Appendix D presents the framework inputs from the case study participants and their stakeholder (Chapter 6) Appendix E presents a cross-case analysis for data collection 1 and 2, excluded inputs to the framework and evaluation criteria for risk assessment, from the case study participants (Chapter 6).

10

2

Literature review

This chapter introduces the literature on processes for OTS-based custom software projects. Then it reviews literature on risk management of OTS-based custom software projects, stakeholder analysis and research methods in software engineering research. 2.1

Off-The-Shelf-Based Custom Software Project Processes

Off-the-shelf (OTS)-based custom software project processes consist of OTS-based software development and acquisition processes. 2.1.1

OTS-Based Software Development

OTS-based software development refers to software development that integrates OTS software to develop a system. The use of off-the-shelf products in software development changes the traditional development approach [39]. While a variety of definitions of the term off-the-shelf (OTS) have been suggested, this study uses the definition by Torchiano and Morisio [5] because of its broad coverage. Off-the-shelf (OTS) is defined as “a commercially available or open source piece of software that other software projects can reuse and integrate into their own products” [5:91]. In the traditional approach, software development starts with a system requirements definition, then defines a system architecture, and concludes with implementation. In OTS-based system development, there is simultaneous definition and tradeoff among OTS selection, user requirements and design [40].

The use of OTS products in systems development

modifies not only software development [39][40][41] but also software acquisition [42][43][44]. There are additional OTS-specific processes and roles that must be included in custom software development [39][40][41]. These processes are: make vs. buy decisions, OTS selection, OTS product familiarization, learning and understanding OTS products, OTS integration and vendor interaction [39][40][41]. Li et al. [39] point out that OTS-based software development can be a part of traditional and custom-development process lifecycles (e.g. waterfall or evolutionary), combined with some new activities to manage risks. There are also new roles found in OTS-based software development literature: OTS team, a team member responsible for interactions with an OTS vendor [40] and OTS software knowledge keeper [39]. 11

The main stages of OTS-based software development are: requirements, design, coding and integration [40]. For each stage there are OTS-specific and non-OTS activities. Each stage has activities that are concurrent or iterative not only within an activity but also across activities. i.

Requirements: Requirements analysis and OTS selection are performed together. Added activities are listed in a logical order: make vs. buy decisions, requirements definition and analysis, OTS identification and selection, OTS familiarization, feasibility study, definition of high level architecture.

ii.

Design: Activities of design include following aspects: decision about architecture, OTS integration and glue code, and non-OTS design.

iii.

Coding: Coding deals with writing glue code and interface, and non-OTS coding.

iv.

Integration: Integration consists of integration and test, and target system installation and acceptance test.

Coding and integration are implementations of important decisions made in the requirements and design phases [40]. In the Constructive COTS-based application cost model (COCOTS), one of the suites of COCOMO models, there are two COCOTS sub models related to the OTS integration process [45][46]. The first model concerns OTS tailoring, which configures OTS software for specific use, for example, setting and initialising parameters [45][46]. The next model concerns OTS glue code development, which develops code to integrate OTS software into larger systems [45][46]. For OTS-based custom software, development is performed to cover remaining requirements when OTS products cannot by themselves meet the requirements [46][40]. Based on his experience in integrating OTS software into systems at The Boeing Company, Baker [47] describes four levels of coupling between the OTS software and the systems: configuration (for example, setting initialization files); integration – developing external code with minimal internal changes of OTS software; customization – vendor-supported enhancements; and development – non vendor-supported enhancement. These levels of coupling could be considered as the levels of customization in OTS-based software development. A OTS characterization framework [48] described how the degree of OTS customization can be divided into three kinds. The first is required modification [49], which has five possible levels: minimal, 12

parameterization, customization, internal revision, and extensive rework. The second kind is possible modification, which includes five possible levels: none or minimal, parameterization, customization, programming, and source code. The last kind of customisation is interface, and which also has five possible values: none, documentation, API, OO interface, and contract with protocol. 2.1.2

OTS-Based Software Acquisition

One of the challenges in software acquisition when acquiring OTS-based systems is a simultaneous process for OTS selection and system requirements [41][40]. An OTSbased custom software project uses OTS software that may not fit all defined system requirements. Therefore not only must system requirements be defined, but also the system architecture must be simultaneously based on the OTS selection. However, no comprehensive OTS-based software acquisition processes exists or is standardised. There exists a generic practice of software acquisition, IEEE Standard 1062-1998 Edition: Recommended Practice for Software Acquisition [6]. This recommended practice can be applied to software acquisition process regardless of the size and complexity of the software [6]. However, this recommended practice is more applicable for fully developed software and must be tailored to other types of software acquisition [6]. In Table 2.1, we summarize several software acquisition and OTS-based software acquisition process models found in the literature. There are four software acquisition process models:

IEEE Standard 1062-1998 Edition: Recommended Practice for

Software Acquisition [6],

ISO/IEC/IEEE 12207:2008 Standard for Systems and

Software Engineering - Software Life Cycle Processes [50], GARP (Generic Acquisition Reference Process) [51][52], and MPS.BR Model-based software acquisition [53][54]. Three OTS-based software acquisition process models are found in the literature: Commercial Off-The-Shelf (COTS) Acquisition Process (CAP) [55], COTS Software Acquisition Meta-Model (SAMM) [42], and COTS Software Component Acquisition process framework (CSCA) [43]. From Table 2.1, it can be seen that the only OTSspecific process found in existing OTS-based software acquisition process models is the OTS selection process. This has motivated us to identify OTS-based software acquisition processes using a systematic mapping study. 13

Table 2.1 Software acquisition and OTS-based software acquisition process models found in the literature Model/framework

Software acquisition processes

IEEE 1062 Recommended Practice for Software Acquisition [6]

Planning organisational strategy, implementing organisation’s process, determining the software requirements, identifying potential suppliers, preparing contract requirements, evaluating proposals and selecting the supplier, managing supplier performance, accepting the software and using the software Acquisition preparation, acquisition advertisement, supplier selection, contract agreement, agreement monitoring, acquirer acceptance, closure Refer to IEEE 1062 [6] refers to IEEE 1062 [6] and ISO/IEC/IEEE 12207 [50] Initialization Component, Execution Component and Reuse Component

√

Choice phase and implementation phase Planning, analysing and evaluating, negotiating, managing and reusing

-

ISO/IEC/IEEE 12207 [50]

GARP [51][52] MPS.BR Software Acquisition [53][54] CAP [55] SAMM [42] CSCA [43]

Generic software acquisition

OTS-based software acquisition

OTS specific processes -

√ Must be adjusted

14

-

√ √

-

-

√ OTS selection and evaluation as the basis for a make-or-buy decision √ Purchase/iterate approach to select OTS products √ Use make vs. buy decisions implicitly to select OTS products

-

-

-

2.2

Risk Management in OTS-Based Custom Software Projects

This thesis uses the definition of risk proposed by AS/NZS ISO 31000:2009 as “the effect of uncertainty on objectives” [11:1]. In software projects, we can classify project risks into generic and project-specific risks [56]. The first are common to all projects [57][30][58][59] and the latter depend on specific aspects of the project. From the perspective of technical development processes [17], risks specific to the use of OTS products may arise from OTS-based custom software project processes (as mentioned in the previous section) in OTS-based software development [39][40][41] and acquisition [60]. One of the reasons that make the processes different is a simultaneous process for OTS selection, system requirements and system design [40][41]. Boehm provides a foundation of software risk management consisting of two major steps: risk assessment and risk control [30]. Risk assessment involves three activities: risk identification, risk analysis and risk prioritization, and risk control involves three activities: risk-management planning, risk resolution and risk monitoring. As described in Chapter 1, this study focuses on risk assessment in OTS-based custom software projects. The first part of risk assessment, risk identification aims to produce all possible risks for a project [61]. There are various techniques for risk identification: checklist, decision-driver analysis, assumption analysis, decomposition [30] and brainstorming.

Risk analysis aims to provide decision support for other risk

management activities [62]. For the second part of risk assessment, Boehm provides typical techniques for risk analysis: performance models, cost models, network analysis, decision analysis and quality-factor analysis [30]. Based on a literature review, Georgieva et al. [63] classify risk assessment methods into three categories: risk assessment techniques, which mainly use neural network; type of input information, which is either questionnaire or software metrics; and software lifecycle phase. For the last part of risk assessment, risk prioritization produces a ranked ordering of risk items previously identified [30]. Risk prioritisation activities are include risk exposure, risk leverage, compound-risk reduction [30] and risk matrix [38][64]. A risk checklist provides a set of risk items that initially helps identify possible sources of risk and can be used to quickly assess risks associated with a project [65][66]. Boehm identifies 10 software risk items from project managers and user representatives [30]. Based on Boehm’s [30] risk list, Ropponen and Lyytinen [58] develop six components 15

of software development risk from project manager perspectives. Other risk checklists are based on taxonomies. For example, the Software Engineering Institute has proposed a Taxonomy of Software Development Risk [67]. This taxonomy is organised into three major classes: product engineering, development environment and program constraints. This risk taxonomy has been implemented in governmental projects [68] and industry [69] and empirically investigated in Korean projects [70]. The risk taxonomy has also been used as a baseline to develop a risk taxonomy for software maintenance [61]. Using a literature review, Barki et al. [57] surveyed project managers and identified 23 risk items grouped by five categories. Based on literature, previous experiences, local circumstances and usage of language of the organisation, Heemstra and Kusters [22] identified 36 risk items in nine categories to interview each participant of 5 projects within a Dutch governmental organisation. Based on interviews of 14 Irish software project managers, Moynihan identified 21 personal constructs relating to risks [71]. Based on three expert group panels from Hong Kong, Finland and the United States, Schmidt et al. [59] identified 53 risk items grouped by 14 risk categories for the source of risk. From the above example, there are many risk items that should be considered as relevant risk items in a software project [30][65]. Furthermore, a risk checklist must take into account all key stakeholders in a software project [65]. Table 2.2 Risk checklists in the literature extended from Keil et al. [66] Checklist Boehm [30]

Risk items 10 risk categories

Ropponen and Lyytinen [58]

Six risk items

The Software Engineering Institute [67] Webster et al. [61]

Three categories

Barki et al. [57] Heemstra and Kusters [22]

Moynihan [71]

91 risk items, three categories 23 risk items; five categories 36 risk items; nine categories

21 personal constructs 16

Sources Survey of experienced project managers and user representatives Survey (self reported data) of 83 project managers covering nearly 1100 projects Literature combined with experiences Systematic literature review Systematic literature review Literature, previous experiences, local circumstances and usage of language of the organisation Interviews with 14 Irish

Schmidt et al. [59]

relating to risks 53 risk items; 14 risk categories

software project managers International Delphi study with experienced project managers

To effectively implement risk management, Schmidt et al. argue that all stakeholders must cooperate in and be open about risk management [8]. Other studies propose collaborative risk identification and analysis, by confronting each stakeholder’s perception of potential risks, discussing risk probabilities and effects, and

finally

agreeing upon the most relevant risks and their mitigation strategies [22][23]. The Riskit method offers a broad and fully documented approach for risk management including all relevant stakeholders [38][64]. In summary, even though the aforementioned risk management methods are similar to many other risk management [38][64], these risk management methods involve all relevant stakeholders in a software project. However, these risk management methods do not provide detail on how to analyse and compare differences in risk perceptions between stakeholders. Different stakeholders have different perceptions regarding project risks [16]. Therefore, in risk management, it is important to involve all key stakeholders [22][8][23] when aiming to take account of differences in risk perceptions. It is expected that a better understanding of software risks, gained by integrating software developer and acquirer perspectives, will help to manage risk more effectively and avoid the conflict [16]. All of these differences may occur because of differences in backgrounds, experiences, needs, and expectations [18][23]; and each party’s goals and structures [19]. Perceptions are also influenced by the position, role and task of a stakeholder [23]. This study investigates differences in process-related risk perceptions in OTS-based custom software projects between the two main project stakeholders: developers and acquirers. Process-related risks are defined as “risks that are related to the overall process, process elements and their interaction” [21:14]. In OTS-based software development, some studies [72][73] map OTS-based software development issues, challenges and risks to their related stakeholders. However, these studies [72][73] have not analysed differences in risk perceptions among stakeholders. Rose [74] uses his experience to discuss the risks specific to OTS-based software development and maintenance, and describe strategies to mitigate these risks. Boehm et 17

al. [75] reports 16 risks and mitigation strategies for software engineering course projects using OTS software. The work of Li et al. [27] validated occurrences of OTSbased software development risks taken from experience and lessons learned, and is the most related to our research. In this thesis, we partially replicated Li et al. [27] by using Li’s 7 risks, and adding 4 additional risks for OTS-based custom software projects relevant to software developers and acquirers. Besides identifying risks and mitigation strategies, risk management of OTS-based software development also includes risk analysis and prioritization. One approach for risk assessment for OTS integration process risk is the COCOTS Risk Analyser [76]. This uses expert Delphi Analysis and was initially validated in nine small universitybased projects. Another approach for Kotonya and Rashid [77] uses a risk assessment of OTS-based software development modifying a form of the risk-weighting scheme described by Moynihan [71]. This uses questions to assess the extent to which the OTSbased software development processes support a defined risk management strategy. However, none of these approaches take account of different project stakeholders. 2.3

Stakeholder Analysis

In a software project, risks affect all stakeholders [8]. One approach that accounts for different stakeholder involvement in a project is stakeholder analysis [12][13][10]. Stakeholders are defined as anyone who are affected by or can influence the system under development [10][12][13][14]. Stakeholder analysis considers activities and issues such as stakeholder identification, identification of area of interest, stakeholder contribution and expectation, stakeholder influence, strategies to involve stakeholders and stakeholder responsibility [25][26]. Responsibility is defined as “a duty, held by some agent, to achieve, maintain or avoid some given state, subject to conformance with organisational, social and cultural norms” [78:519]. In software-related projects, stakeholder analysis has been used to identify stakeholders, and to identify their roles, their level of involvement [36][37][13] and their risks [9][38][10]. With regard to software project-related risks, Gotterbarn and Rogerson have developed a software development impact statement associating a potential risk that impacts particular stakeholders with each task [9]. The tasks are derived from a work breakdown structure (WBS) of the software project, and might be activities in the WBS or list of requirement specifications [9]. Woolridge et al. [10] have proposed 18

stakeholder risk assessment during the requirements engineering process, comprising stakeholder identification, analysing stakeholder influence on functional requirements, assessing the impact of functional requirements on stakeholders, and assessing and prioritizing stakeholder risks. The Riskit method [38][64] also links risks, project goals and stakeholder in order to rank risks. To manage risks effectively, it is important to

identify risks [16], to involve

stakeholders [22][8][23] and to identify stakeholder responsibility for risks [22]. A power/interest matrix, which combines the power to influence a project and the level of interest that a stakeholder has in the project, can be used to understand the result of stakeholder analysis [79][33][80]. This general idea is adapted in this thesis to analyse differences in risk perceptions between project stakeholders. There are 4 quadrants of the matrix: inform, involve, consult and collaborate (shown in Figure 2.1).

Power

High Keep satisfied Involve Low Minimal effort Inform Low

Key player Collaborate Keep informed Consult High

Interest Figure 2.1 Stakeholder power/interest matrix with stakeholder involvement ‘Inform’ describes the minimal effort of stakeholder involvement in OTS-based custom software project due to low interest and power of decision-making [33]. As this stakeholder has limited power and high interest, they should be informed and ‘consulted’ about key decisions that may affect them [33]. A stakeholder that has low interest and high power must be ‘involved’ in related project activities because they have the power to make decisions that may affect the project [33]. A stakeholder that has high interest and high power will be actively involved in and make decisions on related project activities (‘collaborate’) [33]. In the context of development of a software system, stakeholder responsibility can be modelled using a responsibility model [81][78][34][35]. A responsibility model describes responsibilities within a system under development, agents assigned to these responsibilities and resources used to discharge these responsibilities [78]. A

19

responsibility should also be assigned to the stakeholder who has the competence and capacity needed to discharge it [34][35]. Control in a project involves a controller evaluating and influencing a controllee [82][83]. In software projects, control mechanisms can be classified as: 1) formal control, “which rely on adherence to performance standards or prescribed processes”; and 2) informal controls, “which rely on social or norm-emphasizing strategies” [83:13]. There are two types of formal control: 1) outcome control, which specifies what the controllee should accomplish; and 2) behavior control, which prescribes how the controllee should achieve that outcome [82][83]. There are similarly two types of informal control: 1) clan control, “which refers to mechanisms that minimize differences between the controller and controllee’s goals by promulgating common values, beliefs, and acceptable behaviours” [83:13]; and 2) self control, where, the controllee determines their own goals and actions [82][83]. In practice, software projects combine the four control mechanisms [82][83]. In the outsourcing context, there are several factors used to determine a set of controls: stakeholders’ knowledge, consequent role expectations, perceptions of difficulty for controllee monitoring, project size, and controllee performance in the early part of a project [84][82]. In this thesis, we explored the kinds of controls used by the participants of our case study. Even though Keil et al. [16] point out the importance of reconciling user and project manager perceptions of IT project risks, and comparing stakeholders’ perceptions, their study does not provide detailed steps for reconciling risks. A recent study has also highlighted the importance of reconciling perspective mismatch in managing processes of software project [20]. In this study, Adolph et al. [20] introduce two stages of reconciling perspectives: converging and validating. However, the authors offer no detailed explanation for achieving perspective agreement. This thesis addresses this gap directly. To reconcile differences in stakeholder perspectives, we must first compare stakeholder perspectives. Svahnberg and Wohlin use Analytic Hierarchy Process (AHP) to compare and discuss quality attributes and structures of alternative software architecture decisions [24]. The comparison and discussion aims to create joint and further understanding of the different architecture candidates, and reasons for their differences. In risk management of software development, Heemstra et al. [22][23] confront each 20

team member's perception of potential risks, and discuss risk probabilities and effects to reach the most uniform team decision about risks, probabilities and effects . Their study [22][23] does not provide detailed steps to discuss different risks perceptions and to reach decisions (will be further discussed in Section 7.2). Therefore, this thesis uses this general approach and stakeholder analysis to reconcile differences in process-related risks perceptions between developers and acquirers. 2.4

Research Methods in Software Engineering Research

Easterbrook et al. [85] discuss how the selection of research methods in software engineering research depends on the availability of resources, access to subjects [86], control over variables [86] and skills of the researcher. Another important factor that influences the selection of an appropriate research method is the research question [85]. It can be important to use multiple methods to address the weakness of any one method [85]. There are different research classifications in the literature. Based on the purpose of a piece of research, Robson defines four categories: exploratory (identifying new insights and generating ideas and hypotheses), descriptive (portraying a phenomenon), explanatory (investigating an explanation of a phenomenon) and improving (trying to improve a certain aspect of the studied phenomenon). Another classification is based on research paradigm. There are three research paradigms that are commonly used in software engineering research: qualitative, quantitative and mixed [87][88][89][90]. A qualitative-based study aims to discover trends, patterns and generalization from explanatory and context information of study participants [88]. A quantitative-based study aims to find relationships among dependent and independent variables. A mixed study aims to complement the limitation of each strategy by combining them [87]. This research has used three research methods: systematic mapping study, survey, and case study. We have performed two different systematic mapping studies, detailed in Chapter 3 and 4. The survey method was used in two studies (detailed in Chapter 4 and 5). The case study is detailed in Chapter 6. The nature of this research is exploratory and does not test hypotheses. A qualitative approach was used in this research to address the complexity of software system development and human behavior as the central role in

21

the software system development [87]. The three research methods were used to complement each other and strengthen this research. 2.5

Summary

In this chapter, the literature relevant to OTS-based custom software processes and risks, and stakeholder analysis has been described and analysed. The key points are highlighted below. 

From the acquirer perspective, there is a need to identify OTS-based software acquisition processes.

Even though there is

guidance available on software

acquisition [6], this recommended practice is more applicable for fully developed software and must be tailored to other types of software acquisition [6], such as OTS-based software acquisition. 

Most of the literature focuses on risks from the software developer organisation’s perspective, and little attention has been given to software acquirers [7][15]. However, to manage risks effectively, it is important for all software project stakeholders to understand the risks [16].



Existing methods for stakeholder analysis focus on identifying stakeholders, and analysing roles of, level of involvement of, and risks related to stakeholders. However, these methods do not accommodate differences in stakeholder perceptions.

An ideal method for risk assessment of OTS-based custom software projects should identify, analyse and prioritise risks by involving key stakeholders [22][8][23]. It should also facilitate reconciling differences in risk perception between the developer and acquirer. The overall aims of this research are: 

To understand the processes of OTS-based software acquisition and the risks of OTS-based custom software project for developers and acquirers.



To identify risks [16], to involve stakeholders [22][8][23] when assessing differences in risk perceptions and to identify stakeholder responsibility for risks [22].

22

Three research methods were selected based on available resources, access to and control of subjects, and consideration of the research questions [86][85]. A systematic mapping study was used to answer a type of research question of description and classification [85]. A survey was used to better understand the normal patterns of occurrence of the phenomena by asking frequency and distribution questions [85]. A case study was used to further understand about the phenomena of differences in risk perception in the context of OTS-based custom software projects. The multiple research methods complement each other and are appropriate for their purpose.

23

3

Off-The-Shelf-based Software Acquisition Processes

This chapter describes a systematic mapping study on OTS-based software acquisition. The motivation is to compare and contrast OTS-based software acquisition and nonOTS-based software acquisition. This chapter addresses RQ1 of this thesis. This chapter is organized as follows. The first section introduces the background of this study. The following section describes the mapping study protocol including the results. In next section, we discuss the results and analyse the results based on the research question. Then, threats to validity were defined and discussed. Last section concludes this chapter. 3.1

Introduction

Software development projects increasingly use off-the-shelf (OTS) products, integrating them

into

the

systems

under

development.

Software-developing

organisations can avoid building every part of their product software ‘from scratch’ by reusing technologies available from third parties [1]. Acquiring ‘OTS-based software’, i.e. software that itself uses OTS software platforms or components, can be less expensive than acquiring fully custom-developed software. However, there is a need to better understand the OTS-based software acquisition processes. Most existing studies and guidance on software acquisition do not explicitly deal with the acquisition of OTS-based software. For example, the IEEE Standard 1062-1998 Edition: Recommended Practice for Software Acquisition can be applied to software acquisition process regardless of the size and complexity of the software [6]. However, this recommended practice is more applicable for fully developed software and must be tailored to other types of software acquisition [6]. Therefore, it motivated us to investigate the processes of OTS-based software acquisition using a systematic mapping study. In the context of empirically-based software engineering, our study has used a systematic mapping study or scoping study to map evidence about this topic [91][92]. A systematic mapping study is “a broad review of primary studies in a specific topic area that aims to identify what evidence is available on the topic” [92:vii]. The main goal of a systematic mapping study is to provide an overview of a research area and to identify 24

the nature and quantity of evidence in a research area [92]. This chapter presents the process and results of a mapping study to identify, compare and classify a set of primary studies of software acquisition and OTS-based software acquisition. 3.2

Method

Our mapping study protocol was created using guidance by Petersen et al [93], adapted by combining the last two steps of the protocol. There are five steps in the protocol: definition of research questions, conduct search, screening of papers, keywording using abstracts, and data extraction and mapping process. 3.3

Definition of Research Questions

The research question in a mapping study is a part of the mapping study protocol. Our goal is to better understand how OTS-based software acquisition processes compare to generic software acquisition processes. Our research question is: RQ. “What are the similarities and differences between software acquisition and OTSbased software acquisition from the process perspective?” 3.4

Conduct Search for Primary Studies

A mapping study is based on a systematic search of the literature using a search string. A search string can be structured using the population, intervention or outcome [92]: 1. Population: published articles including empirical studies, industry and government experiences in software acquisition domain 2. Intervention: related processes, practices and techniques in software acquisition 3. Outcomes: quantity and type of software acquisition and OTS-based software acquisition processes, practices and techniques. The search string defined in this mapping study is based on keywords from the research question. The keywords ‘software procurement’ and ‘software purchase’ are used as synonyms for ‘software acquisition’. To extend the search, we also used ‘COTS’, for commercial-off-the-shelf and ‘OTS’ for off-the-shelf combined with one of the following strings: ‘acquisition’, ‘procurement’ or ‘purchase’. The mapping study focuses on processes for OTS-based software acquisition, not processes specific to acquisition of packaged software. Nevertheless in Appendix B, we show some literature found for acquisition of packaged software and ERP systems ([24], [71], [78] and [79]) and for OSS acquisition ([41], [53], [93] and [95]). As we focused on OTS-based 25

software acquisition, the term 'CBSE' was not used because it usually refers to OTSbased software development. We also did not use the keyword “OSS” in the search strings, because the definition and classification of OTS software also includes OSS software and not vice versa [5][48]. All of the strings are combined using Boolean ORs and AND to construct the search string used in this mapping study. The search string is: ‘software acquisition’ OR ‘software procurement’ OR ‘software purchase’ OR ((cots OR ots) AND (acquisition OR procurement OR purchase)). The search results using the search string are shown in Table 3.1 describing publication resources, years of publication, advanced search methods for each of the publication and results. We used Zotero [94], a bibliography management tool, to manage literature search results. Table 3.1 Search results using the search string Publication ACM Portal IEEE Xplore Springerlink Elsevier Wiley InterScience CiteSeerX Manual using Google Scholar

3.5

Year Advanced search (search results) 1985-2010 Title, abstract, keywords 1998-2010 Title, abstract, indexing terms 1998-2010 Title, abstract 1984-2010 Title, abstract, keywords 1990-2010 All Fields, all subjects and Journals 1998-2008 Title, abstract, keywords 1997-2009 - (manual searches)

Number of result 16 172 42 36 51 86 9

Screening Papers for Inclusion and Exclusion (Relevant Papers)

Inclusion and exclusion criteria are used to ensure search query results are relevant to answer the research question. 1. Inclusion: books, papers, technical reports, reference models and standards that relate to software acquisition process. For papers, which reported the same study, the one published in a peer reviewed publication was included, or else the most recent was included. Where a paper reported several studies, each relevant study was treated separately. We focused only on primary studies. 2. Exclusion: hardware acquisition, acquisition risks, OTS-related papers that are not related to software acquisition process. 26

Table 3.2 provides the results of the search after inclusion and exclusion criteria were applied. The results are classified as relating to (generic) software acquisition or to only OTS-based software acquisition. We designed a data extraction form to collect information from each primary study. We collected extracted data in Zotero [94]. The data extraction process, which will be explained in the next section, was performed simultaneously with the final selection process. The form consisted of the following information: publisher, published year, paper classification. The paper classification is described in the end of the next section. Table 3.2 Refined results after paper screening Publication ACM IEEE Springer Elsevier Wiley InterScience CiteSeerX Manual using Google Scholar Total

3.6

Software OTS-based Total acquisition software acquisition 1 3 4 13 19 32 8 16 24 5 5 10 4 6 10 1 5 6 7 2 9 39

56

95

Data Extraction and Mapping of Study

The data extraction process in a mapping study uses a classification scheme [92]. In this chapter we used keywording of abstracts [93] as a technique to extract data. The keywording was conducted by reading abstracts and identifying keywords reflecting topics under investigation. In the case of insufficient information provided by the abstracts and keywords, we also read the introduction and conclusion of the paper. In order to identify direct evidence from primary studies, we defined in the study protocol a classification of software acquisition processes based on IEEE 1062 Recommended Practice for Software Acquisition. These processes are: Planning organisational strategy, implementing organisation’s process, determining the software requirements, identifying potential suppliers, preparing contract requirements, evaluating proposals and selecting the supplier, managing supplier performance,

27

accepting the software and using the software. This topic classification was used to map data extracted from the publications. During the keywording process, new sub-categories were identified and added into the topic classification based on screening results that could not be classified into the topic classification sub-categories but suited the population, intervention and inclusion criteria. Because the purpose of this mapping study was to identify process similarities and differences between software acquisition and OTS-based software acquisition, the mapping study separated publications into two different classes: (generic) software acquisition and OTS-based software acquisition. After finishing the keywording process, new topics were added as sub-categories, as shown in Table 3.3. Appendix B lists the mapping study results with their mapped papers. For the (generic) software acquisition classification, six new topics were added: decision-making: make vs. buy, modelling and simulation, software acquisition improvement, process life cycle, architectural decision-making and managing relationships between developers and acquirers. Seven new topics were added to the OTS-based software acquisition classification: decision-making: make vs. buy OTS products vs. use OSS, process life cycle, architectural decision-making, OTS selection, managing relationships between OTS adoption and acquirers’ organisations, managing relationships between OTS vendors and developers, and managing relationships between developers and acquirers. A process life cycle topic was also added to both of the software acquisition classifications. Three initially-proposed software acquisition topic sub-categories [6] had no matching results from the primary studies. These topics were: managing supplier performance, accepting the software and using the software. After finishing the keywording and classifying the primary studies based on software acquisition and OTS-based software acquisition topics, the frequencies of primary studies were determined, as shown in Table 3.3. Our discussion as follows is based on this table and on a thorough reading of the identified publications. To ensure broader coverage [92], we did not perform specific quality assessment on the selected studies. However, to define quality of the primary studies, we classified the mapping study results (listed in Table 3.3) as follows: empirical research, experience, 28

conceptual framework and standard. Empirical research papers are combination of evaluation and validation research papers [95]. Experience papers, in this mapping study, comprised experience and solution research papers [95]. In addition, we used a category of conceptual framework instead of opinion and philosophical paper [95]. The details of this classification can be found in Table 3.4. We added one paper classification, standard, because we found two standards for software acquisition [6][50] and one best practice for software acquisition improvement

that can be

considered to be a standard [96]. Table 3.3 Frequency of identified processes according to software acquisition classification Process

Software acquisition Em Ex

Planning organisational strategy

Cf

Std

1

OTS-based software acquisition Tot Em Ex 1

Implementing organisation’s process Determining the software requirements

1

Identifying potential suppliers

1

1

Preparing contract requirements Evaluating proposals and selecting the supplier Decision-making: make vs. buy (also vs. buy OTS products vs. use OSS for OTS-based software acquisition classification) (*)(+)

5 3

Process life cycle (*)(+) Architectural decision-making (*)(+) Modeling and simulation (*)

3

5

2

1

10

2

Software acquisition improvement (*)

2

Tot

1

1

1

1

13

14

1

1

1

1

2

2

6

5

2

7

14

3

5

8

1

1

1

5

6

2

5 6

25

31

2

OTS selection (+) Managing relationships between developers and acquirers (*)(+)

Cf

1 5

Managing relationships between OTS adoption and acquirers’ 29

3 1

1

1

1

1

1

7

organisations (+) Managing relationships between OTS vendors and developers (+)

1

1

Total 9 27 4 3 43 18 57 75 Legend: (*): new topics added to software acquisition classification; (+): new topics added to OTS-based software acquisition classification; Em: empirical research paper; Ex: experience paper; Cf: conceptual framework paper; Std: standard paper; Tot: total number of papers Table 3.4 Research paper classification summarised from the paper classification by Wieringa and Heerkens [95] Paper classification proposed by Wieringa and Heerkens [95] Paper classification Description used in this Paper type study Empirical research

Experience

Conceptual framework

Standard

Evaluation research

The paper describes the investigation or the implementation of a problem in practice. The paper uses a sound research method (either empirical or conceptual) to evaluate the investigation or the implementation.

Validation research

The paper describes the investigation of the properties of a solution proposal using a sound research method. The solution is novel and has not yet been implemented. The research methods may be experiments, simulation, prototyping, mathematical analysis etc.

Experience

The paper consist a list of lessons learned from the author's experience without a discussion of sound research methods.

Solution proposal

The paper proposes a novel solution or a significant improvement of an existing technique without a rigorous research method.

Opinion

The paper comprises the author's opinion about what is good or bad and how we should do something.

Philosophical

The paper framework.

Standard

These paper types are not included in the paper classification by Wieringa and Heerkens [95]. In this mapping study, the category of standard was used to classify standard and best practice papers found in generic software acquisition.

Best practice

30

describes

a

new

conceptual

3.7

Discussion

This section provides a discussion on the RQ “What are the similarities and differences between software acquisition and OTS-based software acquisition from process perspective?” OTS-based software acquisition is the acquisition of software that itself uses OTS software platforms or components. We identified OTS-based software acquisition processes from the literature, and compared them with a process standard for software acquisition [6]. The differences between these processes concern the acquisition of OTS products [42][55][43] and also relate to the influence of the use of OTS products on software development approaches [41][39][40]. Traditionally, software development starts with system requirements definition, then defines the system architecture, and continues with implementation. In OTS-based systems development, there is simultaneous definition and tradeoff among the OTS marketplace, system requirements, and system architecture and design [41][39][40]. Even though not all the standard software acquisition processes exist among the software acquisition processes identified from the literature (Table 3.3), both cover the life cycle [6]. The standard identifies processes for managing supplier performance, accepting the software and using the software [6], which were not found in the primary studies. However, the primary studies include implementing the organisation’s process, determining the software requirements and preparing contract requirements topics, which are not found in the software acquisition standard. Elgazzar et al. [97] discuss the planning and contracting phase of OTS-based software acquisition stressing the impact of OTS on requirements and contract structure. There are some commonalities between (generic) software acquisition and OTS-based software acquisition. One common process involves decision-making to make or buy software, but a particular condition of OTS-based software acquisition is the consideration of use of third party Commercial Off-the-Shelf (COTS) products [98][98], Enterprise Resource Planning (ERP) systems [3], and open source software (OSS) [99][100]. Another commonality concerns making architectural decisions during software acquisition [101]. These should be suited to the organisation’s needs [101], corporate governance [102], and system architecture [103]. Another common concern is 31

the nature of the working relationships between developers and acquirers, through cooperation, integration and establishing familiarity [104][105][106][102]. Two processes were found for generic software acquisition during the literature search. These processes are not referenced in the software acquisition standards: modelling and simulation, and software acquisition improvement. However, there was no explicit mention of these processes within the OTS-based software acquisition literature. These can be viewed as gaps in the literature. The main difference from generic software acquisition introduced by OTS-based software acquisition is the relation between OTS selection and determining the software requirements. As shown in Table 3.3, 31 of the total 75 publications on OTS-based software acquisition concern OTS selection. This indicates that in OTS-based software acquisition classification, OTS selection is a key process. As can be inferred from Table 3.3, OTS selection not only influences user requirements, but also architectural decision-making [103].

In regard to software requirements, OTS selection is

intertwined with software requirements definition [103] to avoid risk in OTS selection [107]. Along with determining software requirements and performing OTS selection, software architectural is also defined and adjusted iteratively to build an OTS-based system solution [103]. In OTS-based software acquisition, these three processes are intertwined because OTS product selection cannot be conducted after architectural design. This is because an architecture designed without awareness of available OTS components is unlikely to find appropriate OTS products to meet its needs [103]. There are relationships and organisational issues that must be addressed in OTS-based software acquisition. Two specific issues in OTS-based software acquisition that do not occur in generic software acquisition concern managing the relationships between (third-party) OTS vendors and the acquirers’ organisations, and managing relationships between OTS vendors and developers. In regard to organisational issues, OTS-based software acquisition must consider several characteristics of the organisation and its personnel [108]. Finally, a long-lasting and deep partnership relationships between OTS vendors and developers can provide benefits in commercial negotiations with acquirers [109].

32

In sum, OTS product selection is a significant process in OTS-based software acquisition that distinguishes it from generic software acquisition [6]. Existing software acquisition standards and processes [6] must be adjusted to accommodate the impact of third-party OTS components in software acquisition. 3.8

Threats to Validity

This section discusses construct, internal and external validity for the mapping study by following a previous mapping study [110] to analyse threats to validity. 3.8.1

Construct Validity

Construct validity is about the use of adequate definitions and measures of variables [90]. We focused on three aspects to validate constructs used in the mapping study. The first aspect was keywords used in the search strings. In the mapping study, the synonym of ‘software acquisition’, OTS and acquisition were used as the part of search strings to expand the results of the query. Secondly, the mapping study searched the primary studies using: five major digital library databases: IEEE, ACM, Springer, Science Direct and Wiley Online; additional two manual searches: CiteSeerX and Google scholar. All of these major databases covered variety and important related conferences and journals in software engineering. Therefore, it was expected to find sufficient literature to be analysed. The last aspect, we attempted to define our initial classification to be robust enough for analysis. In order to do this, the mapping study extended keywording of abstract [93] as a categorization scheme. Keywording began by reading abstracts and identifying keywords reflecting topics under investigation. When insufficient information was provided by the abstracts and keywords, the researchers also read the introduction and conclusion of the paper. 3.8.2

Internal Validity

Internal validity is concerned with whether research procedures, treatments or experiences of the research participants influence the researcher's ability to draw correct inferences from the data [90]. There were minimal threats to internal validity on the mapping study because it only used descriptive statistics to analyse mapped papers.

33

3.8.3

External Validity

External validity refers to the generalisability from this study [90].

Because the

conclusions in the mapping study were only specific for OTS-based software acquisition processes, external validity threats are not applicable to the mapping study. 3.9

Summary

OTS-based software acquisition is the acquisition of software that itself uses OTS components or products. We have presented the findings of a systematic mapping study on OTS-based software acquisition. We have suggested that for OTS-based software acquisition, changes should be considered for the existing software acquisition process standard [6]. Both generic and OTS-based software acquisition have the same overall process lifecycle. The main difference in OTS-based software acquisition is that there is a relationship between determining the software requirements and OTS selection. Almost half of publications covering OTS-based software acquisition concern OTS selection. OTS selection is also related to architectural design making [105]. Together, architecture and OTS selection criteria are defined and adjusted iteratively to build an OTS-based system solution [105]. Two additional impacts in OTS-based software acquisition concern managing relationships between OTS vendors and the acquirers’ organisations, and managing relationships between OTS vendors and developers.

34

4

Risks of OTS-based and Custom Software Development and Acquisition

This chapter describes two studies of risks of OTS-based custom software projects: a systematic mapping study and a survey. This chapter addresses RQ2 of this thesis. The remainder of this chapter is organized as follows. Section 4.1 introduces the general background. The next section discusses the research design. Then we present systematic mapping study processes and results, followed by the survey results. We discuss answers to the research question and threats to validity before concluding. 4.1

Introduction

In a software project, risks affect all stakeholders [8]. The risks of the software project arise from the beginning of software acquisition process on software acquirer’s side [8][7]. Most of the literature focuses on the risks from software developer organisation perspective and little attention has been given to software acquirers [7][15]. However, to manage risks effectively, it is important for all software project stakeholders to understand the risks [16]. In addition, there is a need to better understand OTS specific risks related to technical development processes [17] as a consequence of using OTS products. In this study we addressed risks to both developers and acquirers of OTSbased custom software. This chapter reports the process and results of a mapping study of published papers on risks in off-the-shelf and custom-based software acquisition and development. The objective of this mapping study was to identify, classify and compare risks that exist in OTS and custom-based software projects not only from the software development viewpoint but also the software acquisition viewpoint. Schmidt et al. argue that software project risks can best be managed cooperatively between software developers and acquirers [8]. Consequently, identifying risk components of OTS-based custom software is expected to give more understanding of risks of OTS-based custom software projects. In order to investigate risks of OTS-based and custom software development and acquisition in real world settings, we used the mapping study results to design a survey of OTS-based custom software project risks that may be relevant to developers and acquirers. The respondents of the survey were software developers and software acquirers of Indonesian background. 35

4.2

Research Design

This study used a systematic mapping study to identify, classify and compare risks of OTS-based and custom software development and acquisition, and then empirically investigated OTS-specific risks that may be relevant to the developer and acquirer using a structured online questionnaire of Indonesian developers and acquirers. The systematic mapping study provides broad evidence of risks of OTS-based custom software projects. In addition, the mapping study was able to provide a structure to map primary studies of risks of OTS-based and custom software acquisition, which have previously been given less attention than risks of OTS-based and custom software development [7][15]. Our survey provides evidence of the existence of these risks in practice, and complements a previous study [27] by adding information about the acquirer perspective. This survey is exploratory and does not test hypotheses. 4.2.1

Research Questions

This chapter addresses RQ2 of this thesis, and has two sub-research questions (RQ2.1 and RQ2.2).

The first sub-research question aims to identify, classify and compare

risks that exist in OTS-based and custom development not only from the development viewpoint but also the software acquisition viewpoint. This is broken down into four development context in the mapping study: OTS-based software development, OTSbased software acquisition, custom software development and custom software acquisition. RQ2.1 What risks are related to OTS-based custom software projects? RQ2.1.1 What risks are related to OTS-based software development? RQ2.1.2 What risks are related to OTS-based software acquisition? RQ2.1.3 What risks are related to custom software development? RQ2.1.4 What risks are related to custom software acquisition? The second research question aims to empirically validate the occurrence in practice of risks of OTS-based custom software projects relevant to developers and acquirers. RQ2.2 Which OTS-specific risks relevant to developers and acquirers occur frequently?

36

4.2.2

Research Methods

There are two research methods used in this study: a systematic mapping study and a survey. A systematic mapping study was used for answering RQ2.1. We then compared and selected the mapping study results to find risks that may be relevant to the developers and acquirers. A structured online questionnaire was then used to collect data about the occurrence of these risks. The survey answered RQ2.2 of this chapter. In the following sections, each method will be detailed. 4.3

Systematic Mapping Study on Risks of Off-The-Shelf-Based Custom Software Development and Acquisition

We followed a template for a Mapping Study Protocol from the Evidence-Based Software Engineering research group at Durham University [111]. Based on the template, we developed a mapping study protocol that guided this study. There are five steps of a mapping study described by the template: research questions (mentioned as background in the template), search strategy, selection criteria, data extraction and synthesis. This section describes four steps of our mapping study protocol, excluding research questions. 4.3.1

Search Strategy

A mapping study is based on a systematic search of the literature using defined search strings. The search strings can be structured using population, intervention and outcome [92]. Our search strings target the following: 1. Population: published articles including empirical studies, industry and government experiences in risks of OTS-based software development and acquisition domain 2. Intervention: risks related to process, OTS product, people, project management 3. Outcomes: quantity and type of risks in OTS-based software development and acquisition The process used automated searches on five major software-related digital libraries: IEEE Xplore, ACM Digital Library, Springer, Science Direct and Wiley Online. In this mapping study, the use of automated searching aimed to get broad scope of peerreviewed papers [112] related to the investigated topics. We used Zotero [94], a bibliography management tool, to manage literature search results.

37

We defined four groups of search strings in this mapping study derived from the research question keywords, as shown in Table 4.1. ‘COTS’, ‘OTS’, ‘component-based’ and ‘off-the-shelf’ keywords represented the type of systems under study. These keywords were combined with either ‘system’ or ‘software’ using the AND operator. These keywords covered broader software-based system development. The keyword ‘development’ summarized activities related to system development life cycle. The keyword ‘risk’ was used to focus on risk itself and not on factors constituting risk (e.g. possibility, loss, and hazard). For OTS-based software acquisition risk, the search strings also included the keywords ‘procurement’ and ‘purchase’ as synonyms of ‘acquisition’. This study is limited to risks in common to COTS, OTS, componentbased software and OSS [27]. Therefore we did not use OSS keyword in search strings to answer RQ2.1.1 and RQ2.1.2, because the definition and classification of OTS software also includes OSS software and not vice versa [5][48]. For custom software, the keywords ‘custom’, ‘contracted’ and ‘bespoke’ were used. Table 4.1 Search strings used in this study RQ RQ2.1.1

RQ2.1.2

RQ2.1.3

RQ2.1.4

Development context OTS-based software development risks OTS-based software acquisition risks Custom software development risks Custom software acquisition risks

Search strings (COTS OR OTS OR ‘component-based’ ‘OR off-theshelf’) AND (system OR software) AND development AND risk (COTS OR OTS OR ‘component-based’ OR ‘off-theshelf’) AND (system OR software) AND (acquisition OR procurement OR purchase) AND risk (custom OR contracted OR bespoke) AND (software OR system) AND development AND risk

(custom OR contracted OR bespoke) AND (software and system) AND (acquisition OR procurement OR purchase) AND risk  (‘software acquisition’ OR ‘software procurement’ OR ‘software purchase’) AND risk We used the above search strings in IEEE, ACM, Science Direct, Wiley Online, and 

Springer databases. In IEEE, ACM, Science Direct, and Wiley Online database, we used the advanced search feature, which searched for the search strings on title, abstract and keywords of the papers, but in Springer database, we could only search based on title and abstract in its advanced search feature. In Wiley Online we added a search 38

string for publication titles, ‘(computer OR computing OR computation OR information OR software OR system OR informatics OR component)’, to limit searching only to software-related publications. 4.3.2

Selection Criteria

After the search, the results were manually vetted. Inclusion and exclusion criteria were used to ensure that search query results are relevant to answer the research questions, and are shown in Table 4.2. Using these criteria, there were two steps in selecting primary studies. The initial selection was conducted by reading abstracts and keywords reflecting the development context under investigation. The classifications of the development context for this study were: OTS-based software development risks, OTSbased software acquisition risks, custom software development risks and custom software acquisition risks. In the case of insufficient information provided by the abstracts and keywords, in the final step we read the candidate primary studies thoroughly to select the relevant papers. In this final step we also conducted data extraction (discussed in the next sub-section). Table 4.2 Inclusion and exclusion criteria used in this study Inclusion criteria Peer reviewed paper (proceedings and journals)

Exclusion criteria Poster, tutorial, workshop paper, panel discussion paper, guest editors, introduction, abstract paper, book and technical report Risks that were not related to either software acquisition or development Risks that were only mentioned in the introduction or not discussed in the paper In-house software development and acquisition In ACM Digital library, resulted papers that were not published by ACM were excluded to omit the same resulted papers from IEEE, Elsevier Science Inc., Springer-Verlag and Wiley Online.

Risks that were related to either software development or acquisition Risks that were addressed and discussed in a paper Paper published from 1991 until 2011 For papers, which reported the same study, the one published in a peer reviewed publication was included, or else the most recent was included. Where a paper reported several studies, each relevant study was treated separately If a paper contains risks that could be mapped to more than one investigated topic, each identified risk was mapped to all corresponding classifications of development context. If the same papers were resulted from 39

different query strings (see Table 4.1) then only one paper was used. 4.3.3

Data Extraction

We designed a data extraction form to collect information from each selected primary study. We collected extracted data in Zotero [94]. The data extraction process was performed simultaneously with the final selection process by reading each candidate primary study thoroughly. The form is described in Table 4.3. Table 4.3 Data extraction form description Description Publisher Published year Paper classification: classified as either empirical research or experience or conceptual framework [95] Risks: risks found in the primary study Risk category: defined categories to group risks in this study 4.3.4

Synthesis

We extracted keywords from abstracts [93] using the categorization scheme. Using the data extraction results, we formed keywords to create map of the primary studies. We initially classified risks into seven categories: four categories of OTS-based software development processes [40] comprising requirement, design, coding, integration and testing, and 3 non-process categories: OTS product, people, and project management. Keywords were predefined but were extended with new keywords that were relevant to the population, intervention and inclusion criteria. During the keywording process, the category classifications were updated based on screening results that did not match the proposed category classifications but did match the inclusion criteria. 4.3.5

Mapping Study Result

Following systematic search, there were 712 papers found. The search was performed on 5 and 6 April 2011. There were 142 candidate primary studies identified in the initial selection from the search results. In the final selection, 69 papers were determined as primary studies. The search and selection results are presented in Table 4.4. One paper was removed from the category of custom software development to custom software acquisition after being read thoroughly, thus making the final selection for IEEE (5 papers) more than its initial selection (4 papers). 40

Table 4.4 Search and selection results Digital library

IEEE

ACM Springer Science Direct Wiley Online Total QR Total Init Total Fin

OTS-based OTS-based software software development acquisition paper paper QR Fin QR Init Fin (year) Init (year) 142 52 23 33 4 4 (1986 (1995 2011) -2010) 105 (1994 2010) 30 (1997 2010) 12 (1985 2011)

10

10 (1996 2009)

6

14 3

299

4

17 (1997 -2010) 10 3 (2002 -2005) 3 1 (2003) 3

2 (2006 & 2010)

1 3 0

2

Custom software acquisition paper

QR Init Fin QR Init Fin (year) (year) 242 23 6 19 4 5 (1970 (19912011) 2011) 1 38 7 0 18 2 0 (1983(19932010) 2009) 3 12 2 1 3 3 1 (2003(20032010) 2010) 0 17 0 1 2 2 2 (1979(1993 2009) & 1999) 1 1 1 0 5 3 1 (2011) (20032010) 310

56 85

Custom software development paper

10

47 33

14 8

43 9 Note: QR: query result; Init: initial selection; Fin: final selection

After updating the initial categorization scheme (see previous sub-section), we identified 17 risk categories. These categories are presented in Table 4.5: planning, requirements, design, integration and testing, system lifecycle, maintenance, project closure, software, OTS products, cost, environment, people, systems engineering, vendor relationships, project management, contract and legal. Of the initial risk categories, coding was not included in the final risk categories because no papers were mapped to this category. The 133 risks of OTS-based software development are mapped into 13 risk categories (detailed in Table 4.6). The 36 risks of OTS-based software acquisition are grouped into 14 risk categories (presented in Table 4.7). Risks of custom software development are mapped into 11 risk categories consisting of 28 risks (see 41

9

Table 4.8). Finally, there are 20 risks mapped into 11 risk categories for custom software acquisition. These results are broadly in line with previous studies [7][15] pointing out that risks related to software acquisition are less studied than software development. Appendix C lists the mapping study results with their mapped papers. The mapping results are described in the following sub-sections. To ensure broader coverage [92], we did not perform specific quality assessment on the selected studies. However, to define quality of the primary studies, we classified the mapping study results (listed in Table 4.5) into a modified paper classification scheme [95]

as follows: empirical research, experience and conceptual framework (as

discussed in Section 3.6).

42

Table 4.5 Mapping study classification based on type of paper Risk category Planning Requirement Design Integration and testing System lifecycle Maintenance Project closure Software OTS products Cost Environment People Systems engineering Vendor relationships Project management Contract Legal Total

OTS Development OTS Acquisition Custom Development Em Ex CF Total Em Ex CF Total Em Ex CF Total 2 2 4 3 3 1 24 6 5 35 2 2 3 7 5 1 1 2 4 6 1 19 3 5 27 4 4 1 1 7

1

7

5

2 2

2 2

6

2

3 5

4 13

10 1

1

3

22 1 4 4 3

1

9

2

3

5

6

2

1

3

4

1

1

3 2 2 1 1 1

3 1 2 2 1 2 1 1

1 1 1

1 2

Custom Acquisition Em Ex CF Total 1 2 1 3 7 1 1 2 1 1 1 1

1 1 2 8

1

1

1

1 1

2

2

1

1

3

2

1

6

5

1 1 20

4

1 1 1 1 1 1 1 1 1 1 70 24 39 133 10 2 24 36 21 4 3 28 12 Note: Em: Empirical research paper; Ex: Experience paper; CF: Conceptual framework paper

43

1

3

4.3.6

OTS-Based Software Development Risks

Table 4.6 lists 13 risk categories of OTS-based software development. In this mapping study, there are no risks found related to: project closure, software, project management and contract. Table 4.6 OTS software development risks mapped in this study Risk category Planning

Requirements

Design Integration and testing

Total OTS software development risks mapped papers 4 Difficulty in prioritizing features to be implemented; Introducing new OTS candidates is likely and requires replanning; Multi-OTS products from different vendors may result in complicated licensing arrangements 35 Poor requirement definition and analysis; Re-negotiate requirements with the customer, if OTS components could not satisfy all requirements; Lack of OTS-driven requirements engineering process; Unknown security of a new component; Security concern in OTS selection; Requirement changes; OTS components could not be sufficiently adapted to changing requirements; Requirements mismatches with OTS selection; Vendor’s financial stability and technology capability; Wrong component may be selected; Insufficient OTS selection estimation effort; Ability or willingness of the organisation to accept the impact of OTS requirements; Too much time spent in OTS selection; Added complexity of unused OTS features; Overly optimistic OTS package learning curve 6 Lack of early analysis of system quality; Architectural mismatches 27 Underestimation of integration effort; Poor component documentation; Too much effort needed to solve mismatch between OTS components; Difficulty when integrate with other systems; Imposed black box testing of OTS components; Late OTS component integration; Lack of component integration standard; Difficulty for sequencing OTS integration activities; Negative impact on security; Negative impact on reliability; Negative impact on performance; Difficulty to identify whether defects were inside or outside 44

System lifecycle Maintenance

OTS product

Cost Environment People Systems Engineering

the OTS components 4 Difficulty to integrate OTS security aspects into system lifecycle development; Many new non technical activities are introduced in the software engineering process 13 Difficulty to upgrade with the latest version of OTS products; OTS impact on upgrade cycles (installation, impact on system operations, and impact on integration code); OTS version upgrade during development; Negative impact of component update on systems operability; The different customer-vendor evolution cycles may result in an uncertainty about how often OTS components in a system may have to be replaced; Maintenance planning unfeasible; Faulty vendor claims; Higher long-term maintenance cost; Reduced control of future evolution of the system 22 Feature limitation; OTS product defect; Long-term product dependency; High expectation from client; Lack of vendor support; OTS product aging, OTS product obsolescence; Unknown product features; Unknown quality; Overly optimistic expectations of OTS quality attributes; Multitude of licenses; Lack of liabilities and information; Attempting and subsequently failing in the certification process; The vulnerability risks associated with hidden design assumptions in black box components; Difficulty in mapping critical quality attributes to component architectures 1 Increasing in the overall development effort and cost 4 Lack of infrastructure support; Undocumented system and business processes; Domain inadequacies 4 Lack of skilled resources; Inexperienced OTS integrator; Cultural barriers 3 Probability of success of assembled components according to planned schedule; Probability of achieving business value by using the new system; Accomplishing the development within time and budget 45

constraints 9 Vendor did not provide enough technical support/ training; Dependency on external parties (vendor lock-in); Information on the reputation and technical support ability of vendor were inadequate; Difficulty in coordinating meetings with key personnel 1 Dispute and litigation 133

Vendor relationships

Legal Total 4.3.7

OTS-Based Software Acquisition Risks

Table 4.7 reveals 14 risk categories of OTS-based software acquisition. In this mapping study, there are no risks found related to: legal, system lifecycle and design. Table 4.7 OTS software acquisition risks mapped in this study Risk category Planning Requirements

Integration and testing

Maintenance Project closure Software OTS product Cost Environment

Total OTS software acquisition risks mapped papers 3 Underestimated cost estimation; Multi-OTS products from different vendors may result in complicated licensing arrangements 7 Poor requirement definition and analysis; Lack of OTS-driven requirements engineering process; Requirements mismatches with OTS selection; Vendor’s financial stability and technology capability; Wrong component may be selected 4 Underestimation of integration effort; Poor component documentation; Too much effort needed to solve mismatch between OTS components; Poor interoperability with legacy systems 3 Maintenance planning unfeasible; Reduced control of future evolution of the system 1 The project was delivered long after schedule 2 2 1 2

People Systems Eng. Vendor relationships

1 1 5

Project Mgmt.

3

System is not developed according to the user requirements Unknown quality Increasing in the overall development effort and cost Changes in business and organisational environment that affected the project Lack of skilled resources Not addressing changing in system operation Dependency on external parties (vendor lock-in); Loss of bargaining power in vendor relationships; Parties shirking their agreed upon responsibilities; Lack of user commitment; Unwillingness to inform management of problems Poor systems of authority; 46

Unrealistic schedules and budgets; High level of system complexity to be built 1 Unexpected extension of time-and-materials based contract 36

Contract Total 4.3.8

Custom Software Development Risks

Table 4.8 describes 11 risk categories of custom software development. In this mapping study, there are no risks found related to: system lifecycle, maintenance, project closure, OTS products, cost and systems engineering. Table 4.8 Custom software development risks mapped in this study Risk category Planning Requirements Design Integration and testing Software Environment People Vendor relationships

Project Mgmt.

Contract Legal Total

Total Custom software development risks mapped papers 1 Underestimated cost estimation 7 Poor requirement definition and analysis; Requirement changes; Specification uncertainty 1 Minimal performance analysis 1 Difficulty when integrate with other systems 1 Reliability of technology 1 Changes in business and organisational environment that affected the project 2 Personnel shortfalls and personnel changes; Leave of main technician 8 Being involved with an intractable partner; Investing significant resources in a proposal that will not be accepted; Lack of communication; Political and legal environment differences; Exchange rate fluctuations; Lack of customer/user’s capability in the project; Lack of developer’s capability 4 Poor systems of authority; Lack of top management support; High level of system complexity to be built; Unrealistic schedules and budgets 1 Contract-related risks 1 Dispute and litigation 28

47

4.3.9

Custom Software Acquisition Risks

Table 4.9 lists 11 risk categories of custom software acquisition. In this mapping study, there are no risks found related to: integration and testing, maintenance, OTS products, environment, systems engineering, and project management. Table 4.9 Custom software acquisition risks mapped in this study Risk category Planning

Requirements Design System lifecycle Project closure Software Cost

Total Custom software acquisition risks mapped papers 3 Lack of risk assessment at a very early stage of the project; Selecting the lowest price from candidate developer in bidding process; Insufficient feasibility assessment 2 Poor requirement definition and analysis; Requirement creep 1 Architectural missmatch 1 Implement obstacle and stock

People Vendor relationships

Contract Legal Total

4.4

1 Insufficient software project documentation for knowledge management 1 System is not developed according to the user requirements 2 Unexpected transition and management costs; Cost escalation 1 Personnel shortfalls and personnel changes 6 Dependency on external parties (vendor lock-in); Being involved with an intractable partner; Credibility of developer’s capability’s evaluation result; Service debasement; Loss of organisational competencies 1 Unexpected costly contractual amendments 1 Dispute and litigation 20

Survey on Risks of Off-The-Shelf-Based Software Development and Acquisition

The mapping study identified risks from the literature. We complemented that with an industry survey to investigate risks in practice. We performed a structured online questionnaire survey to investigate risk occurrences. The survey focused on 11 processrelated risks of OTS-based custom software projects that may be relevant to both software developers and acquirers (as listed in Table 4.10). 48

4.4.1

Survey Design

The questionnaire targeted developers and acquirers of completed OTS-based custom software projects, using convenience sampling to identify potential respondents. Convenience sampling is reasonable to use in an exploratory study [113]. The sample population of the survey was 111 respondents which had a prior academic-industry relationship with the thesis author.

The population consisted of software acquirers

contracting OTS-based software to external software developers and software organisations developing OTS-based software for their acquirers of different organisations. Of the 111 respondents invited by e-mail, 69 (62%) completed the survey. We posted the questionnaire online using Google Docs. The respondents comprised 35 software developers and 34 software acquirers of Indonesian background. Based on their project descriptions, there were no respondents working on the same project. We excluded OTS software vendors, who build OTS software, because we wanted to focus on the software developers and acquirers. They are the two key stakeholders in software development projects using and integrating OTS software. We selected 11 process-related risks of OTS-based custom software projects from the mapping study results (Table 4.6 and 4.7) that may be relevant to developers and acquirers. The risks are shown in Table 4.10. Process-related risks are defined as “risks that are related to the overall process, process elements and their interaction” [21:14]. We grouped the 11 risks into two categories of process-related risk: ‘selection and integration’, and ‘maintenance’ risks, as in the previous study [27]. We grouped risks related to planning, requirements, and integration and testing as ‘selection and integration’ risks. OTS selection was selected as a process category because in OTSbased software development, the definition of requirements and the planning process take account of OTS selection results [41][40]. Maintenance and vendor relationships were grouped as ‘maintenance’ risks. We decided to investigate OTS-based software maintenance-related risks because the cost of OTS-based software maintenance equals or exceeds that of developing custom software [114]. Also, OTS-based software maintenance is considered to be more difficult than maintaining custom software [115]. The selected risks are derived from studies that can be classified as conceptual framework [73][116][77][118][119], experience [120] and empirical research [27][121][122][123][124]. Even though the risks included those derived from 49

conceptual framework and experience papers, we believe these risks may occur frequently in real world. As can be seen in Table 4.10, there are no studies justifying the following risks for acquirers: not adaptable to requirement changes, requirements not negotiable, upgrade unfeasible, lack of information on vendor and lack of support. Nonetheless, these risks were selected because the acquirers may be affected by and have influence on these risks [10][12][13][14]. Table 4.10 OTS-specific project risks relevant to the developer and acquirer Process

Risk ID R1

Risk Selection effort ill-estimated

R2

Not adaptable to requirement changes Requirements not negotiable

R3

Selection and integration R4

R7 R8

Complicated multi-OTS components arrangement Insufficient OTS component documents Lack of OTS-driven requirements engineering process Maintenance planning unfeasible Upgrade unfeasible

R9

Lack of information on vendor

R10

Lack of support

R5 R6

Maintenance

Developer [27][121] [122] [123] [27] [27][120] [123] [73][77] [118][124] [73][77] [27][123] [27][77] [123] [27][120] [123] [27][120] [122][123] [119]

Acquirer [116]

[73] [116] [116] [73] [116] [116]

Reduced control of future [73] evolution of the system [116] As can be seen in Table 4.10, this study includes 7 out of the 13 risks of OTS-based R11

software development seen in the study by Li et al. [27]: selection effort ill-estimated, not adaptable to requirement changes, requirements not negotiable, maintenance planning unfeasible, upgrade unfeasible, lack of information on vendor and lack of support. The other six risks in their study [27] are specific to developers, for example, difficulty to identify defect location. Respondents were asked to answer which risks in Table 4.10 occurred frequently. By ‘frequently’, we meant to investigate the level of presence of risks in real word settings 50

based on 69 completed OTS-based custom software projects. The respondents could choose one out of 6 alternative answers provided: ‘do not agree at all’, ‘hardly agree’, ‘agree somewhat’, ‘agree mostly’, ‘strongly agree’ and ‘do not know’. To analyse the results, we coded ‘do not agree at all’ to 1 and ‘strongly agree’ to 5 (see the results in Table 4.11). 4.4.2

Data Collection

The questionnaire collected data about risks of completed OTS-based custom software projects. The developers’ completed projects represented various domains: IT sector (15), banking or finance (8), public sector (8), e-commerce (3) and ERP (1). 6 different projects originated from 3 companies. All the developer respondents came from wellestablished companies, i.e.: 8 multi-national software developing companies, 1 service provider, and the others are from medium and large software developing companies. From 35 respondents, only one respondent came from a small company. The mean number of permanent software developers in the projects is 7 and median is 4. The mean number of part-time software developers involved in the projects is 4 and the median is 3. The developer respondents had positions in the completed OTS-based custom software projects as project manager (13), as developer (12), as project manager and software architect (3), as software architect (3), as project manager, software architect and developer (2), as project manager and developer (1), and as software architect and developer (1). Only 1 developer respondent did not complete his/her position. Almost all of the developer respondents held at least a bachelor’s degree in computer science. The software acquirer respondents came from telecommunications (11), government (8), banking (3), automotive (3), university (3), plantation (1), insurance (1), private investment (1), engineering consultant (1), oil and gas (1), and energy (1). From the telecommunications, automotive, and university domains, we gathered more than 1 respondent working in different completed projects. The respondents representing the acquirers had positions in the completed OTS-based custom software projects as project manager (8), system analyst (5), user representation (5), IT architect (3), developer (3), IT staff (2), domain expert (2), client team leader (1) and project steering committee (1). Only 4 acquirer respondents did not mention their positions in the projects. Almost all of the acquirer respondents held at least bachelor’s degree in computer science. 51

The average of percentage of OTS software used by the developer respondents in their developed software was 43.36% and the acquirer respondents was 67.19% (in both developers and acquirers, there were two respondents who did not answer the question of percentage of OTS software usage). Each developer respondent was asked to breakdown their OTS software usage based on the degree of OTS customization [47]. The average of the percentage breakdown of OTS software usage was: 32.60% (configuration),

24.85%

(integration),

23.79%

(customisation)

and

18.76%

(development). 4.4.3

Survey Results

Table 4.11 presents the median and mode of frequency of 11 risk occurrences relevant to the developers and acquirers in the survey. Table 4.11 shows that 9 out of the 11 risks occur frequently in software acquisition, but only 5 out of 11 risks occur frequently in software development (R4, R5, R6, R10 and R11). Three risks, complicated multi-OTS components arrangement (R4), lack of OTS-driven requirements engineering process (R6) and reduced control of future evolution of the system (R11), occur frequently in both software development and acquisition. Insufficient OTS component documents (R5) and lack of support (R10) are two risks that frequently occurred in software development but are infrequent in software acquisition. Table 4.11 Risk occurrences of OTS-based software development and acquisition Risk Development Acquisition Development Acquisition ID Valid Missing Valid Missing Median Mode Median Mode R1 34 1 33 1 2 1 3 3 R2 34 1 24 10 2.5 2 3 3 R3 32 3 32 2 2 2 3 3 R4 34 1 34 0 3 3 and 4 3 3 R5 34 1 34 0 3 3 2 2 R6 34 1 34 0 3 3 3 4 R7 34 1 24 10 2 2 3 3 R8 34 1 24 10 2.5 2 3 4 R9 33 2 34 0 2 2 3 3 R10 34 1 34 0 3 4 2 2 R11 33 2 34 0 3 2 and 3 3 4 Note: Valid: number of respondents answered the survey question on each risk; Missing: number of respondents did not answer the survey question on each risk.

52

4.5

Discussion

In this section, we discuss answers to our research questions arising from the mapping study and industry survey. 4.5.1

RQ2.1.1 What risks are related to OTS-based software development?

Of 17 final risk categories (listed in Table 4.5), OTS-based software development risks consist of 13 risk categories (described in Table 4.6). Most risks identified in OTSbased software development are OTS-specific risks. Design, legal, poor requirements definition and analysis, underestimation of integration effort, undocumented system and business processes, domain inadequacies, lack of skilled resources, cultural barriers and accomplishing the development within time and budget constraints are risks found in OTS-based software development that may occur either in custom or in-house software development [30][57][58][57]. The generic risks, which are common to all software projects, can be found in 13 risk categories of OTS-based software development, except in OTS products. 4.5.2

RQ2.1.2 What risks are related to OTS-based software acquisition?

OTS-based software acquisition risks consist of 14 out of 17 final risk categories (described in Table 4.7). Most risks found in OTS-based software acquisition have the same concern as OTS-based software development. The same risks as OTS-based software development risks are found in requirements, maintenance, OTS product, cost, people, and integration and testing. There are three risks specific only for OTS-based software acquisition: poor interoperability with legacy systems [73] (in integration and testing category); and loss of bargaining power in vendor relationships and parties shirking their agreed upon responsibilities (in vendor relationships category). OTS-based software acquisition risks also have the same risks as custom software development

(unrealistic schedules and budgets,

organisational environment that affected the project,

changes and

in

business

and

underestimated cost

estimation) and custom software acquisition risks (system is not developed according to the user requirements). These findings indicate that OTS-based software acquisition risks also have generic risks of software development project, as reported in these studies [30][57][58][57]. The generic risks can be found in 14 risk categories of OTSbased software acquisition, except in OTS products category. 53

4.5.3

RQ2.1.3 What risks are related to custom software development?

Custom software development risks include 11 risk categories (listed in Table 4.8). There is no risk specific only for custom software development. Most risks found in vendor relationships, project management, contract and legal are associated with either contracted or outsourced software development. Contracted software development needs to address these risks that do not exist in in-house software development. 4.5.4

RQ2.1.4 What risks are related to custom software acquisition?

Table 4.9 provides risks of custom software acquisition consisting of 11 risk categories (out of 17 risk categories). All risks mapped to requirements, software, cost and people are generic risks in software development projects [57][30][58][59]. The other risks identified in custom software acquisition are specific to contracted/outsourced custom software acquisition. 4.5.5

RQ2.1 What risks are related to OTS-based custom software projects?

Taken together, the results of RQ2.1.1 and RQ2.1.2 provide a checklist for identifying OTS-specific risks, which similar as micro-processes [125][17]. A micro-process describes the internal details and working of processes [125]. In addition, these results corroborate the previous studies [72][73] by providing a checklist for identifying OTSspecific risks for their frameworks of component-based systems. The mapping study results align with simultaneous processes for OTS selection, system requirements and system design [41][40] that may increase risks of OTS-based custom software projects. In four development contexts under study, there are four common risk categories: planning, requirement, people and vendor relationships. However, poor requirement definition and analysis is the only generic risk found in all development contexts under study. OTS-specific risks can be found in all risk categories. The results show that most risks found in OTS-based software acquisition have the same concern as OTS-based software development. In both custom software acquisition and development, specific risks are related to contracted or outsourced software development projects. These software project risks should be managed cooperatively between the software acquisition and development [8].

54

These findings provide a better understanding of risk components of OTS-based custom software projects. The results can serve as a checklist for risk identification [126] to help to identify more risks [66], and to assess relevancy of risks [126][66]. 4.5.6

RQ2.2 Which OTS-specific risks relevant to developers and acquirers occur frequently?

There are three points of comparison between our survey results and a previous study of OTS-based software development risk management [27]. Firstly, in this survey, six infrequently occurring risks are consistent with risks found in the previous study (these six risks are infrequent) [27]. Secondly, lack of support, which occurred frequently in this study, is inconsistent with the previous study (this risk is infrequent) [27]. We added four risks that were not investigated in the previous study [27], as follows: complicated multi-OTS components arrangement [73][77], insufficient OTS component documents [118][124], lack of OTS-driven requirements engineering process [73][77] and reduced control of future evolution of the system [119]. Thus the previous study [27] did not identify all potential risks; so there is a need to identify and analyse additional risks [127]. As can be seen in Table 4.11, among the 11 risks under study, more risks occurred more frequently in software acquisition compared to software development (nine vs. five risks out of the 11 risks). We observe three kinds of difference between developers and acquirers. Firstly, there are three risks that occurred frequently in both software development and acquisition: complicated multi-OTS components arrangement (R4), lack of OTS-driven requirements engineering process (R6) and reduced control of future evolution of the system (R11). Secondly some risks occurred frequently in software acquisition but not in software development: selection effort ill-estimated (R1), not adaptable to requirement changes (R2), requirements not negotiable (R3), maintenance planning unfeasible (R7), upgrade unfeasible (R8), and lack of information on vendor (R9). Thirdly, insufficient OTS component documents (R5) and lack of vendor technical support and training (R10) are two risks that frequently occurred in software development but interestingly occurred infrequently in software acquisition. This probably indicates that developers need more OTS software documentations, and vendor support and training for integrating OTS software, compared to acquirers. 55

4.6

Threats to Validity

In this section we discuss construct, internal and external validity for the mapping study and survey. We followed a previous mapping study [110] to analyse threats to validity of this mapping study. 4.6.1

Construct Validity

Construct validity is about the use of adequate definitions and measures of variables [90]. To validate constructs used in this mapping study, we focused on three aspects. The first aspect is the set of keywords used in the search strings. We used the keyword ‘risk’ as the part of our search strings because this keyword is specific and well-established. In the mapping study, the synonym of OTS, acquisition and custom were used as the part of search strings to expand the results of the query. Another aspect of the construct validity is we searched the primary studies using five major digital library databases: IEEE, ACM, Springer, Science Direct and Wiley Online. All of these major databases cover variety and important related conferences and journals in software engineering. Therefore, we expected to find sufficient literature to be analysed. The last aspect is we attempted to define our initial classification to be robust enough for analysis. In order to do this, we extended keywording of abstract [93] as a categorization scheme. Keywording began by reading abstracts and identifying keywords reflecting topics under investigation. When insufficient information was provided by the abstracts and keywords, we also read the introduction and conclusion of the paper. To further reduce construct validity problems we designed the survey based on a previous study [27] with little modification (specific only to this study). We used seven risks from the previous empirical study [27] and four additional risks from the literature, as shown in Table 4.10. The questionnaire used in this study was reviewed by 3 internal experts and pre-tested using a paper version by 6 industrial respondents. 4.6.2

Internal Validity

Internal validity is concerned with whether research procedures, treatments or experiences of the research participants influence the researcher's ability to draw correct inferences from the data [90]. There are minimal threats of internal validity on this mapping study because it only used descriptive statistics to analyse OTS-based risks.

56

Providing related information at the beginning of a survey is expected to give background and context information for the respondents. In addition, this information may act as an initial filter to ensure that the respondents have needed knowledge and want to share his/her experience. There were fewer than 10 respondent inquiries before and after completing the questionnaire. Furthermore, the computer science educational backgrounds of almost all of the respondents increased confidence of this study for the respondents in understanding the survey questions. 4.6.3

External Validity

External validity refers to the generalisability from this study [90]. The conclusions in this mapping study are specific to OTS-based and custom software development and acquisition risks, the external validity threats are not applicable to this study beyond the scope of OTS-based custom software projects. This survey study has a moderate sample size and was only conducted in Indonesia; therefore it may not represent risks of OTS-based software development and acquisition in general. However, there were a total of 69 respondents: 35 represent software developers and 34 represent software acquirers. The respondents in the sample vary in organisation sizes and acquirer/customer domains, which may reduce threats to external validity. 4.7

Summary

In this chapter, we have presented the design and results of two studies of the risks of OTS-based and custom software development and acquisition: a mapping study and a survey. The main result of the mapping study is the identification of risks of OTS-based and custom software development and acquisition. The secondary result of the mapping study is the classification of risks of OTS-based and custom software development and acquisition into 17 categories: planning, requirements, design, integration and testing, system lifecycle, maintenance, project closure, software, OTS products, cost, environment, people, systems engineering, vendor relationships, project management, contract and legal. Consistent with previous studies [7][15], our results show that both software acquisition is less studied than the software development. Our mapping study identifies generic and specific risks in OTS-based and custom software acquisition and development. In all four development contexts under study, 57

there are four common risk categories: planning, requirement, people and vendor relationships. However, poor requirements definition and analysis is the only generic risk found in all development contexts under study. In both custom software acquisition and development, specific risks are related to contracted or outsourced software development projects. The results also show that most risks found in OTS-based software acquisition have the same concern as OTS-based software development. Technical-related risks are found less often in acquisition and project management related risks are found less often in development. Identifying risk components of OTSbased custom software is a first step in risk management, not only for software developers but also for software acquirers. Software project risks should be managed cooperatively between the software acquirers and developers [8]. Drawing on the mapping study results, we empirically surveyed the occurrence of 11 OTS-specific risks that may be relevant to both the developers and acquirers using a structured online questionnaire. The survey is a partial replication of a study of risks of OTS-based software development [27], complemented with an investigation of the acquirer perspective. This study includes 7 out of the 13 risks of OTS-based software development seen in the study by Li et al. [27], and excludes the other six risks because they are specific to developers, for example, difficulty to identify defect location. The questionnaire collected data on risks of completed OTS-based custom software projects from 35 software developers and 34 acquirers of Indonesian background. This study partially confirms the previous study [27] because the risk of lack of support is inconsistent with the previous study [27]. This study complements the previous study [27] with new findings about the acquirer perspective. The results show that more risks occurred more frequently in software acquisition than the software development. A possible explanation is that acquirers may be more affected by or have more ability to mitigate these risks. Insufficient OTS component documents and lack of vendor technical support and training are two risks that frequently occurred in software development (but not in software acquisition).

A

possible explanation for this might be that the limited ability of developers to control these risks [16][59][128]. These findings provide a better understanding of risks in OTS-based custom software projects from the developer and acquirer perspectives.

58

Some limitations are worth noting. Although we used five major software-related digital libraries, relevant primary studies may be present in other digital libraries. The survey population is Indonesian respondents only, which may limit its external validity.

59

5

Differences in Risk Perceptions

This chapter reports on a study of risk perceptions of developers and acquirers in OTSbased custom software projects. The study used an online questionnaire-based survey and compared stakeholders’ perceptions about risk control over and exposure to 11 risks. This chapter addresses RQ3 of this thesis and has two research questions as follows, within the context of OTS-based custom software projects. RQ3.1: How are OTS-specific risks perceived by developers and acquirers? RQ3.2: How should risk responsibilities be defined between developers and acquirers? This chapter analyses the results of the survey, used in the previous chapter. The structure of this chapter is as follows. Firstly, we briefly introduce the background to the study, and describe the overall research design. Then, we describe a method to analyse the survey data about which stakeholder is affected by and can control each risk. Finally, we discuss the results and threats to validity, and presents conclusions. 5.1

Introduction

The risks of OTS-based software development and acquisition have been introduced in the previous chapter. This chapter further analyses the differences in risk perceptions of developers and acquirer using stakeholder analysis. Risks associated with a software project affect all stakeholders [9][7][8][10]. Previous studies have reported that stakeholders tend to perceive the importance of certain risks as higher than others if they cannot control the risks, and also that different stakeholders tend to identify risk being caused by other stakeholders [128][16][59]. As different stakeholders perceive risk differently [16], therefore there are different perceptions of stakeholder’s responsibility for risks. Stakeholder perceptions vary based on the individual's or organisation's background and structure, experience, needs and expectations [18][19]. To manage risks effectively, it is important to involve stakeholders [22][8], to take account of differences in risk perceptions, and to identify stakeholder responsibility for risks [22]. This chapter focuses on investigating risks relevant to developers and acquirers, and in particular differences in the perception of risks by the developers and acquirers. Another objective of this chapter is to propose risk responsibility based on stakeholder analysis. 60

One approach that accounts for different stakeholder involvement in a project is stakeholder analysis [25][12][26][14][13][10]. Stakeholder analysis is typically used for issues such as stakeholder identification, identification of area of interest, stakeholder contribution and expectation, stakeholder influence, strategy to involve stakeholder and stakeholder responsibility [25][26]. 5.2

Research Design

We performed a structured online questionnaire survey to investigate different perceptions of risks of OTS-based custom software projects (listed in Table 5.1) from developers and acquirers. The results of the survey of the previous chapter were used in this study. We focused on 11 process-related risks relevant to developers and acquirers in OTS-based custom software projects. These are the same risks as the survey in the previous chapter (detailed in Section 4.4.1 and in Table 4.10). Furthermore, we developed a method to analyse the survey data to clarify the nature of the differences between perceived risks. Table 5.1 11 OTS-specific project risks relevant to the developer and acquirer respondents Process Selection and integration

Maintenance

5.2.1

Risk ID R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11

Item Selection effort ill-estimated Not adaptable to requirement changes Requirements not negotiable Complicated multi-OTS components arrangement Insufficient OTS component documents Lack of OTS-driven requirements engineering process Maintenance planning unfeasible Upgrade unfeasible Lack of information on vendor Lack of support Reduced control of future evolution of the system

Survey Design

This chapter used the same survey as in the previous chapter, and so more details of the survey design are provided in Section 4.4.1. We performed a structured online questionnaire survey based on a definition of stakeholders [12][14][13][10] and developed a method based on stakeholder analysis [25][26] to analyse the survey data of differences in OTS-specific risks perceived by 61

software developer and acquirer

respondents. This method is illustrated in Figure 5.1. The sample population of the survey was 111 respondents which had a prior academic-industry relationship with the thesis author. The respondents comprised 35 software developers and 34 software acquirers of Indonesian background. The software developers and acquirers are in different organisations whose backgrounds, experiences, needs and expectations are different [18]. In our survey we asked about 11 process-related risks that should be relevant not only to developers but also to acquirers in OTS-based custom software projects. These are described in Table 5.1. Respondents were asked who was affected by and who could influence these risks. The respondents could choose one out of four options: ‘developer’, ‘acquirer’, ‘both’ or ‘don’t know’. 5.2.2

Method for Analysing Process-Related Risk Perspectives

The analysis method compares perceptions of the respondents about which stakeholder is affected by and can control risks. The method extends the stakeholder analysis approach [25][26], and is illustrated in Figure 5.1. The method uses a template to compare risk perceptions, as summarized in Table 5.2. As illustrated in Figure 5.1, the respondents (software developers and acquirers) completed the survey by choosing the kind of stakeholder (developer, acquirer, both or don’t know) impacted by and can control each risk. All respondents' responses were mapped to step 1 and 2 in a template, shown in Table 5.2. Step 3 will map step 1 and 2 into risk control/impact matrix. Step 4 will compare risks using the risk control/impact matrix. The detailed steps in using the template will be described in this section. Figure 5.2 models the constructs used in the method to analyse the survey.

62

Figure 5.1 A method for analysing differences in off-the-shelf risks between the developer and acquirer used in the survey

Figure 5.2 Relationship model of respondent, risk, risk control and impact to stakeholder

63

The method consists of four steps as in the template and one additional step to propose risk responsibility as follows. 

Step 1/‘Risks impacting stakeholders’ counts the number of each kind of stakeholder affected by each risk.



Step 2/‘Risks stakeholders can control’ counts the number of each kind of stakeholder who can control each risk.



Step 3/‘Mapping stakeholders from step 1 and 2 into risk control/impact matrix’, maps the number of each kind of stakeholders from step 1 and 2 into a risk control/impact matrix (Figure 5.3). This matrix is adapted from a power/interest matrix [79][33][80] (described in Section 2.3 and depicted in Figure 2.1). As this survey focused to investigate different risk perceptions between developers and acquirers, the mapping process was only conducted for two kinds of stakeholders from step 1 and 2, developer and acquirer. Therefore if a respondent answering ‘both’ to the survey question then they are included as both a developer and an acquirer. The mapping process is performed separately for developer and acquirer respondents. Table 5.2 A template to compare risk perceptions of OTS-based custom software project Stakeholder Step 1

Step 2

Step 3

Step 4

Risks impacting stakeholders Risks stakeholder can control

Developer Acquirer Both Don’t know Developer Acquirer Both Don’t know

Mapping Developer stakeholders from step 1 and 2 into risk Acquirer control/ impact matrix (see Figure 5.3) Comparing risks using the risk control/impact matrix

64

Risks R1

R2

...

R11

Risk control (Risk stakeholder can control)

High Keep satisfied (B) Low Minimal effort (A) Low

Key player (D) Keep informed (C) High

Risk impact (Risks impacting stakeholder) Figure 5.3 Risk control/impact matrix, adapted from power/interest matrix [79][33][80] In order to map the number of each kind of stakeholder (developer and acquirer) from step 1 and 2 into the risk control/impact matrix, a center point coordinate of the matrix has first to be defined for each risk under investigation. The horizontal center is half of the number of respondents answering step 1 (risk impact). The vertical center is half of the number of respondents answering step 2 (risk control). The following example demonstrates the mapping process of the developer respondents’ risk perception about themselves. For example, for the risk of not being adaptable to requirement changes (R2) in Figure 5.4, the total number of developer respondents answering step 1 is 34 and step 2 is 35. Hence, the center point coordinate of the risk control/ impact matrix is (17, 17.5). The number of the developers responding that they are impacted by (step 1) and can control (step 2) the risk is 13 and 21 respectively. The number of the developers saying that both stakeholders are impacted by (step 1) and can control (step 2) the risk is 20 and 11 respectively. Therefore, the total number of the developer respondents that perceive themselves as impacted by the risk is 33 and that can control risk is 32. So for the risk of not being adaptable to requirement changes (R2), developers’ perceptions about themselves can be mapped into high risk control and impact (Key player (D)), because they are greater than their center points. 

Step 4/‘Comparing risks using the risk control/impact matrix’ compares mapped risk control/impact matrix result between developers and acquirers. This comparison is performed separately for the developer and acquirer respondents. For example, as can be seen in Table 5.3, the risk of reduced control of future evolution of the system (R11) of the developer respondents have developers mapped into Key player (D) and acquirers mapped into Keep informed (C). These are compared (D:C).

65

The

comparison shows that developer respondents perceive themselves to have more control over the risk than their acquirers. 

Step 5/‘Identify stakeholder agreement on and reconcile differences in risk perceptions between the inputs of developer and acquirer respondents’. This step aims to propose a risk responsibility to mitigate a risk based on a stakeholder agreement about which stakeholder has high risk control. A stakeholder who has high risk control can be considered to be responsible for a risk because to mitigate a risk, high influence and power are needed to control key decisions and task implementation [31][32][33]. In addition, a responsibility should be assigned to a stakeholder who has the competence and capacity needed to discharge the responsibility [34][35]. Competence and knowledge affect the choice of control [82], while resource and capacity were needed to perform control over risks. Moreover, the consideration becomes stronger if both stakeholders also agree on high risk impact. After the responses of developer and acquirer respondents are mapped to and compared using the template (as illustrated in Figure 5.1), the completed templates are then compared to identify stakeholder agreement about risk perceptions. If there are any differences in risk perceptions between the developer and acquirer respondents, stakeholder agreement will be decided from partial agreement between both stakeholder perceptions. The following examples demonstrate partial agreement and reconciliations of different risk perceptions. The acquirer respondents in Table 5.3 perceive the risk of reduced control of future evolution of the system (R11) for developers as Keep satisfied (B) and for acquirers as Keep informed (D). To identify stakeholder agreement about which stakeholder has high risk control, the risk perceptions of the developer and acquirer respondents to themselves and their stakeholders are compared (D:C for the developer respondents and B:D for the acquirer respondents). This comparison reveals that the developer respondents perceived themselves to have higher risk control compared to their acquirers; while the acquirer respondents perceived both stakeholders to have high risk control. There is a partial agreement that the developers are perceived to best control the risk (R11) because the acquirers perceived themselves and also the developers to have high risk control. To reconcile this difference, it can be decided that the developers (as indicated in Part 2 of Table 5.3) can best control the risk of reduced control of future evolution of the system (R11). Agreement on this might be used as a rationale to 66

assign risk responsibility to the developer. The same procedure is used to reconcile different risk impact perceptions between developers (both stakeholders are most impacted by risk R11) and acquirers (the acquirer is most impacted by risk R11). Both respondents agree that the acquirer has a higher risk impact. All of the previous steps has been applied and completed for all risks in Table 5.1. 5.3

Collected Data

Data collection is described in the previous chapter (Section 4.4.2) because the survey was used in this chapter and also the previous chapter. 5.4

Results

This section presents results of the survey organized by the comparison method. Figure 5.4 and 5.5 present the questionnaire results mapped to step 1 and 2 of the method template (see Table 5.2). The results are organized as comparisons of the risk control/impact matrix between kinds of stakeholder (developers and acquirers) separately for each survey respondents (the developer and acquirer respondents) in Part 1 of Table 5.3.

67

Step 1: Risks impacting stakeholder 30 24

25 20 20 15

19

17

16

14

13

12

10

15 15

10 8

7

6 4

5 0

2

1 1

16 12

11

10

19

18

17

1

2 2

1

3

9

8

7 5

4

4

4

2

13

9 3 1

0

0 R1

R2

R3

R4

R5

Developer

R6

R7

Acquirer

Both

R8

R9

R10

R11

Unknown

Step 2: Stakeholder can control risks 35 30 30 25

25

23

21

21

20

27 24

20

18

17

15

11

10 5

7

13 13 6

3

5 0

3

2

0

10

9

88 4

4

0 1

1

3 2

343

R7

R8

2

10 7

54 1

43 0

0 R1

R2

R3

R4

Developer

R5

R6

Acquirer

Both

R9

R10

R11

Unknown

Figure 5.4 The developer respondents’ responses mapped to step 1 and 2 of the template (35 respondents)

68

Step 1: Risks impacting stakeholder 25 20 20 16 14

15

14 12

12

14

10 10

8

14 13 12

9

8

5

2

10 8

11 7

6 6

5

4

3

2

11

10 10 8 888

6

2

1

11

8

6

5

14 12

2

2

2

R10

R11

0 R1

R2

R3

R4

R5

Developer

R6

R7

Acquirer

Both

R8

R9

Unknown

Step 2: Stakeholder can control risks 18

17

16

16

16 14 12 10

13 11

10 9

9

8

12

13 13

12 12 12

8

6 2

11

13 11

11

3

6 2

3

5

4 2

1212 8

6 3

11

12

8

6 4

13

2

5

6

6 4

2

2

2 0 R1

R2

R3

R4

Developer

R5

R6

Acquirer

R7 Both

R8

R9

R10

R11

Unknown

Figure 5.5 The acquirer respondents’ responses mapped to step 1 and 2 of the template (34 respondents) Part 1 of Table 5.3 shows the results of step 4 in the method template (Table 5.2). Part 2 of Table 5.3 shows stakeholder agreement on risk control and impact, and risk responsibility following step 5 in the developed method. Table 5.4 summarizes patterns of the comparisons from Part 1 of Table 5.3. It can be seen from Table 5.3 and 5.4, almost all the developer respondents perceive themselves to have higher risk control compared to their acquirers. There are 3 kinds of differences. The first is D:D, which in Table 5.3 is the risk of requirements not 69

negotiable (R3), where the developer respondents perceive themselves and their acquirers to have the same high risk control and impact. For the second kind, developers perceive themselves to have higher risk control compared to their acquirers, but perceive themselves to have equal risk impact as their acquirers (D:C in Table 5.3 and 5.4). Risks in this group are selection effort ill-estimated (R1), not adaptable to requirement changes (R2), complicated multi-OTS components arrangement (R4), and maintenance planning unfeasible (R7) and reduced control of future evolution of the system (R11). For the last kind, the developer respondents perceive themselves to have higher risk control and impact compared to their acquirers (D:A in Table 5.3 and 5.4), for the risks:

insufficient OTS component documents (R5), lack of OTS-driven

requirements engineering process (R6), upgrade unfeasible (R8), lack of information on vendor (R9) and lack of support (R10). Table 5.3 Analysing of different risk perceptions Part 1 Comparison of risk mapped into the risk control/impact matrix (developer:acquirer) from respondent perspectives Risk (developer:acquirer) Respondents R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 Developer D:C D:C D:D D:C D:A D:A D:C D:A D:A D:A D:C Acquirer D:D D:C D:D D:D D:C D:D D:C D:C D:C B:C B:D Part 2 Agreement of responses of stakeholders on risk control and impact, and risk responsibility Stakeholder agreement on risk control Stakeholder agreement on risk impact Risk responsibility

R1

R2

R3

R4

R5

R6

R7

R8

R9

R10

R11

Dev

Dev

Both

Dev

Dev

Dev

Dev

Dev

Dev

Dev

Dev

Both

Both

Both

Both

Dev

Dev

Both

Dev

Dev

-

Acq

Dev

Dev

Both

Dev

Dev

Dev

Dev

Dev

Dev

Dev

Dev

Table 5.4 Summary of comparison of risks mapped into the risk control/impact matrix between the developer and acquirer respondents (summarized from Table 5.3) Patterns of comparison of risk mapped into the risk control/impact matrix (developer:acquirer) from respondent perspectives D:D D:C D:A B:C B:D Respondent Developer 1 5 5 perspective Acquirer 4 5 1 1

70

For acquirers, there are 4 kinds of differences. For the first kind, acquirer respondents perceive the following risks to be under high control and have high impact for themselves and their developers (D:D): selection effort ill-estimated (R1), requirements not negotiable (R3), complicated multi-OTS components arrangement (R4) and lack of OTS-driven requirements (R6). For the second kind, acquirers perceive their developers to have higher risk control, and perceive themselves and their developer to have higher risk impact (D:C) for the following risks: not adaptable to requirement changes (R2), insufficient OTS component documents (R5), maintenance planning unfeasible (R7), upgrade unfeasible (R8) and lack of information on vendor (R9). For the third kind, acquirers perceive themselves to have higher risk impact but lower risk control compared to their developers (B:C), for the risk of lack of support (R11). Finally, acquirers perceive themselves to have higher risk impact than, but equal risk control to their developer (B:D) for the risk of reduced control of future evolution of the system (R11). Part 2 of Table 5.3 shows agreements of responses of stakeholders on risk control and impact, including reconciliation results of different risk perceptions between the developer and acquirer respondents on the risks of R1, R4, R5, R6, R8, R9, and R11. With regard to risks R1, R4, R6 and R11, the developer respondents perceived themselves to have higher risk control than their acquirers, and the acquirer respondents perceived both stakeholders to have high risk control and impact (for R1, R4 and R6) or to have high risk control (for R11). The differences between the stakeholders can be reconciled by agreeing that the developer has a high level of risk control. For the risks R5, R6, R8 and R9, the developer respondents perceived only themselves as most impacted by these risks (D:A in developer respondents row in Part 1 of Table 5.3), and the acquirer respondents perceived both stakeholders as most impacted by these risks (developer and acquirer of the mapped risk control/impact matrix is either C or D in Part 1 Table 5.3). To reconcile these differences, it could be decided that the developers are highly impacted by these risks (presented in Part 2 of Table 5.3). For risk R11, the responses of both stakeholders agree that the acquirer is most impacted by the risk (reduced control of future evolution of the system). This agreement is decided by reconciling different risk perceptions between the developer (both stakeholders are most impacted by risk R11) and acquirer (the acquirer is most impacted by risk R11). For the risk of lack of support (R10), there was disagreement about who is most impacted by 71

the risk (developers perceive themselves to have higher risk control and impact compared to their acquirer (D:A), while acquirers perceive themselves to have higher risk impact compared to their developers (B:C). 5.5 5.5.1

Discussion RQ3.1: How are OTS-specific risks perceived by the developer and acquirer?

Although a previous study investigated differences in perceptions of IT project risks between users and project managers [16], these risks were not specific to OTS-based custom software projects. Our study focused on OTS-specific project risks relevant to the developer and acquirer, as listed in Table 5.1. In this section, we compared risk perceptions in a risk control/impact matrix. This comparison focused on perceptions of risk control and impact by developers and acquirers. As indicated in Part 2 of Table 5.3, almost all stakeholders agree that the developer can best control risks. There is only one risk, R3 (requirement not negotiable), where both stakeholders each claim to best control this risk. The developer respondents have two different kinds of perceptions about which stakeholder is most impacted by risks. The first, developers perceive that both stakeholders are most impacted by risks (as shown in Part 1 of Table 5.3, for the risks R1, R2, R3, R4, R7 and R11). The second kind, developers perceive themselves to be most impacted by risks (as shown in Part 1 of Table 5.3, for the risks R5, R6, R8, R9 and R10). The acquirer respondents perceive both stakeholders to be most impacted by risks, except for their developers in R10 and R11. In regard to risk of lack of support (R10) and reduced control of future evolution of the system (R11), acquirers perceive themselves to have higher risk impact than their developers. Comparing the developer and acquirer respondents’ perspectives about which stakeholders are most impacted by risks, it could be concluded that for the risks of R1, R2, R4 and R7, both stakeholders agree that they are both highly impacted by the risks. For the risks of R5, R6, R8 and R9, the developers are perceived to be most impacted by these risks (see Part 2 of Table 5.3). For the risk of lack of support (R10), there was 72

disagreement about who is most impacted by the risk. With regard to the risk of reduced control of future evolution of the system (R11), both stakeholders agree that the acquirer is most impacted. The comparison method used in this chapter provided a systematic and practical approach to analyse differences in the perceptions of risks between developers and acquirers. 5.5.2

RQ3.2: How to define risk responsibility between the developers and acquirers?

As can be seen in Table 5.3, for only 3 out of 11 risks investigated, (R2, R3, R7) do both the developers and acquirers have the same perceptions. For the other risks, it is interesting to analyse the differences. Understanding these differences in perceptions about which stakeholder has high risk control is helpful to propose risk responsibility. High influence and power is needed to control key decisions and task implementation [31][32][33] to mitigate risks. This study provided a structured method to compare risk control and impact to identify stakeholder agreement on and reconcile differences in risk perceptions between developer and acquirer respondents. Ultimately, the outputs of the method are proposals for the assignment of risk responsibilities, based on stakeholder agreement on which stakeholder has high risk control. This is consistent with responsibility models [81][78][34][35], which propose responsibility belongs to stakeholders who have the competence and capacity needed to discharge the responsibility [34][35]. Knowing the responsibility could subsequently lead to the identification of roles [81][78] [34][35] to mitigate risks. As shown in Part 2 of Table 5.3, for all risks except requirements not negotiable (R3), the developer would be considered to be responsible for the risks. The developer should inform, consult and involve the acquirer because the acquirer is also impacted by the risks [33]. For the risk of requirements not negotiable (R3), both stakeholders should collaborate to discharge the responsibility [34] of risk mitigation. 5.6

Threats to Validity

As this study used the same survey as the previous chapter, validity threats of this study are largely the same as in the previous chapter. However, instead of using questions adapted from Li et al. study [27], to prevent construct validity problems, we derived the 73

questionnaire questions from a stakeholder definition, which is anyone who are affected by or can influence the system under development

[12][14][13][10]. The use of

stakeholder definition as the questionnaire questions to ensure adequate definitions and measures of variables [90]. 5.7

Summary

This study attempted to analyse differences in shared risks [8] of OTS-based custom software projects by comparing perceptions of risk control and impact of two main stakeholders, the developers and acquirers. We performed an online questionnaire-based survey about OTS-based custom software project risks on Indonesian software developers and acquirers. In the survey, we asked who was affected by and who could influence 11 risks of OTS-based custom software projects relevant to both developers and acquirers. To analyse the survey results, we developed and applied a method for analysing differences in perceptions of OTS-specific risks based on stakeholder analysis [25][12] [26][14] [13][10]. The findings show some differences in risk perception. Both stakeholders can control, and are most impacted by, risks about requirements negotiation. Developer respondents perceived themselves to best control risks, but perceived either themselves or both stakeholders to be most impacted by risks. Acquirer respondents agreed that their developers can best control risks, but perceived both stakeholders as most impacted by risks, except for risks of R10 and R11. For the risk of lack of vendor technical support and training (R10), there was disagreement about who is most impacted by the risk (usually each stakeholder reported themselves). With regard to the risk of reduced control of future evolution of the system (R11), both stakeholders agreed that the acquirer is most impacted. The comparison method was developed to analyse the survey results. We used the method to propose a default position about which stakeholder should be responsible for risks based on stakeholders’ agreement about which stakeholders have high control and power over each risk [31][32][33]. The results show that for all risks (except requirements not negotiable) the responsibility can be proposed to be the developers’. With regard to requirements not negotiable, we proposed both stakeholders had responsibility of risk. 74

Even though we did not investigate the developer and acquirer’s actual perspectives on risks, a study of risk perception is an important approach to understand differences of actual risks between the developers and acquirers. The comparison method analysed respondents’ responses about their risk control and impact, without discussing their responses to their stakeholders. Overall, the comparison method in this study added substantially to our understanding to provide a method-based consideration to analyse different risk perceptions [16].

75

6

Comparison of Process-related Risk Perceptions between Developers and Acquirers

This chapter discusses a multi-case study to analyse differences in the perception of process-related risks between developers and acquirers in OTS-based custom software projects. The aim of this chapter is to develop further understanding about stakeholders’ perceptions of their level of risk control and impact. This chapter complements Chapter 5 in addressing RQ3 of this thesis. This study has two main research questions as follows. RQ3.3: How are OTS-specific risks perceived by developers and acquirers? RQ3.4: How should risk responsibilities be assigned between developers and acquirers for these risks? The first section introduces this chapter. The next section describes a proposed framework, which was modified from a method in the previous chapter, to collect, compare and analyse the case study data. Then, the case study design is detailed. Section 6.4 presents results and findings of this study. Section 6.5 discusses answers to the research questions. Finally we assess threats to validity and conclude. 6.1

Introduction

To manage risks effectively, it is important to identify the risks involved [16], to involve stakeholders [22][8] aiming to take account of differences in risk perceptions and to identify stakeholder responsibility for risks [22]. Even though Keil et al. [16] pointed out the importance of reconciling user and project manager perceptions of IT project risks and compared two stakeholders’ perceptions, their study did not provide detailed steps for reconciling risks. A recent study has highlighted the importance of reconciling perspective mismatch in managing software processes [20]. In their study, Adolph et al. [20] introduced two stages for reconciling perspectives: converging and validating. In this thesis, we analyse differences in the perception of process-related risks between the developers and acquirers in OTS-based custom software projects using a multi-case study. We investigated differences in the perception of 11 process-related risks. Six of these 11 risks relate to the process of OTS selection and integration: selection effort ill-estimated, 76

not adaptable to requirement changes, requirements not negotiable, complicated multiOTS components arrangement, insufficient OTS component documents and lack of OTSdriven requirements engineering process. Five are maintenance-related risks: maintenance planning unfeasible, upgrade unfeasible, lack of information on vendor, lack of support and reduced control of future evolution of the system. Section 4.4.1 provides a more detailed explanation of these 11 risks. Although focusing on the 11 risks could limit the study’s generalisability, we believe that it can help increase our understanding to better assess risks through detailed analysing of differences in risk perceptions between developers and acquirers. Based on the above research questions and access to resources [85][86], which in this study were case study participants, a case study was selected as the research method. Using a case study was expected to provide a deeper understanding of the phenomena under study [86][85][129] by involving stakeholders in risk management and analysing differences in risk perceptions occurring in real contexts. 6.2

Proposed Framework

This study used a framework, based on the method developed in the previous chapter, to collect, analyse, compare and discuss case study data about perceived risks. In the previous chapter, we analysed survey data of process-related risk perceptions about which stakeholder is affected by and can control risks. This study extended that method to make a proposed framework that can be used by developers and acquires to discuss disagreement regarding risk responsibilities. Table 6.1 describe the three changes made to the method used in Chapter 5 to create the framework used in this chapter. Table 6.1 Differences between the case study framework and the method used in Chapter 5 Difference Input

Decision on responsibility

The Chapter 5 method Who was most impacted by these risks and who could best control these risks? (developer/acquirer/both/ don’t know) risk Risk responsibility is defined based on agreement of respondents’ inputs on which stakeholder has high risk control 77

The Chapter 6 framework How was your level of risk control and risk impact? (low/high)

Risk responsibility is defined based on agreement of developer and acquirer on which stakeholder has higher risk control

Decision on The previous method does Dialogue, deliberation and disagreement about not provide a solution communication in the framework if risk responsibility developer and acquirer disagree with a proposed risk responsibility This framework analyses differences in perceived risks that should be relevant to both developers and acquirers of OTS-based custom software. The risks are described in Section 4.4.1 and in Table 4.10. The framework extends the stakeholder analysis approach [10][12][13][25][26], and is illustrated in Figure 6.1. The framework compares level of stakeholder perceptions about risk control and impact (low/high), which are both framework inputs. Framework outputs are stakeholder involvement and risk priority (for each stakeholder). Responsibility for a risk is proposed to belong to a stakeholder who has higher risk control. These outputs and risk responsibilities are then raised with the developer and acquirer for their agreement. The framework is not a risk identification method; stakeholders must provide a list of risks to be analysed using the framework.

Figure 6.1 The proposed framework The framework consists of five steps: 1. Each stakeholder provides inputs to the framework, which are levels of risk control and impact (low/high) for each risk being analysed. 78

2.

The framework inputs are then mapped into a risk impact/control matrix (Figure 6.2), as in the previous chapter [79][33][80] (see Figure 5.3). The mapping process output is a value for each risk: inform (A), involve (B), consult (C) and collaborate (D). Each value indicates the kind of stakeholder involvement. The matrix uses an ordinal scale. Instead of using risk-exposure quantity [30] or multiplication on ordinal scale [38], the framework prioritises risk using available information [38] by comparing the values of two quadrants. Cells above or right have higher risk priority. Quadrant A has low risk priority. Quadrant B and C have medium risk priority, and quadrant D has high risk priority. However, risks in quadrant B and C cannot be compared [69][130]. High Involve (B) Risk control (Risk stakeholder can control) Low Inform (A) Low

Collaborate (D) Consult (C) High

Risk impact (risks impacting stakeholder) Figure 6.2 Risk control/impact matrix with stakeholder involvement 3. The next step is to compare the outputs of the mapping process above between a developer and acquirer. The comparison outputs indicate the kind of stakeholder involvement for each stakeholder and a risk priority. Responsibility for each risk is proposed to belong to the stakeholder who has higher risk control. This is because high influence and power over risk is needed to control key decisions and task implementation [31][32][33]. If both stakeholders have same level of risk control, the responsibility is proposed to belong to both stakeholders. All of the framework’s outputs and proposed risk responsibilities are raised for discussion between both stakeholders for agreement. 4. Stakeholders jointly determine if they agree on the proposed risk responsibilities. The comparison in the framework facilitates discussion between stakeholders by obtaining their opinions and further understanding of their perceptions [24]. If both stakeholders agree then the risk responsibilities are assigned to a stakeholder who has higher risk control and the framework terminates. Otherwise it continues to the next step. 5. This last optional step is dialogue, deliberation and communication to manage risk. If both stakeholders do not agree with the previous step, they can discuss the 79

proposed risk responsibility by considering previous results. The discussion process should skip areas of agreement, which are stakeholder-owned perspectives (inputs and outputs), and focus on points of difference, which are proposed risk responsibilities. Although the framework was designed to be used by a developer and acquirer, to avoid the effects of learning to use the framework [131], case study participants were only asked to complete the framework inputs and to provide agreement on the framework outputs and proposed risk responsibilities (these are detailed in following sections). To increase validity, a new proposed framework should be evaluated [131] and then improved [129] using retrospective data [132]. The framework was initially evaluated using a set of evaluation criteria for risk assessment [133][134] (the results in Section 6.4.2 and discussed in Section 6.5.3), and using the results and findings of this case study. Finally, the framework was improved [129] by comparing [135] existing risk management practices with the framework, and making use suggestions from all participants. 6.3

Case Study Design

The selection of case studies as a research method can depend on the research questions and also access to resources [86][85]. Yin defines case study as “an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident ” [86:18]. Case studies can also be suitable when researchers have little control over events [85][86]. We selected case studies as a research method for all of these reasons. The case study followed existing guidelines [86][131][132][129]. The structure of the case study is outlined below. 1. Case study design The case study design comprises: a. Objectives of the case study b. Case study questions to guide data collection c. Theoretical prepositions as term of reference to conduct and review the study d. The case and unit of analysis selection e. Methods to collect data

80

2. Preparation for data collection: Procedures and protocols for data collection must be defined. 3. Collecting evidence 4. Analysis of collected data 5. Reporting The following section describes the case study according to this structure. 6.3.1

Objective of the Case Study

The objective of the case study was to develop a deeper understanding of the phenomena under study [86][85][129]: stakeholder perspectives in risk management in a real world context. 6.3.2

Questions of the Case Study

There were two stages of data collection in this study, as described in Section 6.3.6. The first stage was a questionnaire-based survey to identify existing stakeholder practices in risk management and to analyse differences in risk perceptions. This provided input data to the framework. The second stage was a structured interview to evaluate and analyse differences in risk perceptions. The questions for each stage are presented below. 1. First stage There were three parts of the questionnaire. Q1.1 (Part 1 of the questionnaire) The questions collected project information about project members, OTS software used in the developed system, type of OTS integration method, type of developed system, and type of acquirer domain. Other important questions were about risk perceptions, collected from stakeholders in this study. Q1.2 (Part 2 of the questionnaire) Q1.2.1 How important to you is it to involve stakeholders in risk management, and why? Q1.2.2 How do you understand the differences in risk perceptions between stakeholders, and how important is it to analyse these differences? Q1.2.3 Who has risk responsibility for this project? Q1.2.4 What are your roles in the project risk management? 81

Q1.2.5 How was your level of risk control and risk impact for each of these risks? (low/high) Q1.3 (Part 3 of the questionnaire) The questions collected participant information: participant position in the project (unit of analysis), participant position in their organisation, experience in software development projects (in years), experience of using OTS in projects (in years), number of OTS-based software development projects involved with, educational background (degree), software engineering/computer science/information system/information technology/telematics related education, business ownership, industry domain (IT industry,

telecommunication

service

provider,

software

vendor/producer,

IT

consultant/software house, software-related company, other). 2. Second stage There were two main questions for the interview: an initial evaluation of the framework, and analysing differences in risk perceptions, as follows. Q2.1. Evaluation of the framework Participants would be asked to initially evaluate the proposed framework using previously proposed evaluation criteria for risk assessment [133][134]. Four criteria were used in this study: 

Adaptability: Q2.1.1

How difficult is it to use the proposed framework in your different

projects? (Portability) (easy/medium/difficult) Q2.1.2

How difficult is it to integrate the proposed framework with your

existing risk management? (Modifiability) (easy/medium/difficult) 

Feasibility (of the input data): Q2.1.3

How difficult is it to get sufficient input data to use the proposed

framework? (Availability) (easy/medium/difficult) Q2.1.4

How varying are groups of risks as the input data for the proposed

framework? (Scope of the input data) (easy/medium/difficult) 

Completeness: Q2.1.5

Does the proposed framework provide enough level of detail for risks?

(Scope of completeness of the proposed framework) (easy/medium/difficult) Q2.1.6

Does the proposed framework cover comprehensive risks needed to be

addressed? (Element (of risk)) 82



Validity: Q2.1.7

How do the framework outputs represent the real phenomenon?

(Relevance) Q2.1.8

Could you implement the framework outputs? (Practicability)

The above questions used related attributes and metrics to measure the proposed framework [133][134]. The metric was extended into 3-points scale (easy-mediumdifficult) to investigate the degree of existence of an attribute [133]. Q2.2 Analysing differences in risk perceptions Questions in this section aimed to identify existing risk management practices of the case study participants (including collective risk management), to get recommendations and comments, and to ask opinions about prediction of other stakeholder's risk perception and comparison between the proposed framework and previous study. Q2.2.1 How do you manage risk collectively with your stakeholder? Q2.2.2 Please describe the process, method and tools used in your current risk management practices. a. Risk identification b. Risk analysis c. Risk prioritization Q2.2.3 How do you prioritise risks? Q2.2.4 What factors influence risk responsibility? Q2.2.5 What are your recommendations to improve risk responsibility rationalisation? Q2.2.6 Overall, what are your comments on the proposed framework? Q2.2.7 How can you integrate the proposed framework to your software process? How important is integrating the proposed framework to your software process? Q2.2.8.a. Can you predict your stakeholder’s level of risk control and impact? Q2.2.8.b.What is your opinion about the proposed framework compared to the framework used in the previous survey? 6.3.3

Theoretical Propositions

The use of propositions in a case study is essential [86]. Theory development prior to case study data collection guides what is being studied, what data to collect and what

83

strategies should be used to analyse data [86][85].

This case study used five

propositions (P): P1 Risks can be managed by involving all stakeholders [8]. P2 Understanding differences of perceptions of OTS-specific risks between developers and acquirers can help to avoid conflict and thus help to ensure successful software projects [16]. P3 Perspective mismatch occurs because of differences between software developers and acquirers as they have different backgrounds, experiences, needs, expectations [18], and goals and structures [19]. P4 Risk responsibilities are assigned to a stakeholder who has the competence and capacity needed to discharge the responsibility [34][35][78]. P5 Consensus building can be facilitated by comparing and discussing perspectives between stakeholders [22][23][24]. Table 6.2 shows mapping between questions of data collection 1 and 2, the case study questions and the related propositions. Table 6.2 Mapping between case study questions, research questions and related propositions Question Topics of question ID Data collection 1 Q1.1 Project information Q1.2.1 Importance to involve stakeholder in risk management Q1.2.2 Analysing differences in risk perspectives between stakeholders Q1.2.3 Risk responsibility Q1.2.4 Role in risk management Q1.2.5 Framework input Q1.3 Participant information Data collection 2 Q2.1 Evaluation criteria for risk assessment [133][134] Q2.2.1 Collective risk management Q2.2.2 Existing risk management practices (identification, analysis, prioritisation) Q2.2.3 Detailed risk prioritisation Q2.2.4 Factors influencing risk responsibility Q2.2.5 Recommendation to improve risk responsibility Q2.2.6 Comments to the proposed framework 84

Research questions

Related propositions

RQ3.3

P1

RQ3.3 and P2 RQ3.4 RQ3.4 P1 RQ3.4 RQ3.3 and RQ3.4 RQ3.3

-

RQ3.4 RQ3.4

P1, P2

RQ3.4 RQ3.4 RQ3.4 RQ3.4

P4 P1, P2, P3 -

Q2.2.7 Q2.2.8

6.3.4

Integration of the proposed framework to RQ3.4 P1, P2, P3 software processes Comparison to a method used in the previous RQ3.3 and P5 study (see Chapter 5) RQ3.4 The Case and Unit of Analysis Selection

The unit of analysis of each case is a process-related risk. The focus is on analysing differences in risk perspectives between developers and acquirers in completed OTSbased custom software projects. This can be classified as a retrospective case study [132]. The case study consisted of four cases and was conducted in Indonesia. Each case had 11 units of analysis. This section introduces the case study participants, covering project information, participant information, and existing risk management practices. The selection of cases was derived from research questions and access to resources [86][85]. This case study used purposive sampling from the survey respondents (see Chapters 4 and 5). One of the benefits having sampling from our survey’s respondents was their familiarity with the 11 process-related risks. The choice of study participants was based on a prior academic-industry relationship with the thesis author. Four case study participants (shown in Table 6.3) were also respondents in our previous survey. There were 69 total survey respondents. Forty-two potential survey respondents, who were selected based on their strong relationship with the thesis author, were invited to participate in this study and eight invited respondents agreed to participate. For the purpose of data collection in this study, described in Section 6.3.6, the participants were asked to involve their stakeholder. Of the eight respondents, only four could involve their stakeholder to provide inputs for the framework. Therefore, four participants were finally selected for this study. The case study participants and their stakeholders varied in terms of their organisation sizes and background. Based on participatory arrangements, the cases are referred to as 1Dev, 2AcqBank, 3AcqTelco and 4AcqGovt. The participants were the main contact persons for the case study. The participants were involved in all stages of the data collections. Each of their stakeholders came from different organisations, and was only involved in providing inputs to the framework (see question Q1.2.5) and participating in the agreement process for framework outputs after the interview. Three stakeholders (not the one for 3AcqTelco) also gave detailed risk perceptions along with the inputs to the framework. In the case of 1Dev and 2AcqBank, 85

detailed risk perceptions were collected using different interviews. The stakeholder of the 2AcqBank participant was also involved in the second data collection interview, where the agreement of the proposed risk responsibilities occurred. The other three stakeholders agreed to the proposed risk responsibilities after the interview. The project stakeholders included one acquirer and three developers, and were known from the survey to have met the requirements to participate in this research. Table 6.3 Case study participants Case name

Participant

Participant stakeholder Position and Involvement background 1Dev Developer – Acquirer – Describing detailed risk a domestic software a perceptions (interviewed by the developer company telecommunication researcher) and agreement process operators in of framework outputs after the Indonesia interview 2AcqBank Acquirer – Developer – Describing detailed risk one of the biggest an internationalperceptions (interviewed by the bank in Indonesia based IT consultant researcher), interview and agreement process of framework outputs in the interview 3AcqTelco Acquirer – Developer – Not describing detailed risk a telecommunication a domestic perceptions but only completing operator in software developer levels of risk control and impact, Indonesia companies and agreement process of framework outputs after the interview 4AcqGovt Acquirer – Developer – Describing detailed risk an Indonesian a domestic perceptions and agreement process governmental software developer of framework outputs after the agency company interview 6.3.5

The Case Description

The following descriptions of the four cases summarise the first data collection excluding inputs to the framework information. Based on participatory agreements for the case study, the real name of the participants, their stakeholder and OTS software that can reveal information related to both stakeholders were coded and anonymous. Table 6.4 summarise the project, participant information and existing risk management practices.

86

1. Case 1: 1Dev The participant of case 1Dev is a domestic software developer company. This software developer organisation has experience in customer relationship management, contract management, dashboard application, workflow management system, interactive multimedia and system integration. Their clients include government, banks, publishers, postal service, university and telecommunication companies. The participant’s stakeholder organisation is an acquirer. It is an information technology division of an Indonesian telecommunication operator. It functions to support products and services for internal usage and for external clients. The division was established in 1977 to support and automate billing processes. In this case, the 11 OTS-specific risks were investigated on a single sign on application. The application was built for one a telecommunication operator in Indonesia (acquirer). The application was used to increase interoperability and extensibility among existing customer care systems by integrating them into one interface and single sign on application. This application was a simplification and unification of pre-sales, sales and after-sales functions in customer relationship management. Therefore, this application is a critical application serving all contact points of the acquirer's service centers for all customer segments and all types of services by minimising delay and transfer of services. The application was built using a commercial PHP server, Oracle database, JavaScript and commercial active directory. The use of the PHP server and JavaScript were proposed by the developer, and the use of Oracle and the commercial active directory were requirements from the acquirer. There were no fundamental changes to system functionality. The possibility for requirements changes to the system were accommodated in the software contract. In this project, the acquirer involvement in managing risks was expected to avoid blaming the developer, and to provide support and resources to mitigate risks. In risk identification and analysis, the developer discussed differences in risk perceptions with the acquirer. The aim of the discussion was to reconcile perceptions and support development processes. However, both stakeholders did not provide the detail of the approach used to reconcile differences in perceptions. The developer also expected that the acquirer could understand proposed solutions and then could cooperate to manage risks. Therefore, the responsibility of risks was on both stakeholders. 87

2. Case 2: 2AcqBank The participant of case 2AcqBank is a Division of Information Technology of one of the biggest banks in Indonesia, playing the role of an acquirer. The division has experience contracting its system development needs to external parties. The participant’s stakeholder organisation acts as a developer. It is an Indonesian division of an American multinational technology and consulting corporation. The company offers a range of integrated solutions, products and services. The company has operated in Indonesia since 1937. In this case, the 11 OTS-specific risks were investigated on customer data integration and master data management systems. These systems need to comply with central bank regulation, and to support core banking services and to understand customer profiles. OTS software was proposed by an IT consultant and agreed by the acquirer. The system was built using COBOL, Oracle, .net and commercial SOA middleware. The SOA middleware vendor, who is an international-based IT consultant, developed the systems. Requirements changes in the systems were accommodated through discussion and negotiation with the developer. There were different approaches and perceptions between both stakeholders to manage risks. According to the acquirer, without risk management, there would be differences in expected results between both stakeholders because the developer might not understand the acquirer’s needs. In risk identification, brainstorming and discussion were used; in risk analysis and prioritization, planning and review meeting were used. Project managers of the acquirer and the developer were responsible to understand differences in risk perceptions. User acceptance testing (UAT) and proof of concept (POC) were used before implementation. 3. Case 3: 3AcqTelco The participant of case 3AcqTelco was a telecommunication operator in Indonesia, acting as an acquirer. The participant’s stakeholder organisation is a developer. It is a contact center service and outsourcing company, supported by a large Indonesian telecommunication operator. The company was established in 1975. The company has 734 employees and contact center operations in eight cities in Indonesia.

88

In this case, the 11 OTS-specific risks were investigated on a customer billing recommender system. This system is critical to remind and support other units that distribute and collect invoices. As there are customers that do not pay invoices if they do not receive them, this system is needed to ensure high levels of invoice collection. The developed system was added to an existing customer billing recommender system. To anticipate changes to the acquirer's organisational structure and customer segments, the system was developed using a parameterized design. The system was built by a domestic software developer company. The developer also has responsibility to maintain the system for 3 years. The system was built using Oracle, PHP, DHTMLX, PHPMailer, FPDF, Asterisk, and jQuery. This OTS software was proposed by the developer. The acquirer stated the importance to involve its developer in risk management. The acquirer required risk analysis for structuring requirements, project planning and scheduling. To meet the requirements, development processes were monitored. Overall, both stakeholders were responsible for risk management. 4. Case 4: 4AcqGovt The participant of case 4AcqGovt was an Indonesian governmental agency, acting as an acquirer. The participant’s stakeholder organisation is a developer. It is an engineeringbased university spin-off company with 15 years of experience in information technology-based projects. The project portfolio includes telecommunication systems, avionics systems, multimedia and online marketing solutions, academics and library, logistics and operational, e-Government, IT consultancies, and IT empowerment. The developer's clients are government agents, corporations and organisations. The developer has 50 university-educated employees. In this case, the 11 OTS-specific risks were investigated on an integrated system of ePlanning and e-Monev (e-Monitoring and Evaluation). The main functionalities of this system include data warehouse, searching, analytical tools, document collaboration and knowledge sharing. The system was built by a domestic software developer company. The system was built using commercial business analytics and business intelligence (BI) software, a commercial document management system, Oracle, MySQL Enterprise database and PHP. This OTS software was the result of an independent study to select 89

OTS software using analytic hierarchy process (AHP), to select OTS software. Requirements changes were anticipated by the acquirer because of the dynamic nature of the acquirer’s organisation. The acquirer pointed out the importance of anticipating and managing risks of development, and focusing and prioritising tasks. Even though there was no formal analysis of differences in risk perceptions, both stakeholders agreed on outputs of project tasks, which were the results of scheduled meeting to reconcile perceptions. In case 4AcqGovt, the acquirer was responsible for risk management. Table 6.4 Summary of case study participants’ project and participant information Participant in case 1Dev 2AcqBank 3AcqTelco 4AcqGovt

Project information Number of teams (full time) 4 15 4 20 Number of teams (part time) 2 3 1 Percentage of a project member 50% 80% 40% 50% who had the most previous experience with general OTSbased development Percentage of the application 40-60 % 30-40% 45% 50% functionality in the whole system was provided by selected OTS components Percentage breakdown of couple level between OTS software and system [47] Configuration 50% 10% 10% Integration 30% 40% 30% 20% Customization 15% 40% 70% 20% Development 5% 10% 50% Participant information Position in the project Project Software Software Project manager developer architect coordinator and system analyst Position in the current Chief Internal Officer Chief organisation Operational software Information Officer developer Officer Experience in software 7 5 2 20 development (year) Experience in using OTS 5 4 1 22 components (year) Experience using OTS 5-7 5 2 >30 components (in projects) Educational background Bachelor Bachelor Bachelor Master (degree) (continued 90

Educational background (major)

6.3.6

Computer Science

to Master) Computer Science (Bachelor)

Computer Science

Computer Science

Data Collection

There were two stages of data collection in this study. The first used a questionnairebased survey. The aims of the survey were to identify existing practices of involving stakeholders in risk management and analysing differences in risk perceptions, and to get inputs to the framework. The second stage collected data from semi-structured interviews and document inspections. The list of interview questions is shown in Section 6.3.2. The aims of this were to initially evaluate and also to analyse differences in risk perceptions. In the first data collection, the four participants were sent a questionnaire. The questions are shown in Section 6.3.2. The participants of this study and their stakeholder then provided inputs to the framework (these are shown in Appendix D). The participants completed all parts of the questionnaire. Two participant’s stakeholders, in the 1Dev and 2AcqBank cases, also completed framework inputs in an interview. Then, the researcher followed the framework’s steps to produce outputs. Prior to the second data collection, the framework outputs (including proposed responsibilities) were sent to all participants to be reviewed, hence can increase construct validity [86]. In the second data collection, semi-structured interviews were used to collect data. The interviews were conducted in each participant’s office in Indonesia. In the interviews, all participants were asked questions by the researcher. The researcher took notes and recorded using audio digital recorder. In the case 2AcqBank, the stakeholder of the participant was also interviewed. The interviews began by explaining the framework. All participants including participant’s stakeholder in case 2AcqBank were asked whether they agrees on the framework outputs (stakeholder involvements, risk priorities and risk responsibilities). All of them also provided information about their risk perceptions. Three participants changed their framework’s inputs (level of risk control and/or impact), before the 91

agreement process in the interview. However, all participants, including the participant’s stakeholder in case 2AcqBank, finally agreed on the framework’s outputs. After the interview, the framework outputs were sent to the stakeholder of case 1Dev, 3AcqTelco and 4AcqGovt for their agreement (the stakeholder of case 2AcqBank gave its agreement in the interview). The stakeholder of case 1Dev, 3AcqTelco and 4AcqGovt also agreed on the framework outputs. In conclusion, the participants and their stakeholders did not meet as a group during this study. Rather, their individual responses independently agreed with each other. The participants were asked to evaluate the framework using evaluation criteria for risk assessment [133][134]. The participants were also asked about existing risks management practices involving their stakeholders, and about suggestions and improvements to the framework. Finally, the researcher also asked about comparison between the framework and the method used in previous survey (in Chapter 5). After the interview, the validation processes were performed by sending transcripts of the interview and the questionnaire to all participants (not to the stakeholders of the participants) to get feedback and confirmation. The transcripts included results of this case study (will be detailed in Section 6.4). All participants confirmed the accuracy of the transcripts. Another form of data collection in the case study was inspection of documents and the participants’ websites. Three participants sent their software contract-related documents. Additional information was also accessed from the stakeholder’s websites. The document provided information about contract and project management. The websites provided information about stakeholder organisation and project description. These data collection support data triangulation, which can increase construct validity [86], and provide evidence of case study context. This study asked the participant to only provide inputs to the framework and answer questions in the first and second stages of data collection. This study did not require the participants to learn and to use the framework [131].

92

6.3.7

Data Analysis

Non-inputs of the framework’s data (participant and project information, existing risk management practices and analysing differences in risk perceptions) were analysed by linking to theoretical propositions and by developing a case description [86]. Differences in risk perceptions and framework outputs agreements between all participants and their stakeholders were analysed by developing a case description, and by using cross case analysis [86]. Cross case analysis was also used to compare and aggregate findings from the responses to the questionnaires and interviews [86]. An example is given in Table 6.7. This analysis is presented in tabular form grouped by case study questions. An initial evaluation of the framework was measured using evaluation criteria for risk assessment [133][134]. This study only used four criteria for evaluating the framework (adaptability, feasibility, completeness and validity) [133][134], as described in Section 6.3.2. Existing risk management practices and suggestions from all participants were then used to improve the framework [129] (provided in Section 6.5.4). 6.4

Result

In this section, the results are organised into three parts: the results of risk perspectives from each case study participant and their stakeholders, the risk assessment evaluation results, and the questionnaire and interviews results (data collection 1 and 2). 6.4.1 Results of Case Study Participants Analysed Using the Framework This section describes risk perceptions of the participants and their stakeholders, analysed using the framework. The results are summarised in Table 6.5. Even though there were initially nine changes to the participants’ inputs (either on risk control or impact), all participants and their stakeholders finally agreed on proposed framework outputs (stakeholder involvement, risk priority and risk responsibility). The changes of the participants’ inputs happened before the agreement of the proposed risk responsibilities during the interview in the following cases: 1Dev (4 changes); 3AcqTelco (1 change); 4AcqGovt (4 changes). Based on the risk control/impact matrix, risk priorities can be mapped into the following: collaborate as high priority risk; involve and consult as medium priority risk; inform as low priority risk. The inputs to 93

the framework are provided in the Appendix D. The next sub-sections details the risk perceptions of each participant and their stakeholder.

94

Risk Selection effort ill-estimated

Table 6.5 Framework outputs (stakeholder involvement and risk priorities) and agreement of risk responsibilities Number of risk Case responsibility 2AcqBank 3AcqTelco 4AcqGovt Dev Acq 1Dev Risk Stakeholder Risk Stakeholder Risk Stakeholder Risk Stakeholder resp. involvement resp. involvement resp. involvement resp. involvement Dev: Collaborate Dev Acq: Inform

Dev: Collaborate Acq: Collaborate Dev: Collaborate Acq: Collaborate Dev: Inform Acq: Collaborate

Both

Dev: Collaborate Acq: Collaborate Dev: Collaborate Acq: Inform

Both

Dev: Collaborate Acq: Collaborate Dev: Inform Acq: Collaborate

Both

Not adaptable to requirement changes

Dev: Consult (*) Acq Acq: Collaborate

Requirements not negotiable

Dev: Collaborate Both Acq: Collaborate

Complicated multi-OTS components arrangement Insufficient OTS component documents Lack of OTSdriven

Dev: Consult Acq Acq: Collaborate

Dev: Consult Acq: Consult

Both

Dev: Collaborate Both (*) Acq: Collaborate

Dev: Collaborate Acq: Consult

Dev

Dev: Collaborate Acq: Inform

Dev

Dev: Collaborate Both Acq: Collaborate

Dev: Collaborate

Both

Dev: Collaborate

Both

Both

Acq

95

Dev

Acq

Dev: Collaborate Acq: Collaborate Dev: Collaborate Acq: Collaborate Dev: Collaborate Acq: Involve (*) Dev: Involve Acq: Inform

Both

1

Both

1

Dev

1

Dev: Collaborate Acq: Involve (*) Dev: Involve Acq: Involve

Both

2

Both

Both

Both

3

1

2

1

3

2

1

2

4

requirements engineering process Maintenance planning unfeasible

Acq: Collaborate

Acq: Collaborate

Dev: Involve (*) Acq: Consult

Dev

Dev: Inform Acq: Inform

Both

Upgrade unfeasible

Dev: Inform Acq: Involve

Acq

Dev: Inform Acq: Inform

Both

Lack of information on vendor

Dev: Collaborate Both Acq: Collaborate

Dev: Collaborate Acq: Collaborate Dev: Collaborate Acq: Inform

Both

Lack of support Dev: Consult (*) Acq: Involve

Acq

Dev

Dev: Collaborate Acq: Inform (*) Dev: Inform Acq: Inform

Dev

Dev: Involve Acq: Involve

Both

Both

Both

Dev: Collaborate Acq: Collaborate Dev: Collaborate Acq: Collaborate Dev: Collaborate Acq: Collaborate

Both

Dev: Collaborate Acq: Involve (*) Dev: Collaborate Acq: Inform

Dev

1

Dev: Collaborate Acq: Collaborate Dev: Involve Acq: Consult (*)

Both

1

Both

2

2

1

3

3

1

2

Reduced Dev: Consult Acq Dev: Inform Both Both Dev 1 1 2 control of Acq: Collaborate Acq: Inform future evolution of the system Legend: Acq: acquirer; Dev: developer; Risk resp.: Risk responsibility; (*): either level of risk control or impact or both was changed before the agreement of the proposed risk responsibilities during the interview

96

6.4.1.1 Risk Perceptions of Case 1Dev In case 1Dev, the participant was a software developer and the stakeholder of the participant was a software acquirer (see Table 6.3). For the risk of selection effort ill-estimated, the developer was assigned to choose two OTS software: a commercial PHP server and JavaScript. The other two pieces of OTS software were defined by the acquirer: Oracle and commercial active directory. The developer perceived itself to have high risk control and be most impacted by this risk. The acquirer decided to use the PHP server, which complied with the acquirer’s IT organisation procedure, and bought the license of PHP server. The acquirer already had the license of oracle and active directory. The acquirer perceived itself to had low risk control and impact (of the four pieces of OTS software used). Consistent with the previous risk, the developer had high potential impact from the risk of not adaptable to requirement changes, because the developer proposed the PHP server. The developer had low control over the use of commercial OTS software. The acquirer was perceived to be most impacted by this risk because the developed system was used to service customers (critical application). Based on the high potential impact to acquirer’s business, the acquirer perceived itself to have high risk control by contracting the developer to build the system. Both the developer and acquirer had high control and impact over the risk of requirements not negotiable and lack of information on vendors (commercial OTS software vendors). The developer had limited control over the commercial OTS software because the license of the commercial OTS software was owned by the acquirer. The acquirer was perceived to be most impacted by the risk of complicated multi-OTS components arrangement. There were two reasons for high risk impact. Firstly, multi-OTS software is difficult to maintain. Secondly, from an organisation-wide audit process perspective, multi-OTS software is difficult to audit. Therefore, the acquirer used higher control to reduce the impact of this risk. For the risk of insufficient OTS component documentation, the developer argued that the risk responsibility should be shared by both parties because the PHP server was 97

proposed by the developer and then agreed to be used by the acquirer. To control for the lack of the PHP server documentation, the developer customised the PHP server. The acquirer argued that the developer should submit user manual documentation. Both stakeholders perceived themselves to have high risk control and impact. The developer used an estimation-based approach to select the PHP server. Therefore, the developer had high risk control for the risk of lack of OTS-driven requirements engineering process. In addition, the developer perceived itself to have high risk impact. The acquirer also perceived itself to have high risk control and impact. Even though there was a new release of any OTS software, the decision to upgrade the OTS software in the system was the developers’. The new release of OTS software might not have been compatible with other used OTS software. Therefore, risk impact was low for the risk of maintenance planning unfeasible because the developer could control the risk (high risk control) by deciding to use existing stable OTS software. However, the acquirer perceived itself to have high risk impact because it was difficult for it to maintain the developed system. On the other hand, the acquirer perceived itself to have low control on OTS software from the commercial OTS software vendors. For the risk of upgrade unfeasible, the developer perceived itself to have low risk control and impact. On the other hand, the acquirer had control over the developer to maintain the developed system but perceived itself to have a low risk impact. Because the license of commercial OTS software belonged to the acquirer, the developer perceived itself to have low risk control for the risk of lack of support. The developer perceived itself to have high impact on this risk due to lack of technical support and user training. However, having a license made the acquirer able to negotiate with commercial OTS software vendors, therefore the acquirer perceived itself to have high risk control. The acquirer perceived itself to have low risk impact because the acquirer believed that a better understanding of OTS software features/capabilities could substitute lack of support from vendor. For the risk of reduced control of future evolution of the system, the developer claimed that evolving system was hard to control (low control) and perceived this to have a high

98

risk impact. The acquirer argued that the risk was important to control by contracting the developer (high control) and also perceived it to have a high risk impact. 6.4.1.2 Risk Perceptions of Case 2AcqBank In the case 2AcqBank, the participant was a software acquirer and the stakeholder of the participant was a software developer (Table 6.3). The system was built using COBOL, Oracle, .net and commercial SOA middleware. Both the acquirer and developer perceived themselves to have high control and impact for the risk of selection effort ill-estimated, lack of information on vendor (commercial OTS software vendors) and not adaptable to requirement changes. For the last risk, based on the software contract, both stakeholders agreed that requirement changes must be accommodated. According to the acquirer, the control and impact of the risk of requirements not negotiable was high. The acquirer argued that the developed system must meet business process requirements (high impact) and budget. To control the risk, because of limited budget, the acquirer had an option to replace the developer if it could not meet the requirements. However, the developer perceived this risk as to be under low control and to have low impact. Both the acquirer and developer perceived themselves to have high impact for the risk of complicated multi-OTS components arrangement due to high cost for maintenance. However, both perceived themselves to have low control for this risk. For the risk of insufficient OTS component documentation, the acquirer believed that the documents were needed (high impact). However, the acquirer found it impossible to completely examine the documents (low control). For this risk, the developer had high risk control and impact because the documents were needed and able to be accessed by the developer. Both the acquirer and developer perceived themselves to have high control and impact for the risk of lack of OTS-driven requirements engineering process. The acquirer maintained high control of requirement engineering process by executing a rigorous proof of concept (POC) process. This started by tender, and then used a product demo and proof of concept. 99

Both the acquirer and developer perceived themselves to have low control and impact for the risk of maintenance planning unfeasible and upgrade unfeasible. For the second risk, based on the acquirer and developer responses, there would be fewer problems to upgrade the developed system because the upgraded system would be tested. For the risk of lack of support, the acquirer perceived this as low risk control and impact but the developer perceived it as high risk control and impact. The acquirer had low impact of this risk because all possible problems would be handed to commercial OTS software vendors. The developer also believed that the support should be delivered by the commercial OTS software vendors or its official representatives. For the risk of reduced control of future evolution of the system, both the acquirer and developer perceived themselves to have low risk control and impact. The acquirer did not require the latest release of OTS software as long as the system functioned. However, if the latest version of OTS software could fix important bugs then the OTS software would be updated. 6.4.1.3

Risk Perceptions of Case 3AcqTelco

In case 3AcqTelco, the participant was a software acquirer and the stakeholder of the participant was a software developer. However, the developer in case 3AcqTelco did not give detailed risk perceptions. Thus, all description here is based on the acquirer’s perspective. The system was built using Oracle, PHP, DHTMLX, PHPMailer, FPDF, Asterisk and jQuery. For the risk of selection effort ill-estimated, both stakeholders perceived themselves high risk control and impact. The control of this risk was revealed by the following OTS selection processes. Firstly, the acquirer described a list of requirements. Secondly, the developer offered OTS software to be used. Then, the OTS software was tested by the acquirer using benefit cost analysis of the OTS software’s features. The acquirer also compared the use of the OTS software by existing developer’s customers, which were in auto finance, bank, airline and automotive industries. For the risk of not adaptable to requirement changes, the acquirer perceived itself to have low risk control and impact. The specification of requirements was defined by the acquirer and was stable (detailed in the risk of maintenance planning unfeasible). Based 100

on the contract, the acquirer had the low risk impact because the developer had a responsibility to meet the requirements. However, the developer perceived itself to have high risk control and impact. Both the acquirer and developer had high control and impact for the risk of requirements not negotiable. The acquirer used a software contract to control this risk. Requirement specification and OTS software features were documented in the software contract. The contract included shared and individual roles. The contract was negotiable, if needed changes were not set previously. The acquirer controlled the risk of complicated multi-OTS components arrangement by asking in the contract for the developer to manage multi-OTS components-related problems. The developer gave information regarding OTS software, such as the price of OTS software license. Then, the acquirer bought licenses for the OTS software. Problems occurring during the contract period must be managed by the developer, including commercial OTS software vendor-related risks. From the acquirer perspective, the high impact of this risk related to the nature of the developed system, which was a critical system. On the other hand, the developer perceived this risk as low risk control and impact for itself. The acquirer perceived the risk of insufficient OTS component documentation as having low risk control and impact but the developer perceived this risk as having high control and impact. The acquirer expected knowledge transfer, e.g. user training and user manual. Both the acquirer and developer had high control and impact for the risk of lack of OTS-driven requirements engineering process. Candidates for OTS software were derived from an IT architecture from the acquirer’s organisation (predefined operating systems, web servers and programming languages). Therefore, the list of requirements described proposed OTS software, including the use of a trusted engine (see the risk of selection effort ill-estimated). For the risk of maintenance planning unfeasible, the acquirer perceived the risk as having low control and impact because the contract, which lasted 3 years, did not prioritise the use of the latest release of OTS software. To anticipate changes to the 101

acquirer's organisation structure and customer segments, the system was developed using a parameterised design (for example, setting configuration files). The developer perceived this risk as having high control and impact, which may relate to the long term of the maintenance. Both the acquirer and developer had low control and impact for the risk of upgrade unfeasible. From the acquirer’s perspective, this risk related to the risk of maintenance planning unfeasible. The developed system was tested using data from customer growth over the last 5 years. As this risk related to the previous risk, the acquirer expected the developer to have high risk control and impact. Both the acquirer and developer had high control and impact for the risk of lack of information on vendor (commercial OTS software vendors). The acquirer perceived a high impact for this risk because the developed system was critical to support nationalwide operation. The acquirer had high control through benchmarking the use of OTS software in the OTS selection. The quality of OTS software was a major factor in its selection. For the risk of lack of support and reduced control of future evolution of the system, both the acquirer and developer perceived themselves to have high control and impact. For the first risk, the acquirer perceived itself to have high risk impact because the acquirer not only needed support but also related documentation. The acquirer asked the developer to provide a new document for the changed system creating high risk control. For the second risk, the high risk control for the acquirer was shown by the use of a parameterised design to accommodate future changes. 6.4.1.4

Risk Perceptions of Case 4AcqGovt

In case 3AcqTelco, the participant was a software acquirer and the stakeholder of the participant was a software developer. The system was built using a commercial business analytics and business intelligence (BI) software, commercial document management system, Oracle, MySQL Enterprise and PHP. According to the acquirer and developer, the control and impact of the risk of selection effort ill-estimated was high. The acquirer employed an independent consultant to conduct a study of OTS software selection using analytic hierarchy process (AHP). The 102

results of this study were then used to define features and functionalities of the developed system. Thus, the acquirer perceived OTS software selection-related risk as having high impact. Both the acquirer and developer though that the risk of not adaptable to requirement changes had high control and impact. The acquirer expected that all OTS software used would still give optimal benefits if requirements changed. The acquirer had low impact and high control for the risk of requirements not negotiable. Based on the contract, requirements and development were the developer’s responsibility (low risk impact for the acquirer). The acquirer perceived itself to have high risk control because the acquirer regarded the terms of reference (TOR) as a control. The acquirer also assumed that if the developer agreed with TOR then the developer should understand system requirements. The developer perceived itself to have high risk control and impact for this risk. The acquirer perceived risk impact as low for the risk of complicated multi-OTS components arrangement because the license arrangements of commercial OTS software were not complex. In addition, the risk control for this risk was low because the TOR managed multi-OTS components arrangement implicitly. According to the acquirer, the developer could ask for additional OTS software licenses as long as stated in the contract between the acquirer and commercial OTS software vendors. However, the developer perceived itself to have low risk impact and high risk control for this risk. The developer expected that the acquirer should have high risk control because the acquirer had agreed on the proposed analysis and design of the developed system (including all related risks). Therefore, both stakeholders were responsible. For the risk of insufficient OTS component documentation, the acquirer believed itself to have low impact and high control and the developer believed itself to have high control and impact. The acquirer had high risk control because there was control over output documentation for the developed system. Both the acquirer and developer had low impact but high control for the risk of lack of OTS-driven requirements engineering process. The acquirer thought if had low risk impact because the development was contracted to an external party (the developer) 103

based on the TOR. The acquirer used the results of a technical study, informing the need of integration of multi-OTS software, because a single platform was not sufficient (high risk control). Therefore, system development needed modification to integrate multiOTS software to meet requirements. Both stakeholders also perceived the risk of maintenance planning unfeasible to have low impact but high control for themselves. The acquirer thought the risk had low impact because the development was contracted to an external party (the developer), based on the TOR. Future changes were anticipated by the acquirer because of the dynamic nature of the acquirer’s organisation (high risk control). For the risk of upgrade unfeasible, the acquirer perceived itself to have low impact and high control, but the developer perceived itself to have high control and impact. The acquirer regarded the terms of reference (TOR) as a control. Furthermore, if the developed system did not meet requirements then, based on TOR, the acquirer would not accept the developed system. However, the developer expected that the acquirer should also be responsible for this risk because the acquirer was the party that ordered the update to the developed system. The acquirer indicated having low control and impact for the risk of lack of information on vendor (commercial OTS software vendors). As system development was contracted to the developer, the acquirer perceived this risk as having low impact. According to the acquirer, the developer was very concerned about information available from commercial OTS software vendors. Based on the TOR, the developer was responsible to manage not only open source but also commercial OTS software used in the system. On the other hand, the developer perceived this risk to have high control and impact. Both stakeholders had high control and impact for the risk of lack of support. The acquirer argued that lack of support would affect internal capacity building to maintain and enhance the developed system. Internal capacity building was needed to reduce vendor lock-in. In addition, the acquirer required that user training must be performed by commercial OTS software representatives or certified person from commercial OTS software vendors.

104

For the risk of reduced control of future evolution of the system, the acquirer perceived itself to have high impact and low control. This risk had high potential impact because of significant changes of policies. This risk had low control because the maintenance was performed by the developer. On the other hand, the developer perceived the risk to have low impact and high control. 6.4.2

Risks Assessment Evaluation Results

Table 6.6 presents the results of evaluating the framework using evaluation criteria [133][134]. Table 6.6. Comparison of evaluation criteria for risk assessment for four participants Attribute

Case name (based on the case study participant’s opinions) 1Dev 2AcqBank 3AcqTelco 4AcqGovt Portability Easy Medium Easy Easy Modifiability Medium Easy Medium Medium Availability Medium Easy Easy Medium Scope of the Easy Easy Easy Medium input data Scope of Medium Medium Easy Easy completeness of the proposed framework Element (of Cover various Cover various Cover various Depend on the risk) and different risks (in the risks and the framework’s risks beginning) list of risk outputs. could be adjusted Relevance Represent real Relevant with Represent real Map risks real risks (to risks stakeholders define (completed by concerns and responsibility on the acquirer responsibilities risks in the and developer) of tasks. project contract) Practicability 1. Manage risks Define risks of Used to Are applicative (in term: who acquirer and discuss about should be developer in and take responsible, software action on risks when and how development. manage risks) 2. Define and detail Scope of Work (SoW)

105

6.4.3

Results of Data Collection 1 and 2

Table 6.7 summarises the questionnaire and interview answers, excluding answers to questions about the framework inputs and evaluation. Complete results can be found in Appendix E.

106

Table 6.7 Example of cross-case analysis for data collection 1 and 2, excluding framework inputs and evaluation criteria for risk assessment Question

Case name (based on participant’s opinions) 1Dev 2AcqBank Factors influencing The one who has Stakeholder having risk responsibility responsibility on risks. knowledge (on a risk). Defined in Scope of Work. Risk outside responsibility of acquirer and developer must be discussed with respective stakeholder. Recommendation to improve risk responsibility

Prediction of other stakeholder‘s input on the framework

Detailing Scope of Work (after using the framework)

Cannot predict

Defined in the project contract.

Defined in software contract.

Cannot predict because of different interests. 107

3AcqTelco Defined in the software contract.

4AcqGovt Technical and non-technical understanding of risk.

Acquirer organisation's Communication to achieve shared policy about understanding. stakeholder role and responsibility on system Resource and capacity. development. Terms of reference. Negotiation. Describing level of Defining stakeholder responsibility each stakeholder and clearly. shared responsibility. Shared understanding of the definition of Adding cost/benefit control. analysis and comparison. Detailed communication instruments, e.g. meeting and Terms of reference (ToR).

Cannot predict

Clarification the understanding of each stakeholder’s roles and responsibilities on (ToR). Cannot predict unless both stakeholders have a shared understanding

Comparison to a method used in Chapter 5

This framework is better Using this framework, stakeholder should be able to determine risk control.

This framework is better because this framework compared actual risk perceptions.

108

This framework is better because this framework compared actual risk perceptions and use simple level of risk control and impact.

This framework is better because the framework provided structured, methodical and measured approach to analyse risk control and impact.

6.5

Discussion

This section discusses RQ3.3 and RQ3.4. 6.5.1

RQ3.3: How are OTS-specific risks perceived by developers and acquirers?

As our study focused on the perceptions of risk control and impact, we aimed to identify key findings emerging from the cross-case analysis. We analysed stakeholder involvement in Table 6.5 to identify key findings about risk priority. Finally, we investigated commonalities of risk priorities between both stakeholders. 6.5.1.1 Classification of Key Results about Risk Control and Impact Before performing the cross-case analysis, we classified key results about risk control and impact into 12 important results based on

categories from the Software

Engineering Institute’s Taxonomy of Software Development Risk [67], as described in Table 6.8. Since not all results could be classified into specific attributes of the risk taxonomy, elements (which consist of attributes) were also used to classify the results. We initially used four elements (requirements, design, integration and test, and contract) and two attributes (budget and vendors). However, the risk taxonomy is a generic risk categorisation scheme; therefore, as suggested in [67], we added two elements as risk categories: OTS software selection, which is specific to OTS-based software custom projects, and maintenance. Maintenance was added because the risk taxonomy was specific to software development. However, to avoid redundant information caused by important results being classified into more than one category, we combined categories to simplify the presentation of the risk classification. We had 12 categories: requirements and contract; OTS software selection; design; maintenance; maintenance, design and contract; maintenance and contract; contract; contract and OTS software selection; vendor; vendor, OTS software selection and integration; vendor and maintenance; and budget. Three results related to requirements and contracts. Firstly, acquirers considered that developers had a responsibility to meet the requirements. Secondly, the acquirer regarded the contract as a risk control mechanism. In the 4AcqGovt case, for the risk of requirements not negotiable, it was found that requirements and development are the developer’s responsibility, and that the acquirer would not accept a developed system that did not meet requirements (the risk of upgrade unfeasible). Finally, the contract 109

was used to change and negotiate requirements. In the 2AcqBank case for the risk of not adaptable to requirement changes, the contract was used to control for requirement changes by accommodating those requirements changes. Both stakeholders agreed that the developed system must meet system requirements. In addition, in the 3AcqTelco case, for the risk of requirements not negotiable, the contract was negotiated to accommodate changes needed and not previously set. Two results related to OTS software selection. Firstly, from the acquirer perspective, an independent study of OTS selection using AHP could define provided features and functionalities of the system requirements (the risk of selection effort ill-estimated in case 4AcqGovt). Secondly, from the developer perspective in case 1Dev, because the acquirer had agreed to the proposed OTS software from the developer, the responsibility was shared. There is one result related to design. In the 4AcqGovt case, for the risk of complicated multi-OTS components arrangement, the developer expected that after the acquirer had agreed to the proposed analysis and design of the developed system (including all related risks), the responsibility should be shared. Regarding maintenance-related risks, acquirers had two different kinds of perspective and developers had one. According to the first acquirer perspective, multi-OTS software was difficult to maintain and audit; therefore, more control was needed. According to the second acquirer perspective, to build internal capacity building and decrease vendor lock-in, the acquirer required that user training must be performed by OTS software vendor representatives or certified persons from the OTS software vendor. The developer decided to use existing stable OTS versions to lower the risk impact of maintenance planning unfeasible. Maintenance could also be related to design and contract from the perspective of the acquirer. Future changes, which might be caused by the dynamic nature of the acquirer’s organisation, were anticipated by the acquirers (in the 3AcqTelco and 4AcqGovt cases). One of the strategies used to reduce control of future evolution of the system was a parameterised design approach (case 3AcqTelco).

110

Three results are related to maintenance and contract. Firstly, from the perspective of the acquirer in the 4AcqGovt case, to control the risk of reduced control of future evolution of the system, the developer performed system maintenance. Secondly, according to the acquirer, based on the contract, the responsibility of vendor-related information (such as vendor support and customer service) was on the developer (the risk of lack of information on vendor). Finally, because the license of the OTS software belonged to the acquirer, the developer perceived themselves to have high impact due to lack of technical support and user training (in case 1Dev for the risk of lack of support). When acquirers acquire a critical system, they tried to reduce risks through their contract with developers. From the acquirers’ perspective, there were two risks associated with contracting the developer: complicated multi-OTS components arrangement in the 3AcqTelco case and not adaptable to requirement changes in the 1Dev case. As stated in the contract in the 3AcqTelco case, problems occurring during the contract period must be managed by the developer, including OTS software vendorrelated risks. By contracting the developer, the acquirer in the 1Dev case wanted to reduce risk impact of requirement changes in a critical system. Contract-related risks were also found in other categories. For example, based on contract, the developer chose OTS software; therefore the developer had high impact as a consequence of proposing and choosing OTS software (in case 1Dev for the risk of selection effort ill-estimated). Furthermore, as stated by the acquirer in the 4AcqGovt case, the responsibility for understanding OTS-driven requirements and maintenance planning was on the developer’s. As found in the 1Dev case, the developer and acquirer had different perception of limited control over OTS software. The developer perceived themselves to have low control over OTS software in the integration and selection phases and the acquirer perceived themselves to have low control over OTS software in the maintenance phase. As both stakeholders experienced high risk impact, a possible explanation for the difference in limited control over OTS software could be attributed to the importance of OTS software during each phase for each stakeholder. In the risk classification, the limited control over OTS software was classified as vendor, as shown in Table 6.8.

111

Two acquires had the same perception about OTS software vendors regarding the risk of lack of support. In the 2AcqBank case, the acquirer perceived themselves to have low risk impact because all support-related problems would be handed to the OTS software vendor. The acquirer in the 1Dev case argued that having a license of OTS software made them able to negotiate with the OTS software vendor. Regarding budget, two acquirers perceived themselves to have high risk control. For the risk of requirements not negotiable, the acquirer in the 2AcqBank case would have chosen another developer to suit the existing budget and business process requirements. In the 4AcqGovt case, the acquirer expected the OTS software to meet their budget for the optimal benefit using OTS software if the requirements changed.

112

Table 6.8 Classification of key results regarding risk control and impact Category Requirements and contract

Case 1Dev

2AcqBank The contract was used to control for requirement changes (acquirer and developer)

3AcqTelco Based on the contract, the developer had a responsibility to meet the requirements (acquirer) The contract might be negotiated to accommodate needed changes not set previously (acquirer)

OTS software selection

Because the acquirer agreed on proposed OTS software then the responsible should be shared (developer)

Design

113

4AcqGovt Requirement and development were the developer’s responsibility (acquirer) The acquirer would not accept the developed system that did not meet requirements (acquirer) The responsibility for understanding OTS-driven requirements was on the developer (acquirer) Using a method in OTS software selection (i.e. AHP) could define provided features and functionalities of the system requirements (acquirer) Because the acquirer agreed on proposed analysis and design of the developed system then the responsible should be

shared (developer) Maintenance

Maintenance, design and contract

Maintenance and contract

Contract

Multi-OTS software were difficult to maintenance and audit (acquirer)

The support of OTS software vendor was required to build internal capacity building and decrease vendor lock in (acquirer)

The developer decided to use existing stable version (developer)

Changes were anticipated by the acquirer by using a parameterised design approach (acquirer)

The developer did not have the license of OTS software; therefore the developer perceived to have high impact due to lack of technical support and user training (developer)

Changes were anticipated by the acquirer (acquirer)

The acquirer perceived to have low control because the system maintenance was performed by the developer (acquirer) The responsibility for maintenance planning was on the developer (acquirer)

The impact of critical system made the acquirer contracting the developer

The impact of critical system made the acquirer contracting the 114

The responsibility of information on vendor was on the developer (acquirer)

Contract and OTS software selection

Vendor

Vendor, OTS software selection and integration Vendor and maintenance

Budget

(acquirer) The developer chose OTS software; therefore the developer had high impact as a consequence of proposing and choosing OTS software (developer) Having the license of OTS software made the acquirer to be able to negotiate with OTS software vendor (acquirer) The developer perceived to have low control over OTS software in integration and selection phase (developer) The acquirer perceived to have low control over OTS software in maintenance phase (acquirer)

developer (acquirer)

The acquirer perceived to have low risk impact because all supportrelated problems would be handled to OTS software vendor (acquirer)

The acquirer would choose other developer to suit with existing budget and business process requirements (acquirer) 115

The acquirer expected OTS software to meet budget for optimal benefit of the use of OTS software if requirements changed (acquirer)

6.5.1.2

Cross-Case Analysis

Here we identify key findings emerging not only from each individual case but also across cases [136]. As the focus of this study is to investigate risk perceptions using the stakeholder analysis approach, the findings were based on the relationship between control and impact of investigated risks. Firstly, similar relationships were generalised to identify the findings not only within each case but also across cases [136]. Secondly, the findings were selected from each category by comparing evidence either within each case or across cases [136]. Sometimes several categories in Table 6.8 were merged or synthesised to generalise a finding. There are 11 key findings on risk control and impact. Finding 1 (high risk impact and control): A high risk impact made acquirers apply higher control over the risk. Finding 1 is synthesised from three categories: maintenance, contract and budget (see Table 6.8). In the maintenance category, the type of impact are the difficulty of maintaining and auditing multi-OTS software (in the 1Dev case), and the need to build internal capacity and decrease vendor lock-in (in the 4AcqGovt case). For the second impact (in the maintenance category), user training must be performed by OTS software vendor representatives or certified persons from the OTS software vendor. In the contract category, because of the impact of critical application, the acquirer in the 1Dev and 3AcqTelco cases contracted the developer to build the application. Thirdly, in the budget category, the type of impact is not meeting the budget for the optimal use of OTS software if requirements change (in the 4AcqGovt case), and not meeting business process requirements and having a limited budget (in the 2AcqBank case). For the second impact in the budget category, acquirers could choose another developer to suit the existing budget and business process requirements. Finding 2 (OTS selection): An acquirer can use OTS selection results to gain higher control when defining system requirements, which has high impact. In the 4AcqGovt case, the OTS selection was performed by an independent study of OTS selection using AHP. Finding 3 (maintenance): A developer can decide to use an existing stable OTS version to control the impact of system maintenance (as in the 1Dev case), providing higher control over this risk. 116

Finding 4 (maintenance): Acquirers can anticipate future changes by asking their developer to use a parameterised design (for example, setting configuration files). Evidence of this can be found under the maintenance, design and contract category in the 3AcqTelco and 4AcqGovt cases. Finding 5 (contract): To lower risk impact, acquirers may contract an external party. The relationship between an acquirer and its external party must be governed by a software contract. Support for this finding can be classified into two groups: low risk impact and low risk control; and low risk impact and high risk control. There were three instances of the first group (low risk control and low risk impact). All support-related problems were handled to OTS software vendor in the 2AcqBank case. Secondly, based on the contract, in the 3AcqTelco case, the acquirer had the low risk impact because the developer had a responsibility to meet the requirements. In the 4AcqGovt case, the responsibility of vendor-related information was the developer’s. There were four instances of the second group (low risk impact but high risk control). Firstly, as shown in the 4AcqGovt case, the developer should have responsibility on requirement changes and development, just as with understanding requirements (in the second instance), and maintenance (in the third instance). Finally, in the 4AcqGovt case, the acquirer used the contract to define system acceptance criteria. The first group, low risk control and low risk impact, shows how the acquirer can lower risk impact by contracting external parties. For the second group, the acquirer had higher control because the contract was used to control the developer to have responsibility on requirement changes, development and maintenance. Finding 6 (contract): An acquirer might have lower control when it contracts a developer (as in the 4AcqGovt case) and uses OTS software (as in the 1Dev case). In accordance with finding 5, finding 6 shows that risk impact cannot totally be transferred to an external party. From a risk control perspective, finding 5 and 6 illustrate that even though a software contract can be used to control an external party, an acquirer will reduce its control once it contracts an external party. Finding 7 (contract): Using a software contract to control risks leads to consequences regarding what must be agreed in the contract. In the 1Dev case, based on the contract, the developer chose OTS software. Therefore the developer had high impact as a 117

consequence of proposing and choosing OTS software. Furthermore, using the contract to control for requirement changes required both stakeholders to accommodate the changes (in case 2AcqBank and 3AcqTelco). Finding 8 (licensing OTS software): A developer without an OTS software license has lower control because they lack support from the OTS software vendor. This can have high risk impact, as seen in the 1Dev case. Finding 9 (licensing OTS software): An acquirer with an OTS software license can better negotiate with an OTS software vendor, as seen in the 1Dev case. Finding 10 (limited control over OTS software): Stakeholders have different concerns about their limited control over OTS software. Developers tend to have limited control over OTS software in the OTS selection process and OTS integration, and acquirers tend to have limited control over OTS software in maintenance, after the end of a contract. Finding 11 (shared responsibility): Both stakeholders share risk responsibility if acquirers agree on the OTS software selection (as in the 1Dev case), and on the developer’s proposed system design decision (as in the 4AcqGovt case). The findings above emerged from cross-case analysis, and indicate that the analysis framework helps to compare risk perceptions, to better understand stakeholders’ risk perceptions. The comparison facilitates the agreement of proposed framework outputs and risk responsibilities between both stakeholders. Furthermore, the comparison of risk perceptions also support a discussion between stakeholders by providing opinions, further understanding [24] and expectations of each stakeholder. If both stakeholders disagree with proposed framework outputs, including proposed risk responsibilities, or other stakeholder’s perceptions, the discussion is useful to elicit the reason of this disagreement [24]. 6.5.1.3 Key Results about Risk Priority Ten interesting results based on similarities found in developers’ and acquirers’ priority of each risk are detailed in Table 6.5. The first six results concern developers. There are three risks for which the developers have high priority: selection effort ill-estimated, insufficient OTS component documents and lack of information on vendor. The 118

developers perceived OTS software selection-related risk as a high priority. A possible explanation is that they might take into account their experience and familiarity with candidate OTS software when selecting OTS software [39]. Consistent with previous explanations in this study, documentation is more needed by the developers to integrate OTS components. Finally, developers need vendor-related information during the software project, such as vendor support and customer service. For the risks of not adaptable to requirements changes and lack of support, developers do not have low risk priority, three high and one medium. For these two risks, the medium risk priority comes from the developer perspective in case 1Dev regarding the limited control over OTS software [5][40][41][137]. The sixth result is that the developers do not have high risk priority (one low and three medium risk priorities) for the risk of complicated multi-OTS components arrangement. This is because, as discussed earlier, the developer did not own an OTS software license. For acquirers, there are three results. There are two risks for which the acquirers do not have high risk priority: maintenance planning unfeasible (for which they have two low and two medium) and upgrade unfeasible (for which they have two low and two medium). There are different responses regarding the risk of maintenance planning unfeasible. For example, in the 1Dev case, the acquirer had a high risk impact due to the difficulty of maintaining OTS software, but had a low control over the OTS software vendor for maintaining OTS software. In the 3AcqTelco case, the acquirer perceived themselves to have low risk control and impact because of not prioritising the latest version of OTS software, and because of using a parameterised design. The acquirer in the 3AcqTelco case for the risk of upgrade unfeasible had the same (low) level of risk control and impact, and also the same reason as the risk of maintenance planning unfeasible. The reason was the developed system did not prioritise the use of the latest release of OTS software. However, the acquirer in the 2AcqBank case for the risk of upgrade unfeasible had the same (low) level of risk control and impact, but for a different reason, which was that upgrade would be feasible due to implementing testing and bug fixing. The last result from the acquirers’ perspective is that they do not have low risk priority (instead, they have three high and one medium) for the risk of requirements not negotiable. In the 4AcqGovt case, the acquirer believed that software requirements and development are the developer’s responsibility, as the acquirer 119

contracted the developer. The acquirer perceived this to be a high level of control. Thus, the acquirer assumed that requirements-related risks did not give high impact to them. Both stakeholders did not have a low risk priority for the risk of lack of OTS-driven requirements engineering process. There are two examples showing that both the developer (in the 1Dev case) and the acquirer (in the 4AcqGovt case) applied the OTSselection process in requirements engineering. After summarising the above ten key results, there are six additional findings identified from this study: Finding 12: Developers have high priority for the risk of selection effort ill-estimated, insufficient OTS component documents and lack of information on vendor. Finding 13: Developers do not have low risk priority for the risks of not adaptable to requirements changes and lack of support. Finding 14: Developers do not have high risk priority for the risk of complicated multiOTS components arrangement. Finding 15: Acquirers do not have high risk priority for the risk of maintenance planning and upgrade unfeasible. Finding 16: Acquirers do not have low risk priority for the risk of requirements not negotiable. Finding 17: Both stakeholders do not have low risk priority for the risk of lack of OTSdriven requirements engineering process. 6.5.2

RQ3.4: How should risk responsibilities be assigned between developers and acquirers for these risks?

All participants pointed out the importance of involving stakeholders and analysing differences in risk perspectives [8]. However, the study identified various challenges to this: 

No specific method was in use to analyse differences in risk perceptions.



Different approaches were used to define risk responsibility.



Different roles were defined for risk management. 120



No formal risk management approach was shared by stakeholders.



Different techniques were in use to identify and analyse risk (common risk identification techniques included brainstorming and discussion).



Different techniques were used to prioritise risks.



Stakeholders were not always able to predict other stakeholders’ perceptions.

This study proposes a framework to collect, analyse, compare and discuss differences in risk control and impact perceptions between the participants and their stakeholders. The framework outputs are stakeholder involvements and risk priorities for each stakeholder. Risk responsibility is also proposed to belong to the stakeholder who has higher risk control. This is because high influence and power over risk is needed to control key decisions and task implementation [31][32][33], to mitigate risk. The responsibility should be assigned to a stakeholder who has the competence and capacity needed to discharge the responsibility [34][35]. However, if both stakeholders have the same level of risk control, the risk responsibility is assigned to both stakeholders. These outputs and risk responsibilities are then proposed to a developer and acquirer for agreement. If both stakeholders can agree on the outputs and risk responsibilities, the risk responsibilities are assigned as agreed; otherwise, both stakeholders can discuss the proposed risk responsibilities by considering the analysis. However, although the framework was designed to be used by a developer and acquirer, to avoid the effects of learning to use [131] the framework, the case study participants were only asked to complete the framework inputs and to agree on the framework outputs and proposed risk responsibilities. Table 6.5 shows that all participants and their stakeholders agreed on proposed risk responsibilities and risk priorities. In discussing the agreement of proposed framework outputs and risk responsibilities, there was initially one participant stakeholder (in the 4AcqGovt case) who did not agree with the five proposed risk responsibilities, but this participant stakeholder finally agreed after discussing the process of determining risk responsibility in the framework with the researcher. This study found that the sources of risk responsibility are software project contract, knowledge of risk, negotiation, communication, organisation policy, and resource and capacity. Project contracts, knowledge on risk, and organisation policy can be classified 121

as formal controls, while negotiation and communication can be classified as informal controls [84][82]. Knowledge can complement

control [84],

and competence and

knowledge affect the choice of controls [82], while resource and capacity were needed to control risks. Therefore, the framework supports responsibility model [81][78][34][35], which assigns responsibilities to the stakeholder who has competence and capacity needed to discharge the responsibility. Defining responsibilities could help identify roles [81][78] [34][35] to mitigate risks. The results show that the responsibilities of the 11 risks vary as summarised in Table 6.5. The developers were assigned to have risk responsibilities in eight of the risks, the acquirers in six of the risks, and both stakeholders in all 11 risks. See Table 6.5 under the heading of Number of Risk Responsibilities for the risk of lack of OTS-driven requirements engineering process, all participants and their stakeholders agreed on proposed risk responsibility, which is ‘both stakeholders’. However, they disagreed for other risks. One of the possible explanations why all participants and their stakeholders did not agree is because

stakeholder perceptions vary based on individual's or

organisation's background, experience, need and expectation [18][19]. This may also be why participants were not able to predict other stakeholders’ perceptions. Two of the eight risks (insufficient OTS component documents and maintenance planning unfeasible) for which the developers had responsibility, the risk responsibilities are in the developers and both stakeholders. Documentation is more needed by the developers to integrate OTS components and also to maintain the system. However, the acquirers have more responsibility for the risk of complicated multi-OTS components arrangement than the developers. A possible explanation is because the acquirers owned the license for OTS software. Three participants changed their input of risk perception (provided in Section 6.4.1). As mentioned in the data collection section, the changes of the framework’s inputs, risk control and/or impact, happened before the agreement process in the interview. There are two possible explanations for the changes. Firstly, the participants in case 1Dev, 3AcqTelco and 4AcqGovt got more understanding of the changed inputs during the agreement process because the participants could confirm their understanding about risks under study with the researcher during the interview. Secondly, the participants in case 1Dev, 3AcqTelco and 4AcqGovt 4 were affected by their stakeholder answers of 122

the framework’s inputs. The first type of changes were data-driven and the second type of changes either data-driven or expectation-driven [138]. If a stakeholder does not agree with other stakeholders’ perception on the proposed framework outputs, the disagreement should be discussed (step 5 of the framework). This discussion builds consensus and provides shared understanding, and is in line with a study comparing software architectures [24] and risk management [22][23]. In sum, the framework provides structured and methodological steps to define a risk and role responsibility by comparing risk perceptions of both stakeholders. 6.5.3

Evaluation Criteria for Risk Assessment Findings

Based on the evaluation criteria for risk assessment [133][134] listed in Section 6.3.2, all participants indicated that the framework could be able to be applied to different projects. The participants also thought that the framework could be modified by adding other risk attributes (e.g. cost/benefit) and by quantifying risk. The participants reported that the framework had a sufficient feasibility of input data. There was a need to better define and justify the levels of risk control and impact (according to two out of four participants). The participants also noted that scope of completeness of the framework would need to be adjusted for a more detailed risk assessment. However, two participants argued that the framework was simple to use. According to the participants, the framework could also cover various and different risks because the framework provided an input template and did not pre-define risks to be analysed. The list of risks in the questionnaire could be adjusted. All participants indicated that the framework could represent real risks; half of the participants reported that the framework could be used to compare stakeholder perspective and be used to propose risk responsibilities. Overall, all participants agreed that the framework outputs could be used to manage risks, and help to inform scope of work (SoW) about responsibilities of process-related risks.

123

6.5.4

Improving the Proposed Framework

Participants proposed improvements by comparing existing risk management practices (found in this case study) with the framework, and making use of suggestions from all participants. Three main improvements were proposed as follows. 1. Stakeholders should use the framework. All stakeholders should be involved in assigning risk responsibilities (three cases). The other case’s participant pointed out that the framework should be initiated by the software acquirer, but that the framework would be more effective if it was initiated by someone with the personal strength to reconcile differences in risk perceptions [20]. 2. Stakeholders should compare risks using the framework. The outputs proposed were stakeholder involvement and risk priority (including risk responsibilities). Three framework improvements were suggested: 

Risks should be analysed at the beginning of a project



There is a need for a shared understanding of the definition of control and impact. Therefore, the framework needs an introduction, a glossary of used terms (such as control and impact) and related context, verification of stakeholder’s answers, and a session to discuss differences of understanding of definition of control and impact. The introduction and verification are already in the steps of the framework.



The framework could include other risk attributes (such as cost/benefit) and risk quantification for more detailed risk assessment.

3. The framework could be used to detail the level of stakeholder responsibility in software contract and the scope of work (SoW). The framework could also be used to clarify understanding of each stakeholder’s roles and responsibilities in the ToR. Based on the participants’ opinions given above, the framework has potential use for reconciling different process perspectives by defining roles for processes in OTS-based custom software projects. This study validates the potential uses and benefits of the framework. The aims of analysing risks are to develop shared understanding, shared responsibility, stakeholder commitment and shared perspectives; to avoid conflict; and to focus and to prioritise tasks. Furthermore, the framework provides early understanding of risk perceptions, 124

facilitating the planning of a software project and formulating the software contract. Overall, all participants agree that the framework can be used to initiate and support the discussion to reconcile different risk and process perspectives between stakeholders. Based on our case study results, compared with standard clauses on a software contract [139] or existing organisational policies, stakeholders were expected to

clarify

understanding of each stakeholder’s roles and responsibilities in the terms of reference (ToR) (see Recommendation to improve risk responsibility

from participant

4AcqGovet in Table 6.7). It was assumed that the tender winner had understood the ToR. However, the tender winner tended to be overoptimistic and the winner did not discuss who would control and be impacted by risks. Thus, the framework provides a bi-directional approach to discuss and negotiate risk responsibility. A summary of the case study evidence and related propositions and literature related to evaluation, and improvement of the framework is presented in Table 6.9 Table 6.9 Summary of case study evidence and related propositions/literature Evidence of related case study question (question ID) Project information is described in Section 6.3.5 (Q1.1) Importance to involve stakeholder in risk management (Q1.2.1) Importance to analyse differences in risk perspectives between stakeholders. No specific method used to analyse differences in risk perceptions (Q1.2.2) Different approaches to define risk responsibility (Q1.2.3) Different roles in risk management (Q1.2.4) Framework outputs are presented in Table 6.5 (Q1.2.5) Participant information is described in Section 6.3.5 (Q1.3) Results of evaluation criteria of risk assessment are presented in Section 6.4.2 (Q2.1) No formal risk management shared by stakeholders (Q2.2.1) Different techniques used to identify and analyse. Shared risk identification techniques: brainstorming and discussion (Q2.2.2) Different technique used to prioritise risks (Q2.2.3)

Related propositions/literature Support proposition P1 Support proposition P2. Challenges for proposition P2. Challenge for proposition P1 Challenge for proposition P1 Challenge for proposition P1, P2 Challenge for proposition P1, P2

Challenge for proposition P1, P2 The sources of risk responsibility: software project Support proposition P4 and 125

contract, knowledge on risk, negotiation, communication, organisation policy, and resource and capacity (Q2.2.4) The use of the framework to detail scope of work and software contract, detail of stakeholder and shared responsibility (two participant), shared understanding of the definition of control, adding required attribute to analyse and clarification the understanding of each stakeholder’s roles and responsibilities on (ToR) (Q2.2.5) Risk expectation driven by risk perception (two participants) (Q2.2.6) Understanding risk control, impact and responsibility. Adding required attribute to analyse (Q2.2.6) Integration of the framework in the beginning of the software project (three participants) (Q2.2.7) Inability to predict other stakeholder perceptions (Q2.2.8.a) Comparison of actual risk perceptions better than perception-based risk (two participants). Structured, methodical and measured approach (Q2.2.8.b) 6.5.5

[84][82] Improvement for proposition P1, P2, and solution for P3

This finding [18][138] -

supports

Improvement for proposition P1, P2, and solution for P3 Stakeholder perceptions vary [18] Objectively reviewed decision [140]

Case Study Findings Related to OTS-based Custom Software Project Stakeholders

There was one risk related to responsibility among the acquirer, developer and OTS software vendor found in this study. The participant in case 4AcqGovt illustrated a risk related to OTS software vendors when a system was infected by a computer virus and need to be re-installed. This risk could only be solved by the OTS software vendor. The question was who would be responsible to mitigate this risk, the acquirer or developer? According to the participant in case 4AcqGovt, the framework could solve this case by showing that both stakeholders might not prepare a mitigation plan or have the resources to mitigate this risk. The framework could inform the acquirer that its developer did not have the capability to re-install the software. 6.5.6

Case Study Findings Related to Expectations of Level of Risk (Control/Impact)

Two participants discussed expectations of level of risk control and impact. The participant of case 1Dev pointed out that its acquirer expected the participant to have high risk control/impact. Another participant, in case 3AcqTelco, expected its developer to have high risk control/impact after discovering its developer’s answer to another risk 126

related to the first risk. The participant in case 3AcqTelco asked the researcher, “Why does the risk of maintenance planning being unfeasible have high risk control and impact, but the risk of upgrade being unfeasible have low risk control and impact?”). The participant believed that there was a relation between these two risks. These findings support propositions of relationship between perception and expectation [18][138] (see Table 6.9). These findings also indicate that identifying stakeholders’ risk expectations can help to ensure risks are addressed [38]. 6.5.7

Case Study Findings Related to Processes (Role and Responsibility)

Investigating process-related risks in this study could lead us to identify and define roles or responsibilities for software processes. As can be seen in Table 6.7 (Factors influencing risk responsibility and Recommendation to improve risk responsibility), the framework has a potential use to inform the software contract and scope of work. The findings showed that the framework could also be used to clarify understandings about roles and responsibilities. The potential use of the framework to analyse differences in the process perspective is consistent with one of the uses of stakeholder analysis in software-related projects, which is to identify process roles [36][37][13]. 6.6 6.6.1

Threats to Validity Construct Validity

To minimise threats to construct validity, related and relevant theories were used prior to case study design. Interview transcripts were reviewed by the participants [86]. Data triangulation was also used to increase construct validity [86]. The questions used in this study were reviewed by two internal experts. 6.6.2

Internal Validity

To control confounding factors [131], this case study involved developers and acquirers as the participant and their stakeholder in the case study. Moreover, the case study participants were respondents of our survey (detailed in Chapters 4 and 5). To avoid the effects of learning to use the framework [131], the participants and their stakeholders were only asked to complete the framework inputs. Therefore, internal validity of this study was expected to increase as the case study focused on evaluating and improving of the framework, and not learning how to use the framework [131].

127

In all cases, even though there were possibilities that inaccurate answers were provided to meet the expectations of others, these answers were minimised in this research. First, one of the benefits using a completed project in each case is to encourage the participants and their stakeholder to give inputs of the framework based on facts. Our case study has found that there were no specific methods used to analyse differences in risk perceptions. Therefore the use of retrospective risk perceptions in the participants' completed project was expected not significantly influencing opinions given by the respondents. Second, the participants and their stakeholder were only asked to provide their risk control and impact (as inputs of the framework) and not others’ expected risk control and impact. In addition, when each party provided its answers, it would not know other's answers. 6.6.3

External Validity

The use of a case study limits the generalisability of its results to theoretical propositions and not to populations [86]. However, all participants agreed that this study could be used in other OTS-based custom projects and other project types.

6.6.4

Reliability

To increase reliability [86], this study followed existing case study guidelines [86] [131][132][129]. This study also recorded and maintained a case study database to increase reliability [86]. To simplify the presentation of the thesis, we decided to not include quotes of the participants (there is one quote from the participant (Section 6.5.6).) and to describe and summarise the case study data and outcomes. To support audit trail, we provide cross-case analysis from data collection 1 and 2 in Appendix E. 6.7

Summary

This chapter has investigated differences in perception of 11 risks of OTS-based custom software projects between developers and acquirers by comparing their perceptions of risk control and impact. It used a multi-case study method. Based on the method used to analyse the survey data in Chapter 5, a risk framework was developed to collect, compare and analyse these differences. The framework outputs are stakeholder involvements and risk priorities for each stakeholder. There were four participants in the

128

case study: these were among 42 invited respondents of our survey (see Chapters 4 and 5), who agreed to participate and could also involve their stakeholder in this study. This study yielded 17 detailed findings related to levels of risk control and impact, and proposed risk priorities. As the participants were not able to predict other stakeholders’ perceptions, due to differences in individuals, their organisations’ background, experience, need and expectation [18][19], these findings can serve as an initial checklist for understanding other stakeholders’ risk perspectives. Five findings about risk perceptions are specific to developers. Firstly, developers often decide to use existing stable OTS versions to control the impact of system maintenance. Secondly, developers without an OTS software license often lack support from OTS software vendors. There are three different findings regarding risk priority. Firstly, developers have high priority for the risk of selection effort ill-estimated, insufficient OTS component documents and lack of information on vendor. Secondly, developers do not have low risk priority for the risks of not adaptable to requirements changes and lack of support. Lastly, developers do not have high risk priority for the risk of complicated multi-OTS components arrangement. Eight findings about risk perceptions are specific to acquirers. The first is that risks with high impact make acquirers apply higher control over these risks. Secondly, acquirers can use OTS selection results to define system requirements. Thirdly, in maintenance, acquirers can anticipate future changes by asking developers to use a parameterised design (for example, setting configuration files). In addition, there are two findings related to the contract: to lower risk impact, acquirers can contract an external party; however acquirers will have lower control when they contracts developers and use OTS software. With an OTS software license, acquirers can help negotiate with an OTS software vendor. There are two findings related to risk priority. Acquirers have high risk priority for the risk of requirements not negotiable. Lastly, acquirers have low risk priority for the risk of maintenance planning and upgrade unfeasible. The other four findings about risk perceptions relate to both stakeholders. Firstly, using a software contract to control risk leads to consequences regarding what must be agreed 129

in the contract. Secondly, stakeholders have different concerns about their limited control over OTS software: developers tend to have limited control over OTS software in the OTS selection process and OTS integration, while acquirers tend to have limited control over OTS software in maintenance, after the contract ends. Thirdly, both stakeholders have risk responsibility if an acquirer agrees on OTS software selection, and a developer’s proposed system design decision. Lastly, both stakeholders do not have low risk priority for the risk of lack of OTS-driven requirements engineering process. The framework proposed a default position about which stakeholder should be responsible for a risk, based on which stakeholder has higher control and power over each risk [31][32][33]. Higher risk control indicates that a stakeholder has the competence and capacity needed to discharge the responsibility [34][35]. This study found that the sources of risk responsibility are software project contract, knowledge of risk, negotiation, communication, organisation policy, and resource and capacity. Knowledge can complement control [84], and competence and knowledge affect the choice of control [82], while resource and capacity were needed to perform control over risks. The framework was used to facilitate a discussion between both stakeholders to assign risk responsibility based on the proposed risk responsibility. When stakeholders disagreed with the proposed framework outputs and risk responsibilities, or another stakeholder’s perspective, the discussion was useful to elicit the reason for this disagreement [24]. All participants and their stakeholders agreed on the proposed risk responsibilities and risk priorities. In the context of a specific project, the framework provides an explicit recognition of different risk perceptions, to inform risk management negotiation through dialogue, deliberation and communication [16]. The framework may help stakeholders reduce ‘gut feeling’ judgments and ensure decisions on the assignment of risk responsibilities are more objectively reviewed [140]. Although these findings might not be very surprising and are not generalised beyond OTS-based custom software projects and the 11 risks under study, the findings do provide insight into developers’ and acquirers’ perceptions of risk control and impact. The framework in this study added substantially to our understanding of how to provide a methodical approach to analyse different process-related risk perceptions [16]. The 130

framework is expected to increase understanding of risk responsibility for both stakeholders by discussing, negotiating [24] and clarifying risk responsibility in two directions instead of deriving it from standard contract clauses [139] or existing organisational policies. Moreover, the framework has a potential use for analysing differences in process perspectives, by identifying stakeholders’ roles and their level of involvement [36][37][13]. We also believe that the method used in this study could be applied to non-OTS-based custom software projects, to analyse differences in processrelated risk perceptions. From a methodological perspective, another contribution of this study is to extend the use of stakeholder analysis. Previously, stakeholder analysis has only been used to identify stakeholders, and to identify their roles, their level of involvement [36][37][13] and their risks [9][38][10], without comparing their different perceptions. Our study proposed a default position about which stakeholders should be responsible for a risk based on which stakeholder has higher control and power of each risk [31][32][33]. Further investigation on other related factors influencing risk responsibility is strongly recommended. To analyse differences in risk perception involving stakeholders other than developers and acquirers, the framework should be adjusted, evaluated and validated. There are another two limitations. Firstly, there was possibility of subjective understanding of the concepts of risk control and impact by all participants and their stakeholders. Secondly, one of four participants’ stakeholders did not give detailed opinions about risk perspectives.

131

7

Discussion and Conclusion

In a software project, risks affect all stakeholders [7][8][9][10]. Software developers and acquirers perceive risk differently [16] because they have different backgrounds, experiences, needs, expectations [18], and goals and structures [19]. It is necessary to reconcile stakeholder perceptions when managing processes and risks in software projects [16][20]. Prior studies have noted the importance of reconciling user and project manager perceptions of IT project risks [16], and reconciling perspective mismatch in managing software project processes [20]. However, these prior studies do not provide detailed steps on how to reconcile differences in stakeholder perspectives. This research has investigated OTS-based custom software project, and has identified risks involved [16], and analysed differences in risk perceptions and proposed stakeholder responsibility for risks [22]. This has been done, not only from developers’ perspective, but also from acquirers’ perspective. Firstly, we investigated the processes involved in OTS-based software acquisition using a systematic mapping study. Then, we used a second systematic mapping study to identify risks of OTS-based custom software projects from developers’ and acquirers’ perspectives. Finally, we extended an existing stakeholder analysis framework [25][12][26][14][13][10] to compare, analyse and discuss [22][23][24] risk perceptions between developers and acquirers of OTS-based custom software projects in a survey and a multi-case study. This chapter has five sections. Firstly, it reviews the research contributions. Then this research is compared with related work. The next section discusses implications for research, including future work. The fourth section discusses implications for practice and the last section identifies limitations. In the comparison with related work and research contribution sections, we refer to the research questions: RQ1: What are the processes of OTS-based software acquisition? RQ2: What are the risks in OTS-based custom software projects? RQ3: How should software developers and acquirers analyse differences in perceptions of process-related risks in OTS-based custom software projects?

132

7.1

Research Contributions

This thesis makes several findings and contributions to empirical software engineering knowledge, as follows. RQ1 contributions: We identified six OTS-based software acquisition processes. From the first mapping study, we identified six OTS-based software acquisition processes: decision-making to make or buy software, architectural decision-making, OTS selection, managing relationships between developers and acquirers, managing relationships between OTS adoption and acquirers’ organisations, and managing relationships between OTS vendors and developers. Our case study findings validate the existence of these processes. These processes should be considered by an acquirer when acquiring OTS-based software, in addition to the existing software acquisition process standard [6]. Although our mapping study results have shown that both the software acquisition process standard and OTS-based software acquisition have the same overall process lifecycle, there are process differences between them. The first difference is that in OTS-based software acquisition, architecture and OTS selection criteria are defined and adjusted iteratively in conjunction with software requirements specifications. Secondly, there are some OTS adoption characteristics that must be considered when acquiring OTS software (for example, the credibility of the decision maker and previous exposure to acquisition decisions) [108]. Thirdly, managing relationships between OTS vendors and developers can provide benefits in commercial negotiations with acquirers [109]. Based on the mapping study results, there are three processes that could be added into (generic) software acquisition and OTS-based software acquisition: decision-making to make or buy software, architectural decision-making during software acquisition, and managing relationships between developers and acquirers. Based on the literature review, there are three processes in common between acquirers and developers of OTSbased software: make vs. buy decisions, OTS selection and vendor interaction. A better understanding of OTS-based software acquisition and development processes, and of differences in backgrounds, experiences, needs, expectations [18], goals, and structures [19] should improve understanding of differences in process-related risk perceptions between both stakeholders. 133

RQ2 contributions: We identified risks of OTS-based and custom software development and acquisition. We have identified and classified risks of OTS-based and custom-based software, not only from developers’ perspective but also from acquirers’ perspective. These risks have been classified into 17 categories: planning, requirements, design, integration and testing, system lifecycle, maintenance, project closure, software, OTS products, cost, environment, people, systems engineering, vendor relationships, project management, contract and legal. The results consist of specific and generic risks in OTS-based and custom software acquisition and development. OTS-specific risks can be found in all risk categories. The results show that most risks found in OTS-based software acquisition have the same concerns as risks in OTS-based software development. In both custom software acquisition and development, specific risks are related to contracted or outsourced software development projects. Technical-related risks are found less often in acquisition and project management-related risks are found less often in development. There are generic risks, which are common to all software projects, in 13 risk categories of OTS-based software development and 14 risk categories of OTS-based software acquisition except in the risk category of OTS products. In this study, poor requirement definition and analysis, found in the requirements category, is the only generic risk found that was shared between OTS-based and custom-based software development and acquisition. RQ2 and RQ3 contributions: The survey and case study show that in OTS-based custom software projects comparing and discussing risk perceptions about risk control and impact between both stakeholders enhance our understanding of their risk perceptions and differences in their risk perceptions. In our survey and case study, we focused on 11 process-related risks that should be relevant not only to developers but also to acquirers in OTS-based custom software projects. There are four detailed findings about comparing and discussing risk perceptions between stakeholders.

134

 RQ2 contributions: There are different perceptions about risk occurrences in OTS-based custom software projects between the developer and acquirer perspectives (survey-based findings). The survey partially confirms a previous study [27], and complements it with findings about the acquirer perspective. The results of the survey focus on 11 process-related risks, and show that more of these risks occurred more frequently in software acquisition than in software development (nine vs. five risks out of these 11 risks; as shown in Table 4.11). Six risks occur more frequently in software acquisition than in software development: selection effort ill-estimated, not adaptable to requirement changes, requirements not negotiable, maintenance planning unfeasible, upgrade unfeasible and lack of information on vendor. Two risks occur more frequently in software development than in software acquisition: insufficient OTS component documents and a lack of vendor technical support and training. There are three risks that occurred equally frequently in both software development and acquisition: complicated multi-OTS component arrangement, lack of OTS-driven requirements engineering process, and reduced control of future evolution of the system.  RQ3 contributions: There are similarities and differences in respondents’ (developers and acquirers) perceptions about which stakeholder is affected by and can control risks (survey-based findings) Comparing the results of respondents’ perceptions about which stakeholder is affected by and can control risks have shown that both stakeholders can control, and are most affected by, risks about requirements negotiations. Developer respondents perceived themselves to best control risks, but perceived either themselves or both stakeholders to be most impacted by risks. Acquirer respondents agreed that their developers can best control risks, but perceived both stakeholders as most impacted by risks, except for risks of lack of support and reduced control of future evolution of the system. For the risk of lack of vendor technical support and training, there was disagreement about who is most impacted by the risk (usually each stakeholder reported themselves). With regard to the risk of reduced control of future evolution of the system both stakeholders agreed that the acquirer is most impacted. A comparison method was developed to analyse the survey results. We proposed a default position about which stakeholder should be responsible for risks, based on 135

respondents’ agreement about which stakeholders have high control of each risk. Responsibilities for all risks (except requirements not negotiable) can be proposed to belong to the developer. With regard to requirements not negotiable, we proposed that both stakeholders share responsibility for the risk. The proposed risk responsibility does not consider all potentially-related factors to allocate risk responsibility, but the method is consistent with responsibility models [81][78][34][35]. A responsibility model describes responsibilities within a system under development, agents assigned to these responsibilities and resources used to discharge these responsibilities [78]. In addition, a responsibility should also be assigned to the stakeholder who has the competence and capacity needed to discharge the responsibility [34][35]. Competence and knowledge affect the choice of control [82], while resource and capacity were needed to perform control over risks.  RQ3 contributions: We made 17 findings about level of risk control and impact, and proposed risk priority (case study-based findings) Based on the evidence from the cross-case analysis, it can be argued that each stakeholder has their own and shared perceptions regarding the risks of OTS-based custom software projects. The case study revealed 17 detailed findings related to levels of risk control and impact, and proposed risk priorities. There are five findings about risk perceptions specific to developers. Developers tend to perceive risks related to OTS vendors (such as a lack of technical support and training, a lack of vendor information, and OTS software documentation) as more important than the use of multi-OTS components. Developers also perceive risks related to requirements and OTS selection to be important. To control the impact of system maintenance, developers can often decide to use an existing stable version of OTS software. Eight findings about risk perceptions are specific to acquirers. Generally, acquirers tend to target higher control over a risk having high impact. Acquirers perceive risks related to requirements as more important than risks related to maintenance and upgrade. Acquirer can use OTS selection results to define system requirements. An acquirer can lower a risk’s impact by contracting an external party. For example, all support-related problems will be handled to an OTS software vendor. However, contracting a developer and using OTS software will give acquirers less control over the risks. An acquirer can 136

negotiate with an OTS vendor to manage the use of and support for OTS software. To anticipate future change, an acquirer can ask a developer to use a parameterised design (for example, setting configuration files). Four findings about risk perceptions relate to both stakeholders. Firstly, using a software contract to control risks leads to consequences regarding what must be agreed in the contract. As mentioned by a case study participant, its developer might not fully understand a contract because they might only use it to win a project. Next, developers and acquirers have different kinds of control over OTS software: developers tend to have limited control over OTS software in the OTS selection process and OTS integration because of not having the license of OTS software, while acquirers tend to have limited control over OTS software in maintenance because of limited access and knowledge over OTS vendor and software. Thirdly, there is a shared risk responsibility for agreement on OTS software selection, and the developer’s proposed system design decisions. Fourthly, both stakeholders have a high risk priority for the risk of lack of OTS-driven requirements engineering process, which is related to how requirements formulation is affected by the availability of OTS software and how requirements are mapped to OTS software [77]. The above findings show that developers and acquirers have different concerns about risk controls and responsibilities. Developers tend to focus on risks related to OTS vendors, OTS software integration, OTS selection and requirements. Acquirers tend to focus on risks related to requirements and OTS selection. A better understanding of these different concerns can help stakeholders to analyse related risks.  RQ3 contributions: The case study findings enhance our understanding of the importance of risks from developers’ and acquirers’ perspectives by taking account of the level of risk control and impact. The case study findings show that: the higher a risk control, the more important the risk; for risks with lower control, the higher the impact, the more important the risk is (this is further discussed in the next section).

137

RQ3 contributions: A proposed framework to analyse differences in risk perceptions and to assign risk responsibilities between developers and acquirers. We proposed a framework to analyse differences in risk perceptions and to assign risk responsibilities between a developer and acquirer. The framework was initially developed in our survey to analyse the survey data of differences in perceived risks between developers and acquirers (see Chapter 5). The framework extends stakeholder analysis [25][12][26][14][13][10] and can be used to compare, analyse and discuss level of risk control and impact between a developer and acquirer. The framework proposes a default position about which stakeholders should be responsible for a risk, based on which stakeholder has higher control over the risk. In our case study, the framework was used to facilitate a discussion between both stakeholders to assign a risk responsibility based on the proposed risk responsibility. Our findings show that all participants and their stakeholders agreed on proposed risk responsibilities and risk priorities. As confirmed by the case study findings, a responsibility should also be assigned to the stakeholder who has the competence and capacity needed to discharge the responsibility [34][35] and should be agreed between both stakeholders. In the case that stakeholders disagree with the proposed framework outputs and risk responsibilities, or the other stakeholder’s perspective, the discussion is useful to elicit the reason for this disagreement [24]. In the context of a specific project, the framework provides an explicit recognition of different risk perceptions, to inform risk management negotiation through dialogue, deliberation and communication [16]. The framework could help stakeholders reduce ‘gut feeling’ judgments and to ensure that decisions regarding the assignment of risk responsibilities are more objectively reviewed [140]. From a methodological perspective, another contribution of this research is to demonstrate that stakeholder analysis can be used to reconcile differences in risk perceptions between software project stakeholders. Previously, stakeholder analysis has only been used to identify stakeholders, to identify their roles, their level of involvement [36][37][13] and their risks [9][38][10], without comparing their different perceptions. Overall, the framework provides methodical and practical guidelines to reconcile differences in process and risks perspectives between developers and acquirers in OTSbased custom software projects. 138

7.2

Comparison with Related Work

The existing software acquisition standard, IEEE Standard 1062-1998 Edition: Recommended Practice for Software Acquisition [6], can be applied to software acquisition regardless of the size and complexity of the software [6]. However, this recommended practice is more applicable for fully developed software and must be tailored to other types of software acquisition [6]. From the literature, there are additional OTS-specific processes that must be

included in OTS-based software

development: make vs. buy decisions, OTS selection, OTS product familiarization, learning and understanding OTS products, OTS integration and vendor interaction [39][40][41].

As

software

projects

involve

and

interact

with

acquirers

[2][141][142][143][144], it is interesting to also investigate the role of acquirers in OTS-specific processes when acquiring OTS software. The findings for RQ1 identify six OTS-based software acquisition processes: decisionmaking to make or buy software, architectural decision-making, OTS selection, managing relationships between developers and acquirers, managing relationships between OTS adoption and acquirers’ organisations, and managing relationships between OTS vendors and developers. Our case study results and findings provide empirical evidence that these processes exist in practice. For example, the acquirer in the 1Dev case performed OTS software selection. There are three processes in common between OTS-based software acquisition and development processes: make vs. buy decisions, OTS selection and vendor interaction. The responsibilities of these common OTS-based software processes will vary depending on stakeholders’ goals and structures [19]. This research focuses on OTS-based process-related risks, as proposed in [77][74]. However, this research does not study risk mitigation. Nonetheless, our proposed framework needs acquirer involvement, which one of risk mitigations previously proposed for risk management of OTS-based custom software projects [74][27]. Instead of using a purely technical approach [145][76], we focus on human and social factors in risk management of OTS-based custom software projects. This research investigates risks from human and social perspectives [146][147]

by taking into

account developer and acquirer perspectives. To manage risks effectively, it is important for all software project stakeholders to understand the risks [16]. To initially 139

answer RQ2, our study identified and classified risks of custom and OTS-based software, from both perspectives. These results complement existing lists of software project risks, which are from the perspectives of project managers [30][58][57][71][59] and risk team members [22]. Our work is also in line with value-based software engineering because it deals with different stakeholders’ perspectives to overcome the limitations of a value-neutral approach [148][149]. An example of the value-neutral approach is when every requirements, use case, object and defect is treated as equally important [148][149]. In this study, perspectives of both stakeholders are investigated because risks can affect both stakeholders [8]. This research supports previous works [8][22][23] arguing that, to manage risks, key stakeholders must be involved from the beginning of software projects. However, there are several differences between this research and these prior works. Firstly, compared with Schmidt et al. [8], who argued to cooperatively manage risks, our research gives detailed procedures to asses differences in risk perceptions between both stakeholders. Secondly, prior research on control requires a ‘risk advisor’, whose tasks are to mediate between both stakeholders and to manage risk [22][23]. In contrast, our risk framework only includes developers and acquirers as the framework users, because a study by Keil et al. [66] noted that an ‘outside consultant’ did not identify more risks than insiders nor make more risk-averse decisions. To analyse differences in risk perceptions, we proposed a risk framework for answering RQ3. In our framework, different perceptions are compared and discussed to facilitate negotiation. This is a widely used approach in software engineering to reconcile conflicting perceptions [149]; it is used in risk management [22][23], software architecture [24] and requirement elicitation [150][149]. Risk assessment as a group task is more reasonable than as a task for an individual project manager because considering all stakeholders' perceptions increases stakeholder commitment and acceptance [22][23]. Heemstra et al. [22][23] proposed a collaborative risk assessment approach by first confronting the risk perceptions of each risk team member, discussing risk probabilities and effects, and finally agreeing on the most relevant risks and risk reduction actions, including risk responsibilities. In contrast with Heemstra et al. [22][23], our risk framework supports a structured discussion [24] using the framework outputs (stakeholder involvements, risk priorities, and proposed risk responsibilities) for 140

making risk responsibility decisions. The benefits of using structured discussion are that stakeholders can better understand and learn about other stakeholders’ perceptions, enabling them to make a more informed decision [24]. In requirements management, the EasyWinWin approach is a requirement elicitation and negotiation methodology, which supports expectation management, adopts prioritisation techniques and is supported with groupware tools [150][149]. There are two tactics from EasyWinWin relevant for our framework: structured procedures to handle conflicted stakeholder goals [150][149]; and the use of groupware to increase productivity of collaborative work [151]. Below we discuss how our framework compares and analyses differences in risk perception of OTS-based custom software projects between developers and acquirers for answering RQ3. In the case study (Table 6.5), four developers identified ‘collaborate’/high risk control and impact for three risks (selection effort ill-estimated, insufficient OTS-component documents and lack of information on vendor). It is interesting to compare these risks to the related risks in the survey of risk perceptions in the previous chapter. These case study results corroborate our survey results (detailed in Chapter 5), which showed that developer respondents perceived that they could best control and were most impacted by these risks. However, in the survey, the developer respondents perceived that for the risk of selection effort ill-estimated, both stakeholders were most impacted. This result is confirmed by our case study finding that when acquirers agree with developers on proposed OTS software, then the responsibility should be shared (detailed in Section 6.5.1.1). Regarding the risk of there being insufficient OTS component documents, our survey and case study findings are consistent with Mahmood and Khan’s [124] study reporting that component documentation is needed when integrating OTS software. In addition, from the developers’ point of view, our survey and case study findings support the previous studies [74][27] that pointed out the importance of monitoring information from OTS vendors regarding product upgrades and technical support, not only during development but also during maintenance. In our framework, the priority of a risk is based on the result of mapping it into a 2x2 control/impact matrix. Risk priority shows which risks are most important to address [30]. Instead of using risk-exposure quantity [30] or multiplication on ordinal scale [38], the framework prioritises risk using available information [38] by comparing values of 141

the mapping results from a developer and acquirer. In contrast with previous studies [16][59][128] reporting that stakeholders tend to perceive the importance of certain risks as higher than others if they cannot control the risks, our case study found that the higher control over a risk, the more important the risk. However, our framework corroborates previous studies [16][59][128] showing that for risks with lower control, the impact of the risks directly affects the importance of the risks. In short, the higher the impact, the more important the risk is. To analyse differences in risk perception of OTS-based custom software projects between developers and acquirers (RQ3), our framework extended an existing stakeholder analysis approach [25][12][26][14][13][10]. Even though using stakeholder analysis is considered important in requirements engineering for identifying stakeholder [37][36][152], our framework does not support stakeholder identification. We assume that the relevant stakeholders, a developer and an acquirer, have been identified. Instead of prioritising stakeholders (as in requirements elicitation) [37][153][154][155], our framework prioritises risks for each stakeholder. In addition, our framework maps levels of risk control and impact into kinds of stakeholder involvement using the control/impact matrix. Our framework can be used to identify stakeholders’ roles, and their level of involvement [36][37][13] and to understand the importance of risks. Our framework assigns a risk responsibility, based on stakeholder agreement, to the stakeholder who has higher risk control (RQ3). This is because higher control and power over risks is needed to control key decisions and act [31][32][33] to mitigate the risks. The case study results show that all participants and their stakeholders agreed on proposed risk responsibilities. Furthermore, our case study finds that the risk responsibility should also be assigned to the stakeholder who has knowledge, resources, and capacity to deal with risks. This confirms the responsibility model [34][35] which assigns a responsibility to the stakeholder who has the competences and capacity needed to discharge the responsibility. Knowledge can complement control [84], and competence and knowledge affect the choice of control [82], while resource and capacity are needed to perform a control. Our findings show that analysing project stakeholders can be integrated into software project risk management. Two examples are similar to our approach. Firstly, a Software Development Impact Statement (SoDIS) associates every software development task 142

with involved stakeholders and then uses structured questions to assess the specific project risks generated by that particular association [9]. Therefore, our framework could provide better justification in cases when SoDIS might assign more than one stakeholder to be responsible for a task-related risk. Secondly, the Outcome-Based Stakeholder Risk Assessment Model (OBSRAM) provides guidance in identifying and managing project risks arising from project stakeholders [10]. OBSRAM identifies stakeholder influence and project's impacts on stakeholders, and then assesses and prioritises stakeholder risks using risk scope, domain scope, risk severity and risk probability. Compared to OBSRAM, in which risk prioritisation does not consider stakeholder influence and project’s impacts on stakeholders, our framework prioritises risks based on stakeholders’ risk control and impact. There are similarities between our framework and two studies of role assignment in software development: role-based software development (RBSD) [156] and the Capabilities-Oriented Software Process Model [157][158]. The common steps between our framework and RBSD are negotiation and assignment of responsibility. However, RBSD uses more factors to assign roles (in our framework called responsibilities): rights, responsibilities, accessibilities, and collaboration methods. The CapabilitiesOriented Software Process Model assigns people to roles according to their capabilities, and capabilities demanded by the roles [157][158]. In The Capabilities-Oriented Software Process Model, the first step to assign a role starts by comparing a personal and role profile. The next step is identifying the closest match between the personal and role profiles. Another capability-based role assignment method uses Little-JIL to assign roles based on available and matching capabilities to perform a task [159].

A

mathematically-based formal model integrating the Delphi method could help project leaders to take into account as many factors as possible in assigning individuals to project roles [160]. It is also interesting to consider the use of network theory in the problem of task allocation for coordinating software development [161] in our future work. One of the aims for eliciting and reconciling differences in stakeholders’ perceptions is to support expectation management [150][149]. Our study on RQ3 found that participants had invalid expectations about each others’ risk perceptions. Therefore in

143

our future work, we would like to integrate our framework with stakeholders’ expectations, as in [150][149][38][64]. 7.3

Implications for Research

Differences in risk perceptions between software project stakeholders can be reconciled [16][20] by comparing and discussing [22][23] these differences, but prior research had not provided an explanation of how to involve stakeholders in this process. One approach that accounts for different stakeholder involvement in a project is stakeholder analysis [25][12][26][14][13][10]. In this thesis, we extended stakeholder analysis to compare, analyse and discuss stakeholders’ process-related risk perceptions. The findings provide a better understanding of risks in OTS-based custom software projects. 

The literature review indicated a lack of research on risks from software acquirer perspectives [7][15]. However, we have found that acquirers could have better control of, and be most impacted by risks, and so there could be more research into acquisition risks.



Further studies are needed to strengthen the risk mapping study results, for example by categorising risks using categories from the Software Engineering Institute’s Taxonomy of Software Development Risk [67].



This research compared perceptions of process-related risks between developers and acquirers. Research comparing other risk attributes (such as the probability of loss and the brief description of risks [29]) should also be conducted, because stakeholders might also differ in their perceptions of these risk attributes. Differences in risk attributes between stakeholders should be considered in collaborative risk management and risk negotiation.



This research shows that there are risks and processes related to OTS software vendors. Further study should further investigate these stakeholders, and the role they play in OTS-based software acquisition.



The proposed risk framework focuses assessing risks, but this is only one part of risk management. Research is also needed to address risk-management planning, risk resolution and risk monitoring.



The framework has shown potential use for assessing differences in perceptions of process-related risk. However, the case study participants also pointed out a potential use for the framework at the beginning of a project to assign roles for software processes. This purpose should be investigated in future research. 144



Further should validate that the framework can be generalised to other risks than the 11 risks examined in this thesis.



The framework assigns risk responsibility according to which stakeholder who has higher control over the risk. The framework does not consider all other potentiallyrelated factors for allocating risk responsibility. Further investigation on other related factors influencing risk responsibility is strongly recommended.

7.4

Implications for Practice

The findings of this research have a number of important implications for future practice. 

Our first mapping study identified six OTS-based acquisition processes. We suggest that for OTS-based software acquisition, changes should be considered for the existing software acquisition process standard [6] to include these processes, and for acquirers to consider these processes when acquiring OTS software.



Practical outcome of the second mapping study is a checklist of risks for risk identification in OTS-based custom software projects [126][66]. The checklist can be used to help identify more risks [66] and to assess the relevancy of risks [126][66].



The framework can help stakeholders to reduce reliance on ‘gut feeling’ judgments and to ensure decisions on the assignment of risk responsibilities are more objectively reviewed [140].



The framework used in this research may be able to be applied to other kinds of projects, to analyse differences in risk perceptions.

7.5

Limitations

Finally, a number of important limitations should be considered. 

Even though open source software (OSS) may present different risks, the case study found that the process-related risks were relevant to OSS. Therefore, we believe that process-related risks in this research are common to COTS, OTS, componentbased software and OSS.



Although the framework was designed to be generically applicable to collect, compare and analyse differences in risk perceptions, this capability has yet to be implemented outside OTS-based custom software projects. 145



In comparing levels of risk control and impact (low and high), there was a possibility of stakeholders’ subjective understanding of risk control and impact. To avoid this in our case study, we clarified and asked for stakeholders’ agreement on levels of risk control and impact (see Table 6.5).



This research analysed OTS-based custom software projects and differences in the perception of 11 process-related risks relevant to the developers and acquirers. This could limit generalisability of this research, because there are more risks in this context. However, the framework used to analyse these risks is not a risk identification technique, so that the framework can be used to analyse different risks other than the 11 risks.



To analyse differences in risk perceptions involving stakeholders other than developer and acquirer, the framework should be modified by adding the dimension of risk control/impact matrix, and re-validated.



Having numerous differences in the four cases (Table 6.4), the case study represented different contexts, which were expected to vary the sample of the population under study. However, case studies have a limited ability to generalise results to populations [86]; therefore, different cases.

146

the case study should be replicated in

Bibliography [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17]

[18]

C. L. Braun, ‘A lifecycle process for the effective reuse of commercial off-theshelf (COTS) software’, in Proceedings of the 1999 Symposium on Software reusability, Los Angeles, California, United States, 1999, pp. 29-36. J. Grudin, ‘Interactive systems: bridging the gaps between developers and users’, Computer, vol. 24, no. 4, pp. 59-69, 1991. M. Keil and A. Tiwana, ‘Relative importance of evaluation criteria for enterprise systems: a conjoint study’, Information Systems Journal, vol. 16, no. 3, pp. 237262, 2006. D. Carney, Assembling Large Systems from COTS Components: Opportunities, Cautions, and Complexities. SEI Monographs on Use of Commercial Software in Government Systems. Software Engineering Institute, Pittsburgh, USA, 1997. M. Torchiano and M. Morisio, ‘Overlooked aspects of COTS-based development’, IEEE Software, vol. 21, no. 2, pp. 88-93, 2004. IEEE, ‘IEEE recommended practice for software acquisition’, IEEE Std 1062, 1998 Edition, 1998. . E. Rosendahl and T. Vullinghs, ‘Performing Initial Risk Assessments in Software Acquisition Projects’, Software Quality — ECSQ 2002, 2002. C. Schmidt, P. Dart, L. Johnston, L. Sterling, and P. Thorne, ‘Disincentives for communicating risk: a risk paradox’, Information and Software Technology, vol. 41, no. 7, pp. 403-411, May 1999. D. W. Gotterbarn and S. Rogerson, ‘Responsible risk analysis for software development: creating the software development impact statement.’, Communications of AIS, vol. 2005, no. 15, pp. 730-750, 2005. R. W. Woolridge, D. J. McManus, and J. E. Hale, ‘Stakeholder Risk Assessment: An Outcome-Based Approach’, IEEE Software, vol. 24, no. 2, pp. 36-45, 2007. AS/NZS ISO 31000:2009 : Risk management - Principles and guidelines. Standards Australia, 2009. R. E. Freeman, Strategic management: A stakeholder approach, vol. 1. Pitman, 1984. E. A. Whitley and A. Pouloudi, ‘Stakeholder identification in interorganizational systems: gaining insights for drug use management systems’, European Journal of Information Systems, vol. 6, no. 1, pp. 1-14. J. Kaler, ‘Differentiating Stakeholder Theories’, Journal of Business Ethics, vol. 46, pp. 71-83, 2003. M. Jorgensen, ‘How to Avoid Selecting Bids Based on Overoptimistic Cost Estimates’, IEEE Software, vol. 26, no. 3, pp. 79-84, 2009. M. Keil, A. Tiwana, and A. Bush, ‘Reconciling user and project manager perceptions of IT project risk: a Delphi study1’, Information Systems Journal, vol. 12, no. 2, pp. 103-119, 2002. L. Zhu, R. Jeffery, M. Staples, M. Huo, and T. Tran, ‘Effects of Architecture and Technical Development Process on Micro-process’, in Software Process Dynamics and Agility, vol. 4470, Q. Wang, D. Pfahl, and D. Raffo, Eds. Springer Berlin / Heidelberg, 2007, pp. 49-60. S. Hornik, Houn-Gee Chen, G. Klein, and J. J. Jiang, ‘Communication skills of IS providers: an expectation gap analysis from three stakeholder perspectives’, 147

[19] [20] [21] [22] [23] [24] [25] [26] [27]

[28] [29] [30] [31] [32] [33] [34] [35]

IEEE Transactions on Professional Communication, vol. 46, no. 1, pp. 17- 34, 2003. R. Sabherwal, ‘The evolution of coordination in outsourced software development projects: a comparison of client and vendor perspectives’, Information and Organization, vol. 13, no. 3, pp. 153-202, Jul. 2003. S. Adolph, P. Kruchten, and W. Hall, ‘Reconciling perspectives: A grounded theory of how people manage the process of software development’, Journal of Systems and Software, vol. 85, no. 6, pp. 1269-1286, Jun. 2012. A. Bröckers, ‘Process-based software risk assessment’, in Software Process Technology, W. Schäfer, Ed. Springer Berlin Heidelberg, 1995, pp. 9-29. F. J. Heemstra and R. J. Kusters, ‘Dealing with risk: a practical approach’, Journal of Information Technology, vol. 11, no. 4, pp. 333-346, 1996. F. J. Heemstra, R. J. Kusters, and H. de Man, ‘Guidelines for managing bias in project risk management’, in 2003 Proceedings International Symposium on Empirical Software Engineering, 2003, pp. 272 - 280. M. Svahnberg and C. Wohlin, ‘Consensus Building when Comparing Software Architectures’, in Product Focused Software Process Improvement, M. Oivo and S. Komi-Sirviö, Eds. Springer Berlin Heidelberg, 2002, pp. 436-452. E. S. Andersen, K. V. Grude, and T. Haug, Goal Directed Project Management: Effective Techniques and Strategies, 4th ed. London: Kogan Page Business Book, 2009. A. L. Jepsen and P. Eskerod, ‘Stakeholder analysis in projects: Challenges in using current guidelines in the real world’, International Journal of Project Management, vol. 27, no. 4, pp. 335-343, May 2009. J. Li, O. P. N. Slyngstad, M. Torchiano, M. Morisio, and C. Bunse, ‘A State-ofthe-Practice Survey of Risk Management in Development with Off-the-Shelf Software Components’, IEEE Transactions on Software Engineering, vol. 34, no. 2, pp. 271-286, 2008. B. Boehm and C. Abts, ‘COTS integration: plug and pray?’, Computer, vol. 32, no. 1, pp. 135-138, 1999. F. D. Patterson and K. Neailey, ‘A Risk Register Database System to aid the management of project risk’, International Journal of Project Management, vol. 20, no. 5, pp. 365-374, Jul. 2002. B. W. Boehm, ‘Software risk management: principles and practices’, IEEE Software, vol. 8, no. 1, pp. 32-41, 1991. L. C. Ballejos and J. M. Montagna, ‘Method for stakeholder identification in interorganizational environments’, Requirements Eng, vol. 13, no. 4, pp. 281-297, Sep. 2008. L. W. Smith, ‘Project clarity through stakeholder analysis’, Crosstalk The Journal of Defense Software Engineering, p. December 2000. E. Manowong and S. Ogunlanas, ‘Strategies and Tactics for Managing Construction Stakeholders’, in Construction Stakeholder Management, WileyBlackwell, 2010, pp. 121-137. I. Sommerville, ‘Models for Responsibility Assignment’, in Responsibility and Dependable Systems, G. Dewsbury and J. Dobson, Eds. London: Springer London, 2007, pp. 165-186. I. Sommerville, ‘Causal Responsibility Models’, in Responsibility and Dependable Systems, G. Dewsbury and J. Dobson, Eds. London: Springer London, 2007, pp. 187-207. 148

[36] [37]

[38]

[39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53]

I. Alexander and S. Robertson, ‘Understanding project sociology by modeling stakeholders’, IEEE Software, vol. 21, no. 1, pp. 23-27, 2004. H. Sharp, A. Finkelstein, and G. Galal, ‘Stakeholder identification in the requirements engineering process’, in Proceedings of Tenth International Workshop on Database and Expert Systems Applications, 1999., 1999, pp. 387391. J. Kontio, G. Getto, and D. Landes, ‘Experiences in improving risk management processes using the concepts of the Riskit method’, in Proceedings of the 6th ACM SIGSOFT international symposium on Foundations of software engineering, Lake Buena Vista, Florida, United States, 1998, pp. 163-174. J. Li, F. Bjørnson, R. Conradi, and V. Kampenes, ‘An empirical study of variations in COTS-based software development processes in the Norwegian IT industry’, Empirical Software Engineering, vol. 11, no. 3, pp. 433-461, 2006. M. Morisio, C. B. Seaman, V. R. Basili, A. T. Parra, S. E. Kraft, and S. E. Condon, ‘COTS-based software development: Processes and open issues’, Journal of Systems and Software, vol. 61, no. 3, pp. 189-199, Apr. 2002. L. Brownsword, T. Oberndorf, and C. A. Sledge, ‘Developing new processes for COTS-based systems’, IEEE Software, vol. 17, no. 4, pp. 48-55, 2000. M. Mosko, H. Jiang, A. Samanta, and L. Werner, ‘COTS Software Acquisition Meta-Model’, 2000. P. Ulkuniemi and V. Seppanen, ‘Definition of a COTS software component acquisition process framework: the case of a telecommunications company’, 2002, pp. 48-54. M. Ochs, D. Pfahl, G. Chrobok-Diening, and B. Nothhelfer-Kolb, ‘A COTS Acquisition Process: Definition and Application Experience’, 2000, pp. 335–343. J. Baik, N. Eickelmann, and C. Abts, ‘Empirical software simulation for COTS glue code development and integration’, in Proceedings of 2001 Computer Software and Applications Conference, 2001, pp. 297-302. B. Boehm, D. Port, Ye Yang, and J. Bhuta, ‘Composable process elements for developing COTS-based applications’, in Proceedings of 2003 International Symposium on Empirical Software Engineering, 2003, 2003, pp. 8-17. T. Baker, ‘Lessons Learned Integrating COTS into Systems’, in COTS-Based Software Systems, vol. 2255, Springer Berlin / Heidelberg, 2002, pp. 21-30. M. Morisio and M. Torchiano, ‘Definition and Classification of COTS: A Proposal’, London, 2002, vol. Lecture Notes In Computer Science; Vol. 2255, pp. 165 - 175. D. Carney and F. Leng, ‘What do you mean by COTS? Finally, a useful answer’, IEEE Software, vol. 17, no. 2, pp. 83-86, 2000. ISO/IEC-IEEE, ‘ISO/IEC/IEEE Standard for Systems and Software Engineering - Software Life Cycle Processes’, IEEE STD 12207-2008, 2008. . T. Gantner and T. Häberlein, ‘GARP — The Evolution of a Software Acquisition Process Model’, Software Quality — ECSQ 2002, 2002. G. Getto, T. Gantner, and T. Vullinghs, ‘Software Acquisition: Experiences with Models and Methods’, 2000. K. Chaves Weber, E. E. Ramalho de Araujo, D. Scaler, E. L. Pereira de Andrade, A. R. Cavalcanti da Rocha, and M. A. Montoni, ‘MPS Model-Based Software Acquisition Process Improvement in Brazil’, in Proceedings of 6th International Conference on the Quality of Information and Communications Technology, 2007, pp. 110-122. 149

[54] [55] [56] [57] [58] [59] [60] [61] [62] [63]

[64]

[65] [66] [67] [68] [69]

M. A. Montoni, A. R. Rocha, and K. C. Weber, ‘MPS.BR: a successful program for software process improvement in Brazil’, Software Process: Improvement and Practice, vol. 14, no. 5, pp. 289-300, 2009. M. Ochs, D. Pfahl, G. Chrobok-Diening, and B. Nothhelfer-Kolb, ‘A COTS Acquisition Process: Definition and Application Experience’, 2000, pp. 335–343. B. W. Boehm and R. Ross, ‘Theory-W software project management principles and examples’, IEEE Transactions on Software Engineering, vol. 15, no. 7, pp. 902-916, 1989. H. Barki, S. Rivard, and J. Talbot, ‘Toward an Assessment of Software Development Risk’, Journal of Management Information Systems, vol. 10, no. 2, pp. 203-225, Oct. 1993. J. Ropponen and K. Lyytinen, ‘Components of software development risk: how to address them? A project manager survey’, IEEE Transactions onSoftware Engineering, vol. 26, no. 2, pp. 98-112, 2000. R. C. Schmidt, K. Lyytinen, M. Keil, and P. E. Cule, ‘Identifying software project risks: An international Delphi study’, Journal of Management Information Systems, vol. 17, no. 4, pp. 5-36, 2001. D. S. Kusumo, L. Zhu, M. Staples, and H. Zhang, ‘A Systematic Mapping Study on Off-The-Shelf-based Software Acquisition’, in Proceedings of ACIS 2011, Sydney, 2011. K. P. B. Webster, K. M. de Oliveira, and N. Anquetil, ‘A risk taxonomy proposal for software maintenance’, in Proceedings of the 21st IEEE International Conference on Software Maintenance, 2005. ICSM’05, 2005, pp. 453 - 461. T. Aven and V. Kristensen, ‘Perspectives on risk: review and discussion of the basis for establishing a unified and holistic approach’, Reliability Engineering & System Safety, vol. 90, no. 1, pp. 1-14, Oct. 2005. K. Georgieva, A. Farooq, and R. R. Dumke, ‘Analysis of the Risk Assessment Methods – A Survey’, in Software Process and Product Measurement, A. Abran, R. Braungarten, R. R. Dumke, J. J. Cuadrado-Gallego, and J. Brunekreef, Eds. Springer Berlin Heidelberg, 2009, pp. 76-86. B. Freimut, S. Hartkopf, P. Kaiser, J. Kontio, and W. Kobitzsch, ‘An industrial case study of implementing software risk management’, in Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering, Vienna, Austria, 2001, pp. 277-287. P. L. Bannerman, ‘Risk and risk management in software projects: A reassessment’, Journal of Systems and Software, vol. 81, no. 12, pp. 2118-2133, Dec. 2008. M. Keil, L. Li, L. Mathiassen, and G. Zheng, ‘The influence of checklists and roles on software practitioner risk perception and decision-making’, Journal of Systems and Software, vol. 81, no. 6, pp. 908-919, Jun. 2008. M. J. Carr and S. L. Konda, ‘Taxonomy-Based Risk Identification’, Software Engineering Institute, no. June, pp. 1-24, 1993. R. C. Williams, J. A. Walker, and A. J. Dorofee, ‘Putting risk management into practice’, IEEE Software, vol. 14, no. 3, pp. 75-82, 1997. B. Freimut, S. Hartkopf, P. Kaiser, J. Kontio, and W. Kobitzsch, ‘An industrial case study of implementing software risk management’, SIGSOFT Softw. Eng. Notes, vol. 26, no. 5, pp. 277–287, Sep. 2001. 150

[70] [71] [72] [73] [74] [75] [76] [77] [78]

[79] [80] [81] [82] [83] [84] [85]

[86] [87]

K.-S. Na, J. T. Simpson, X. Li, T. Singh, and K.-Y. Kim, ‘Software development risk and project performance measurement: Evidence in Korea’, Journal of Systems and Software, vol. 80, no. 4, pp. 596-605, Apr. 2007. T. Moynihan, ‘How experienced project managers assess risk’, IEEE Software, vol. 14, no. 3, pp. 35 -41, Jun. 1997. P. Brereton and D. Budgen, ‘Component-based systems: a classification of issues’, Computer, vol. 33, no. 11, pp. 54-62, 2000. P. Vitharana, ‘Risks and challenges of component-based software development’, Communication of the ACM, vol. 46, no. 8, pp. 67-72, 2003. L. Rose, ‘Risk Management of COTS Based Systems Development’, in Component-Based Software Quality, 2003. B. Boehm, D. Port, Y. Yang, and J. Bhuta, ‘Not All CBS Are Created Equally: COTS-Intensive Project Types’, COTS-Based Software Systems, 2003. Y. Yang, B. Boehm, and D. Wu, ‘COCOTS risk analyzer’, in Fifth International Conference on Commercial-off-the-Shelf (COTS)-Based Software Systems, 2006, 2006, p. 8 pp. G. Kotonya and A. Rashid, ‘A strategy for managing risk in component-based software development’, in Proceedings of 27th Euromicro Conference, 2001, pp. 12-21. I. Sommerville, R. Lock, T. Storer, and J. Dobson, ‘Deriving Information Requirements from Responsibility Models’, in Advanced Information Systems Engineering, vol. 5565, P. Eck, J. Gordijn, and R. Wieringa, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, pp. 515-529. S. Olander and A. Landin, ‘Evaluation of stakeholder influence in the implementation of construction projects’, International Journal of Project Management, vol. 23, no. 4, pp. 321-328, May 2005. G. M. Winch, ‘Managing Project Stakeholders’, in The Wiley Guide to Managing Projects, Hoboken, NJ, USA: John Wiley & Sons, Inc., 2004, pp. 321339. A. Blyth, ‘Using Stakeholders, Domain Knowledge, and Responsibilities to Specify Information Systems’ Requirements’, Journal of Organizational Computing and Electronic Commerce, vol. 9, no. 4, pp. 287-296, Dec. 1999. V. Choudhury and R. Sabherwal, ‘Portfolios of Control in Outsourced Software Development Projects’, Information Systems Research, vol. 14, no. 3, pp. 291314, 2003. A. Tiwana and M. Keil, ‘Control in Internal and Outsourced Software Projects’, Journal of Management Information Systems, vol. 26, no. 3, pp. 9-44, Dec. 2009. A. Tiwana and M. Keil, ‘Does peripheral knowledge complement control? An empirical test in technology outsourcing alliances’, Strategic Management Journal, vol. 28, no. 6, pp. 623–634, 2007. S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian, ‘Selecting Empirical Methods for Software Engineering Research’, in Guide to Advanced Empirical Software Engineering, F. Shull, J. Singer, and D. I. K. Sjøberg, Eds. London: Springer London, 2008, pp. 285-311. R. K. Yin, Case Study Research: Design and Methods, 4th ed. Los Angeles, CA: Sage Publications, Inc, 2008. C. B. Seaman, ‘Qualitative methods in empirical studies of software engineering’, IEEE Transactions on Software Engineering, vol. 25, no. 4, pp. 557-572, 1999. 151

[88] [89] [90] [91] [92] [93] [94] [95] [96] [97] [98] [99] [100] [101] [102] [103] [104]

[105]

[106]

J. Carver, C. B. Seaman, and R. Jeffery, ‘Using Qualitative Methods in Software Engineering’, Los Angeles, CA, USA, 2004. C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in software engineering: an introduction. Kluwer Academic Publishers, 2000. J. W. Creswell, Research design: Qualitative, quantitative, and mixed methods approaches, vol. 3rd. Sage Publications, 2009. D. Budgen, M. Turner, P. Brereton, and B. Kitchenham, ‘Using Mapping Studies in Software Engineering’, 2008, pp. 195–204. B. Kitchenham and S. Charters, ‘Guidelines for performing Systematic Literature Reviews in Software Engineering’, Keele University and Durham University Joint Report, EBSE 2007-001, 2007. K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, ‘Systematic Mapping Studies in Software Engineering’, University of Bari, Italy, 2008. ‘Zotero’, http://www.zotero.org/, 2011. R. Wieringa, N. Maiden, N. Mead, and C. Rolland, ‘Requirements engineering paper classification and evaluation criteria: a proposal and a discussion’, Requirements Eng, vol. 11, no. 1, pp. 102-107, Nov. 2005. CMMI Product Team, ‘CMMI for Acquisition, Version 1.2’, SEI, Carnegie Mellon, CMU/SEI-2007-TR-017 ESC-TR-2007-017, Nov. 2007. S. Elgazzar, A. Kark, E. Putrycz, and M. Vigder, ‘COTS Acquisition: Getting a Good Contract’, COTS-Based Software Systems, 2005. J. S. Seibel, T. A. Mazzuchi, and S. Sarkani, ‘Same vendor, version-to-version upgrade decision support model for commercial off-the-shelf productivity applications’, Systems Engineering, vol. 9, no. 4, pp. 296-312, 2006. J. Holck, M. K. Pedersen, and M. H. Larsen, ‘Open Source Software Acquisition: Beyond the Business Case’, presented at the ECIS 2005, Regensburg, Germany, 2005. L. Morgan and P. Finnegan, ‘Open innovation in secondary software firms: an exploration of managers’ perceptions of open source software’, SIGMIS Database, vol. 41, no. 1, pp. 76-95, 2010. L. Briand, S. Carrière, R. Kazman, and J. Wüst, ‘COMPARE: A Comprehensive Framework for Architecture Evaluation’, Object-Oriented Technology: ECOOP’98 Workshop Reader, 1998. J. Holck, M. H. Larsen, and M. K. Pedersen, ‘Managerial and Technical Barriers to the Adoption of Open Source Software’, COTS-Based Software Systems, 2005. C. Albert and L. Brownsword, ‘Meeting the Challenges of Commercial-Off-TheShelf (COTS) Products: The Information Technology Solutions Evolution Process (ITSEP)’, COTS-Based Software Systems, 2002. W. Aigner, P. Regner, T. Wiesinger, and J. Kung, ‘Supporting public software acquisition workflows - implications for data models’, in Proceedings of 15th International Workshop on the Database and Expert Systems Applications, 2004, pp. 1016-1022. M. Haglind, L. Johansson, and M. Rantzer, ‘Experiences integrating requirements engineering and business analysis. An empirical study of operations and management system procurement projects’, in Proceedings of Third International Conference on Requirements Engineering,1998, pp. 108-117. A. Heiskanen, M. Newman, and J. Similä, ‘The social dynamics of software development’, Accounting, Management and Information Technologies, vol. 10, no. 1, pp. 1-32, Jan. 2000. 152

[107] G. Shaffer and G. McPherson, ‘FAA COTS Risk Mitigation Guide: Practical Methods For Effective COTS Acquisition and Life Cycle Support’, Federal Aviation Administration, 2002. [108] L. D. Ball, I. G. Dambolena, and H. D. Hennessey, ‘Identifying early adopters of large software systems’, SIGMIS Database, vol. 19, no. 1, pp. 21-27, 1987. [109] T. Helokunnas and M. Nyby, ‘Collaboration between a COTS Integrator and Vendors’, Software Quality — ECSQ 2002, 2002. [110] E. Engström and P. Runeson, ‘Software product line testing – A systematic mapping study’, Information and Software Technology, vol. 53, no. 1, pp. 2-13, Jan. 2011. [111] EBSE RG, ‘Template for a Mapping Study Protocol’, http://www.dur.ac.uk/ebse/resources/templates/MappingStudyTemplate.pdf, Apr2009. [Accessed: 05-Mar-2011]. [112] B. A. Kitchenham, D. Budgen, and O. Pearl Brereton, ‘Using mapping studies as the basis for further research - A participant-observer case study’, Information and Software Technology, vol. 53, no. 6, pp. 638-651, Jun. 2011. [113] D. Greer and R. Conradi, ‘Software project initiation and planning - an empirical study’, IET Software, vol. 3, no. 5, pp. 356-368, 2009. [114] D. J. Reifer, V. R. Basili, B. W. Boehm, and B. Clark, ‘Eight lessons learned during COTS-based systems maintenance’, IEEE Software, vol. 20, no. 5, pp. 94 - 96, Oct. 2003. [115] D. Carney, S. A. Hissam, and D. Plakosh, ‘Complex COTS-based software systems: practical steps for their maintenance’, Journal of Software Maintenance: Research and Practice, vol. 12, no. 6, pp. 357–376, 2000. [116] Yi Ding and N. Napier, ‘Measurement Framework for Assessing Risks in Component-Based Software Development’, in Proceedings of the 39th Annual Hawaii International Conference on System Sciences, 2006, vol. 9, p. 230b. [118] N. Admodisastro and G. Kotonya, ‘Architectural Analysis Approaches: A Component-Based System Development Perspective’, in High Confidence Software Reuse in Large Systems, vol. 5030, H. Mei, Ed. Springer Berlin / Heidelberg, 2008, pp. 26-38. [119] Vu Tran and Dar-Biau Liu, ‘A risk-mitigating model for the development of reliable and maintainable large-scale commercial-off-the-shelf integrated software systems’, in Proceedings of Reliability and Maintainability Symposium, 1997, pp. 361-367. [120] L. Rose, ‘Risk Management of COTS Based Systems Development’, in Component-Based Software Quality, vol. 2693, Springer Berlin / Heidelberg, 2003, pp. 352-373. [121] D. Port and Y. Yang, ‘Empirical Analysis of COTS Activity Effort Sequences’, in COTS-Based Software Systems, vol. 2959, R. Kazman and D. Port, Eds. Springer Berlin / Heidelberg, 2004, pp. 169-182. [122] B. Boehm, D. Port, Y. Yang, and J. Bhuta, ‘Not All CBS Are Created Equally: COTS-Intensive Project Types’, in COTS-Based Software Systems, vol. 2580, H. Erdogmus and T. Weng, Eds. Springer Berlin / Heidelberg, 2003, pp. 36-50. [123] J. Li, R. Conradi, O. P. N. Slyngstad, M. Torchiano, M. Morisio, and C. Bunse, ‘Preliminary Results from a State-of-the-Practice Survey on Risk Management in Off-the-Shelf Component-Based Development’, in COTS-Based Software Systems, vol. 3412, X. Franch and D. Port, Eds. Springer Berlin / Heidelberg, 2005, pp. 278-288. 153

[124] S. Mahmood and A. Khan, ‘An industrial study on the importance of software component documentation: A system integrator’s perspective’, Information Processing Letters, vol. 111, no. 12, pp. 583-590, Jun. 2011. [125] L. Osterweil, ‘Unifying Microprocess and Macroprocess Research’, in Unifying the Software Process Spectrum, vol. 3840, M. Li, B. Boehm, and L. Osterweil, Eds. Springer Berlin / Heidelberg, 2006, pp. 68-74. [126] P. L. Bannerman, ‘Risk and risk management in software projects: A reassessment’, Journal of Systems and Software, vol. 81, no. 12, pp. 2118-2133, Dec. 2008. [127] D. Hillson, ‘Using a Risk Breakdown Structure in project management’, Journal of Facilities Management, vol. 2, no. 1, pp. 85-97, 2003. [128] M. Keil, P. E. Cule, K. Lyytinen, and R. C. Schmidt, ‘A framework for identifying software project risks’, Communication of the ACM, vol. 41, no. 11, pp. 76-83, 1998. [129] P. Runeson and M. Höst, ‘Guidelines for conducting and reporting case study research in software engineering’, Empir Software Eng, vol. 14, no. 2, pp. 131164, Dec. 2008. [130] L. A. Cox, ‘Limitations of Risk Assessment Using Risk Matrices’, in Risk Analysis of Complex and Uncertain Systems, vol. 129, Boston, MA: Springer US, 2009, pp. 101-124. [131] B. Kitchenham, L. Pickard, and S. L. Pfleeger, ‘Case studies for method and tool evaluation’, IEEE Software, vol. 12, no. 4, pp. 52-62, 1995. [132] J. E. Cook, L. G. Votta, and A. L. Wolf, ‘Cost-effective analysis of in-place software processes’, IEEE Transactions on Software Engineering, vol. 24, no. 8, pp. 650-663, 1998. [133] W. M. Garrabrants, A. W. Ellis, L. J. Hoffman, and M. Kamel, ‘CERTS: a comparative evaluation method for risk management methodologies and tools’, in Proceedings of the Sixth Annual Conference on Computer Security Applications, 1990, pp. 251-257. [134] S. Lichtenstein, ‘Factors in the selection of a risk assessment method’, Information Management & Computer Security, vol. 4, no. 4, pp. 20-25, Oct. 1996. [135] M. G. Mendonca and V. R. Basili, ‘Validation of an approach for improving existing measurement frameworks’, IEEE Transactions on Software Engineering, vol. 26, no. 6, pp. 484 -499, Jun. 2000. [136] K. M. Eisenhardt, ‘Building Theories from Case Study Research’, The Academy of Management Review, vol. 14, no. 4, pp. 532-550, Oct. 1989. [137] J. Voas, ‘Maintaining component-based systems’, IEEE Software, vol. 15, no. 4, pp. 22 -27, Aug. 1998. [138] Gero J.S. and Fujii H., ‘A computational framework for concept formation for a situated design agent’, Knowledge-Based Systems, vol. 13, no. 6, pp. 361-368, 2000. [139] K. C. Lam, D. Wang, P. T. K. Lee, and Y. T. Tsang, ‘Modelling risk allocation decision in construction contracts’, International Journal of Project Management, vol. 25, no. 5, pp. 485-493, Jul. 2007. [140] M. Jorgensen, ‘Practical guidelines for expert-judgment-based software effort estimation’, IEEE Software, vol. 22, no. 3, pp. 57-63, 2005. [141] M. Keil and E. Carmel, ‘Customer-developer links in software development’, Communication of the ACM, vol. 38, no. 5, pp. 33-44, 1995. 154

[142] P. Brereton, ‘The software customer/supplier relationship’, Communication of the ACM, vol. 47, no. 2, pp. 77-81, 2004. [143] S. Sawyer, ‘Software development teams’, Communication of the ACM, vol. 47, no. 12, pp. 95–99, Dec. 2004. [144] P. Waterson, S. Weibelzahl, and D. Pfahl, ‘Software Process Modelling’, in Software Process Modeling, D. S. T. Acuña and D. N. Juristo, Eds. Springer US, 2005, pp. 111-139. [145] K. Ballurio, B. Scalzo, and L. Rose, ‘Risk Reduction in COTS Software Selection with BASIS’, COTS-Based Software Systems, 2002. [146] M. John, F. Maurer, and B. Tessem, ‘Human and social factors of software engineering: workshop summary’, SIGSOFT Softw. Eng. Notes, vol. 30, no. 4, pp. 1–6, Jul. 2005. [147] H. Sharp and H. Robinson, ‘Some social factors of software engineering: the maverick, community and technical practices’, SIGSOFT Softw. Eng. Notes, vol. 30, no. 4, pp. 1–6, May 2005. [148] B. Boehm, ‘Value-based software engineering: reinventing’, SIGSOFT Softw. Eng. Notes, vol. 28, no. 2, p. 3–, Mar. 2003. [149] P. Grünbacher, S. Köszegi, and S. Biffl, ‘Stakeholder Value Proposition Elicitation and Reconciliation’, in Value-Based Software Engineering, S. Biffl, A. Aurum, B. Boehm, H. Erdogmus, and P. Grünbacher, Eds. Springer Berlin Heidelberg, 2006, pp. 133-154. [150] B. Boehm, P. Grunbacher, and R. O. Briggs, ‘Developing groupware for requirements negotiation: lessons learned’, IEEE Software, vol. 18, no. 3, pp. 46 55, May 2001. [151] J. Fjermestad and S. R. Hiltz, ‘Group Support Systems: A Descriptive Evaluation of Case and Field Studies’, Journal of Management Information Systems, vol. 17, no. 3, pp. 115-159, Dec. 2000. [152] L. C. Ballejos and J. M. Montagna, ‘Modeling stakeholders for information systems design processes’, Requirements Eng, vol. 16, no. 4, pp. 281-296, Nov. 2011. [153] S. L. Lim, D. Quercia, and A. Finkelstein, ‘StakeNet: using social networks to analyse the stakeholders of large-scale software projects’, in 2010 ACM/IEEE 32nd International Conference on Software Engineering, 2010, vol. 1, pp. 295 304. [154] S. L. Lim, D. Damian, and A. Finkelstein, ‘StakeSource2.0: using social networks of stakeholders to identify and prioritise requirements’, in 2011 33rd International Conference on Software Engineering (ICSE), 2011, pp. 1022 -1024. [155] S. L. Lim and A. Finkelstein, ‘StakeRare: Using Social Networks and Collaborative Filtering for Large-Scale Requirements Elicitation’, IEEE Transactions on Software Engineering, vol. 38, no. 3, pp. 707 -735, Jun. 2012. [156] H. Zhu, M. Zhou, and P. Seguin, ‘Supporting Software Development With Roles’, IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, vol. 36, no. 6, pp. 1110 -1123, Nov. 2006. [157] S. T. Acuña and N. Juristo, ‘Assigning people to roles in software projects’, Software: Practice and Experience, vol. 34, no. 7, pp. 675–696, 2004. [158] S. T. Acuna, N. Juristo, and A. M. Moreno, ‘Emphasizing human capabilities in software development’, IEEE Software, vol. 23, no. 2, pp. 94 - 101, Apr. 2006. [159] A. Wise, A. G. Cass, B. S. Lerner, E. K. McCall, L. J. Osterweil, and S. M. S. Jr, ‘Using Little-JIL to Coordinate Agents in Software Engineering’, in Engineering 155

of Software, P. L. Tarr and A. L. Wolf, Eds. Springer Berlin Heidelberg, 2011, pp. 383-397. [160] M. André, M. G. Baldoquín, and S. T. Acuña, ‘Formal model for assigning human resources to teams in software projects’, Information and Software Technology, vol. 53, no. 3, pp. 259-275, Mar. 2011. [161] C. Amrit, ‘Coordination in software development: the problem of task allocation’, SIGSOFT Softw. Eng. Notes, vol. 30, no. 4, pp. 1–7, May 2005.

156

Appendix A: Glossary Off-the-shelf (OTS): a commercially available or open source piece of software that other software projects can reuse and integrate into their own products. Acquirer: a person or organisation that acquires or procures a system or software product (which may be part of a system) from a supplier. Developer: a person or organisation that performs development activities (including requirements analysis, design, testing through acceptance) during the software life cycle process. Off-the-shelf-based (OTS) custom software project: a type of software projects consists of OTS-based software development and acquisition processes. OTS-based software development: software development that integrates OTS software to develop a system. For OTS-based custom software, development is performed to cover remaining requirements when OTS products cannot by themselves meet the requirements. OTS-based software acquisition: software acquisition of software that itself uses OTS software platforms or components. For OTS-based custom software, custom development is performed to cover remaining requirements when OTS products cannot by themselves meet the requirements. Risk control: actions implementing risk management. Risk impact: loss associated with not meeting the expected outcome. Risk perception: the judgment that people make when they asked to characterise and analyse risks.

157

Appendix B: Mapped papers for OTS-based software acquisition mapping study Mapped papers for software acquisition process categories Type of paper (total mapped papers/references) Empirical Experience Conceptual Standard framework Planning organizational 1 strategy [2] Implementing organization’s process Determining the software requirements Identifying potential 1 suppliers [11] Preparing contract requirements Evaluating proposals 5 and selecting the [15][32][28][29] supplier [31] Decision-making 3 2 1 [8][24][26] [9][18] [22] Process life cycle 10 2 2 [16][33] [6][19][20][17][34] [1][25] [4][36][37][38][39] Architectural decision 1 [3] Modelling and 3 2 simulation [5][23][30] [12][13] Managing relationship 2 1 between developer and [14][27] [21] acquirer Software acquisition 5 1 1 improvement [4][7][19][17][34] [10] [35] Process

158

Mapped papers for OTS-based software acquisition process categories Process Planning organizational strategy Implementing organization’s process Determining the software requirements Identifying potential suppliers Preparing contract requirements Evaluating proposals and selecting the supplier Decision-making Process life cycle Architectural decision OTS selection

Managing relationships between developers and acquirers Managing relationships between OTS adoption and acquirers’ organisations Managing relationships between OTS vendors and developers

Type of paper (total mapped papers/references) Empirical Experience Conceptual framework 1 [47] 1 [47] 1 13 [59] [50][51][60][72][74][75][80] [81][85][92][42][47][48] 1 [47] 1 [47] 2 [47][50] 5 2 [41][83][87] [56][65] [88][93] 3 5 [78][79][83] [55][44][63][95][94] 1 5 [53] [43][54][64][90][91] 6 25 [57][59][61] [45][49][51][52][54][58][65] [83][86][89] [66][67][69][70][71][72][73] [74][76][77][80][82][84][92] [42][48][62][68] 1 [53] 1 [40] 1 [46]

References: [1] J. Mogilensky, “Future approaches to software procurement,” Baltimore, Maryland, United States: ACM, 1990, pp. 543-550. [2] W. Richmond, P. Nelson, and S. Misra, “An empirical analysis of software life spans to determine the planning horizon for new software,” Information Technology and Management, vol. 7, Apr. 2006, pp. 131-149. 159

[3] L. Briand, S. Carrière, R. Kazman, and J. Wüst, “COMPARE: A Comprehensive Framework for Architecture Evaluation,” Object-Oriented Technology: ECOOP’98 Workshop Reader, 1998. [4] T. Gantner and T. Häberlein, “GARP — The Evolution of a Software Acquisition Process Model,” Software Quality — ECSQ 2002, 2006. [5] T. Häberlein and T. Gantner, “Process-Oriented Interactive Simulation of Software Acquisition Projects,” EurAsia-ICT 2002: Information and Communication Technology, 2002. [6] P. Regner, T. Wiesinger, J. Küng, and R. Wagner, “Software Acquisition Based on Business Models,” Electronic Government, 2004. [7] I. Garcia, C. Pacheco, and P. Sumano, “Use of Questionnaire-Based Appraisal to Improve the Software Acquisition Process in Small and Medium Enterprises,” Software Engineering Research, Management and Applications, 2008. [8] P. Gupta, S. Gould, and B. Pola, ““To Pirate or Not to Pirate”: A Comparative Study of the Ethical Versus Other Influences on the Consumer’s Software Acquisition-Mode Decision,” Journal of Business Ethics, vol. 55, Dec. 2004, pp. 255-274. [9] M.A. Serva, S.A. Sherer, and J.C. Sipior, ““When Do You ASP?” The Software Life Cycle Control Model,” Information Systems Frontiers, vol. 5, Apr. 2003, pp. 219-232. [10] L. Ibrahim and A. Pyster, “A single model for process improvement,” IT Professional, vol. 6, 2004, pp. 43-49. [11] D. Ladd, “A Software Procurement and Security Primer,” IEEE Security & Privacy, vol. 4, 2006, pp. 71-73. [12] H. Alfaraj and Shaowen Qin, “Business Process Modeling for Software Acquisition: A Literature Review,” 2008, pp. 1-3. [13] C. Conwell, R. Enright, and M. Stutzman, “Capability Maturity Models support of modeling and simulation verification, validation, and accreditation,” 2000, pp. 819-828 vol.1. [14] M. Haglind, L. Johansson, and M. Rantzer, “Experiences integrating requirements engineering and business analysis. An empirical study of operations and management system procurement projects,” 1998, pp. 108-117. [15] M. Jorgensen, “How to Avoid Selecting Bids Based on Overoptimistic Cost Estimates,” IEEE Software, vol. 26, 2009, pp. 79-84. [16] “IEEE Recommended Practice for Software Acquisition,” IEEE Standard 1062, 1998 Edition, 1998. [17] K. Chaves Weber, E. Ramalho de Araujo, D. Scaler, E. Pereira de Andrade, A. Cavalcanti da Rocha, and M. Montoni, “MPS Model-Based Software Acquisition Process Improvement in Brazil,” 2007, pp. 110-122. [18] C. Beath and G. Walker, “Outsourcing of application software: a knowledge management perspective,” 1998, pp. 666-674 vol.6. [19] Sha Wong, “Software acquisition management experience learnt in a multi discipline and multi contract project environment,” 2000, pp. 239-247. [20] B. Farbey and A. Finkelstein, “Software acquisition: a business strategy analysis,” 2001, pp. 76-83. [21] W. Aigner, P. Regner, T. Wiesinger, and J. Kung, “Supporting public software acquisition workflows - implications for data models,” 2004, pp. 1016-1022. [22] F. Shull, “Who Needs Evidence, Anyway?,” IEEE Software, vol. 24, 2007, pp. 10-11. [23] S.J. Choi and W. Scacchi, “Modeling and simulating software acquisition process 160

[24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43]

architectures,” Journal of Systems and Software, vol. 59, Dec. 2001, pp. 343-354. P. Nelson, A. Seidmann, and W. Richmond, “Software Acquisition: The Custom/Package and lnsource/Outsource Dimensions,” Elsevier, 1998, pp. 341367. J.E. Tardy, “Strategies for software acquisition,” Journal of Systems and Software, vol. 18, Jul. 1992, pp. 281-285. T. Rands, “The key role of applications software make-or-buy decisions,” The Journal of Strategic Information Systems, vol. 1, Sep. 1992, pp. 215-223. A. Heiskanen, M. Newman, and J. Similä, “The social dynamics of software development,” Accounting, Management and Information Technologies, vol. 10, Jan. 2000, pp. 1-32. E. Paschetta and A. Tsoukiàs, “A real-world MCDA application: evaluating software,” Journal of Multi-Criteria Decision Analysis, vol. 9, 2000, pp. 205-226. F. Fabbrini, M. Fusani, G. Lami, E. Sivera, and E. Sivera, “A SPICE-based software supplier qualification mechanism in automotive industry,” Software Process: Improvement and Practice, vol. 12, 2007, pp. 523-528. T. Häberlein, “Common structures in system dynamics models of software acquisition projects,” Software Process: Improvement and Practice, vol. 9, 2004, pp. 67-80. O. Demirors, E. Demirors, and A. Tarhan, “Managing instructional software acquisition,” Software Process: Improvement and Practice, vol. 6, 2001, pp. 189203. A.A. April and D. Al-Shurougi, “Software Product Measurement for Supplier Evaluation,” FESMA-AEMES Software Measurement Conference, 2000, pp. 18– 20. ISO/IEC-IEEE, “ISO/IEC/IEEE Standard for Systems and Software Engineering Software Life Cycle Processes,” IEEE Standard 12207-2008, 2008. M.A. Montoni, A.R. Rocha, and K.C. Weber, “MPS.BR: a successful program for software process improvement in Brazil,” Software Process: Improvement and Practice, vol. 14, 2009, pp. 289-300. CMMI Product Team, CMMI for Acquisition, Version 1.2, Carnegie Mellon: SEI, 2007. J. Schreiber, Beschaffung von Informatikmitteln. Pflichtenheft, Evaluation, Entscheidung, Paul Haupt Verlag, 2000. Software Program Managers Network, “Program Managers Guide to Software Acquisition.” “ISPL: Information Service Procurement Library. Managing Acquisition Processes,” 1999. The Opengroup, Beyond the Contract: An Analysis of the business impact of IT procurement best practices, 1999. L.D. Ball, I.G. Dambolena, and H.D. Hennessey, “Identifying early adopters of large software systems,” SIGMIS Database, vol. 19, 1987, pp. 21-27. L. Morgan and P. Finnegan, “Open innovation in secondary software firms: an exploration of managers' perceptions of open source software,” SIGMIS Database, vol. 41, 2010, pp. 76-95. N.A.M. Maiden, C. Ncube, and A. Moore, “Lessons learned during requirements acquisition for COTS systems,” Communication of the ACM, vol. 40, 1997, pp. 21-25. T. Ihme, “A Model for Recording Early-Stage Proposals and Decisions on Using COTS Components in Architecture,” COTS-Based Software Systems, 2003. 161

[44] R.J. Adams and S. Eslinger, “Best Practices for the Acquisition of COTS-Based Systems: Lessons Learned from the Space System Domain,” COTS-Based Software Systems, 2004. [45] F. Sudaman and C. Mingins, “BiCom: An Evaluation Framework for COTS Components,” COTS-Based Software Systems, 2003. [46] T. Helokunnas and M. Nyby, “Collaboration between a COTS Integrator and Vendors,” Software Quality — ECSQ 2002, 2006. [47] S. Elgazzar, A. Kark, E. Putrycz, and M. Vigder, “COTS Acquisition: Getting a Good Contract,” COTS-Based Software Systems, 2005. [48] V. Sai, “COTS Acquisition Evaluation Process: The Preacher’s Practice,” COTSBased Software Systems, 2003. [49] F. Ye and T. Kelly, “COTS Product Selection for Safety-Critical Systems,” COTSBased Software Systems, 2004. [50] S. Lauesen, “COTS tenders and integration requirements,” Requirements Engineering, vol. 11, Apr. 2006, pp. 111-122. [51] J. Pablo Carvallo, X. Franch, and C. Quer, “Defining a Quality Model for Mail Servers,” COTS-Based Software Systems, 2003. [52] A. Bader, C. Mingins, D. Bennett, and S. Ramakrishan, “Establishing Trust in COTS Components,” COTS-Based Software Systems, 2003. [53] J. Holck, M.H. Larsen, and M.K. Pedersen, “Managerial and Technical Barriers to the Adoption of Open Source Software,” COTS-Based Software Systems, 2005. [54] C. Albert and L. Brownsword, “Meeting the Challenges of Commercial-Off-TheShelf (COTS) Products: The Information Technology Solutions Evolution Process (ITSEP),” COTS-Based Software Systems, 2002. [55] R. Hornstein and J. Willoughby, “Realizing the Potential for COTS Utilization: A Work in Progress,” COTS-Based Software Systems, 2002. [56] M.D. Feblowitz and S.J. Greenspan, “Scenario-Based Analysis of COTS Acquisition Impacts,” Requirements Engineering, vol. 3, Mar. 1998, pp. 182-201. [57] I. Gorton and A. Liu, “Streamlining the Acquisition Process for Large-Scale COTS Middleware Components,” COTS-Based Software Systems, 2002. [58] C. Ncube and J. Dean, “The Limitations of Current Decision-Making Techniques in the Procurement of COTS Software Components,” COTS-Based Software Systems, 2002. [59] B.H. Far, V. Mudigonda, and A.H. Elamy, “A General Purpose Software Evaluation System,” 2008, pp. 290-295. [60] D. Rickman, “A new process for requirements understanding,” 2001, pp. 4D4/14D4/6 vol.1. [61] A. Liu and I. Gorton, “Accelerating COTS middleware acquisition: the i-Mate process,” IEEE Software, vol. 20, 2003, pp. 72-79. [62] N. Maiden and C. Ncube, “Acquiring COTS software selection requirements,” IEEE Software, vol. 15, 1998, pp. 46-56. [63] V. Tran, “Component-based integrated systems development: a model for the emerging procurement-centric approach to software development,” 1998, pp. 128135. [64] W. Lemahieu, M. Snoeck, F. Goethals, M. De Backer, R. Haesen, J. Vandenbulcke, and G. Dedene, “Coordinating COTS applications via a business event layer,” IEEE Software, vol. 22, 2005, pp. 28-35. [65] N. Schneidewind, “Cost framework for COTS evaluation,” 1999, pp. 100-101. [66] P. Ulkuniemi and V. Seppanen, “COTS component acquisition in an emerging market,” IEEE Software, vol. 21, 2004, pp. 76-82. 162

[67] S. Lauesen, “COTS tenders and integration requirements,” 2004, pp. 166-175. [68] P. Ulkuniemi and V. Seppanen, “Definition of a COTS software component acquisition process framework: the case of a telecommunications company,” 2002, pp. 48-54. [69] R. Kohl, “Establishing guidelines for suitability of COTS for a mission critical application,” 1999, pp. 98-99. [70] Jinfang Sheng and Bin Wang, “Evaluating COTS Components Using Gap Analysis,” 2008, pp. 1248-1253. [71] X. Illa, X. Franch, and J. Pastor, “Formalising ERP selection criteria,” 2000, pp. 115-122. [72] C. Ncube and N. Maiden, “Guiding parallel requirements acquisition and COTS software selection,” 1999, pp. 133-140. [73] F. Navarrete, P. Botella, and X. Franch, “How agile COTS selection methods are (and can be)?,” 2005, pp. 160-167. [74] J. Carvallo and X. Franch, “On the Use of Requirements for Driving Call-forTender Processes for Procuring Coarse-grained OTS Components,” 2009, pp. 287292. [75] E. Hitt, “Rebuilding the requirements process for aging avionics,” 2000, pp. 4A3/1-4A3/6 vol.1. [76] F. Navarrete, P. Botella, and X. Franch, “Reconciling Agility and Discipline in COTS Selection Processes,” 2007, pp. 103-113. [77] B. Kizzort, “Selection of components for OTS component-based systems,” 2002, pp. 6-2651-6-2659 vol.6. [78] J. Verville and A. Halingten, “A six-stage model of the buying process for ERP software,” Industrial Marketing Management, vol. 32, Oct. 2003, pp. 585-594. [79] P. Poon and Y.T. Yu, “Investigating ERP systems procurement practice: Hong Kong and Australian experiences,” Information and Software Technology, vol. In Press, Corrected Proof. [80] C. Rolland, “Requirements engineering for COTS based systems,” Information and Software Technology, vol. 41, Nov. 1999, pp. 985-990. [81] A. Bruseberg, “The design of complete systems: Providing human factors guidance for COTS acquisition,” Reliability Engineering & System Safety, vol. 91, Dec. 2006, pp. 1554-1565. [82] M. Wybo, J. Robert, and P. Léger, “Using search theory to determine an applications selection strategy,” Information & Management, vol. 46, Jun. 2009, pp. 285-293. [83] C.P. Salter and D.M. Buede, “A lifecycle-based method for the acquisition of commercial-off-the-shelf (COTS) technology to support organizational processes,” Systems Engineering, vol. 4, 2001, pp. 287-304. [84] J. Miller and H.C. Yeoh, “COTS acquisition process: incorporating business factors into COTS vendor evaluation taxonomies,” Software Process: Improvement and Practice, vol. 11, 2006, pp. 601-626. [85] L. Chung and K. Cooper, “Defining goals in a COTS-aware requirements engineering approach,” Systems Engineering, vol. 7, 2004, pp. 61-83. [86] A. Mohamed, G. Ruhe, and A. Eberlein, “Optimized mismatch resolution for COTS selection,” Software Process: Improvement and Practice, vol. 13, 2008, pp. 157-169. [87] M. Keil and A. Tiwana, “Relative importance of evaluation criteria for enterprise systems: a conjoint study,” Information Systems Journal, vol. 16, 2006, pp. 237262. 163

[88] J.S. Seibel, T.A. Mazzuchi, and S. Sarkani, “Same vendor, version-to-version upgrade decision support model for commercial off-the-shelf productivity applications,” Systems Engineering, vol. 9, 2006, pp. 296-312. [89] M. Ochs, D. Pfahl, G. Chrobok-Diening, and B. Nothhelfer-Kolb, “A COTS Acquisition Process: Definition and Application Experience,” 11th ESCOM Conference, Publications, 2000, pp. 335–343. [90] M. Vidger, An Architecture for COTS Based Software, National Research Council Canada Institute for Information Technology, 1998. [91] M. Vigder, Inspecting COTS Based Software Systems. Verifying an Architecture to Support Management of Long-Lived Software Systems, National Research Council Canada Institute for Information Technology, 1998. [92] C. Ncube and N.A.M.P. Maiden, “PORE: Procurement-oriented requirements engineering method for the component-based systems engineering development paradigm,” 1999, pp. 1–12. [93] J. Holck, M.K. Pedersen, and M.H. Larsen, “Open Source Software Acquisition: Beyond the Business Case,” Regensburg, Germany: 2005. [94] M. Mosko, H. Jiang, A. Samanta, and L. Werner, “COTS Software Acquisition Meta-Model,” Proceedings of COTS Workshop, 2000. [95] B.C. Meyers and P. Oberndorf, Managing software acquisition: open systems and COTS products, Addison-Wesley Longman Ltd., 2001.

164

Appendix C: Mapped papers for risks of OTS-based and custom software development and acquisition For each risk category, there is a possibility that a paper was mapped to a risk category more than one, because the paper had more than one risk factor. For example, in “Planning” risk category for risks of OTS-based software acquisition, paper [17] has two risk factors: underestimated cost estimation and multi-OTS products from different vendors may result in complicated licensing arrangements. Mapped papers for risks of OTS-based software development and acquisition Risk category

Risks of OTS-based software development (total mapped papers/references) Empirical Experience Critical framework Planning 2 2 [45][53] [5][36] Requirements 24 6 5 [12][4][54][4][54][52][54][46] [55][14][8][22] [5][36][48][57][23] [52][53][44][49][45][51][52][53] [14][15] [4][54][52][53][53][53][53][53] Design 2 4 [14][55] [2][11][13][50] Integration and 19 3 5 testing [4][54][39][35][53][4][54][6] [55][20][14] [41][50][56][3][5] [52][53][45][4][54][4][54][4] [54][4][54] System lifecycle 1 3 [1][19][38] [53] 165

Risks of OTS-based software acquisition (total mapped papers/references) Empirical Experience Critical framework 3 [17][17][36] 2 2 3 [28][44] [59][22] [17][36][17]

4 [17][17][17][36]

Maintenance Project closure

7 [4][54][53][52][4][54][53]

1 [55]

5 [5][37][5][3][3]

Software OTS product

7 [52][6][6][49][6][49][53]

Cost Environment People Systems Engineering Vendor relationships Project management Contract Legal

5 [55][16][18] [18][10]

2 [6][6] 2 [6][52]

2 [16][16] 2 [55][16]

6 [54][4][53][4][54][53]

2 [55][55]

10 [3][3][3][7][21] [36][40][50][36][50] 1 [3]

3 [9][9][9] 1 [36]

1 [30]

166

1 [54]

1 [28] 1 [28] 2 [28][28] 2 [28] [28] 1 [28]

3 [17][36][17] 2 [32][60] 2 [17][36] 1 [32] 1 [17] 1 [17] 3 [36][60][60] 1 [17]

Mapped papers for risks of custom-based software development and acquisition Risk category Planning Requirements Design Integration testing System lifecycle

Risks of custom-based software development (total mapped papers/references) Empirical Experience Critical framework 1 [25] 5 1 1 [29][24][42][24][42] [58] [31] 1 [42] and 1 [26]

Risks of custom-based software acquisition (total mapped papers/references) Empirical Experience Critical framework 2 1 [61][33] [31] 1 1 [42] [25] 1 [31] 1 [31] 1 [31]

Project closure Software Cost Environment People Vendor relationships Project management

1 [26] 1 [42] 6 2 [24][42][24][24][26][26] [25][25] 4 [26][26][26][42]

1 [31]

1 [43] 2 [27][27]

1 [31]

1 [42] 3 [27][27][27]

167

2 [25][47]

1 [34]

Contract

1 [24]

Legal

1 [27]

1 [27] 1 [27]

References: U. Lindqvist and E. Jonsson, ‘A map of security risks associated with using COTS’, Computer, vol. 31, no. 6, pp. 60-66, 1998. Guijun Wang and G. Cone, ‘A method to reduce risks in building distributed enterprise systems’, in Proceedings of Fifth IEEE International on the Enterprise Distributed Object Computing Conference, 2001, pp. 164-168. Vu Tran and Dar-Biau Liu, ‘A risk-mitigating model for the development of reliable and maintainable large-scale commercial-off-the[3] shelf integrated software systems’, in Proceedings of the Reliability and Maintainability Symposium 1997, pp. 361-367. [4] Jingyue Li, O. P. N. Slyngstad, M. Torchiano, M. Morisio, and C. Bunse, ‘A State-of-the-Practice Survey of Risk Management in Development with Off-the-Shelf Software Components’, IEEE Transactions on Software Engineering, vol. 34, no. 2, pp. 271-286, 2008. [5] G. Kotonya and A. Rashid, ‘A strategy for managing risk in component-based software development’, in Proceedings of 27th Euromicro Conference, 2001, pp. 12-21. J. Bhuta, S. Mallick, and S. V. Subrahmanya, ‘A Survey of Enterprise Software Development Risks in a Flat World’, in Proceedings of [6] First International Symposium on Empirical Software Engineering and Measurement, 2007, pp. 476-478. [7] J. D. Smith, ‘An Alternative to Technology Readiness Levels for Non-Developmental Item (NDI) Software’, in Proceedings of the 38th Annual Hawaii International Conference on the System Sciences, 2005, p. 315a. [8] V. N. Tran and D.-B. Lin, ‘Application of CBSE to projects with evolving requirements-a lesson-learned’, in Proceedings of Sixth Asia Pacific Software Engineering Conference, 1999, pp. 28-37. [9] B. M. Horowitz and J. H. Lambert, ‘Assembling off-the-shelf components: “Learn as you Go” systems engineering’, IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, vol. 36, no. 2, pp. 286-297, 2006. [10] R. Fulton, ‘Assuring certifiability of outsourced software development - a DER’S perspective’, in Proceedings of 24th Digital Avionics Systems Conference, 2005, vol. 2, p. 5 pp. Vol. 2. [11] B. Boehm and J. Bhuta, ‘Balancing Opportunities and Risks in Component-Based Software Development’, IEEE Software, vol. 25, no. 6, pp. 56-63, 2008. [1] [2]

168

[12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24]

P. Gruenbacher, ‘Collaborative requirements negotiation with EasyWinWin’, in Proceedings of 11th International Workshop on the Database and Expert Systems Applications, 2000, pp. 954-958. A. Egyed, N. Medvidovic, and C. Gacek, ‘Component-based perspective on software-mismatch detection and resolution’, IEEE Software, vol. 147, no. 6, pp. 225-236, 2000. V. Tran, Dar-Biau Liu, and B. Hummel, ‘Component-based systems development: challenges and lessons learned’, in Proceedings of Eighth IEEE International Workshop on the Software Technology and Engineering Practice, 1997. [incorporating Computer Aided Software Engineering], 1997, pp. 452-462. J. Akkanen, A. J. Kiss, and J. K. Nurminen, ‘Evolution of a software component - experiences with a network editor component’, in Proceedings of Sixth European Conference on the Software Maintenance and Reengineering, 2002, pp. 119-125. W. Lam and A. J. Vickers, ‘Managing the risks of component-based software engineering’, in Proceedings of Fifth International Symposium on the Assessment of Software Tools and Technologies, 1997, pp. 123-132. Yi Ding and N. Napier, ‘Measurement Framework for Assessing Risks in Component-Based Software Development’, in Proceedings of the 39th Annual Hawaii International Conference on System Sciences, 2006, vol. 9, p. 230b. P. Maki-Asiala and M. Matinlassi, ‘Quality Assurance of Open Source Components: Integrator Point of View’, in Proceedings of 30th Annual International Computer Software and Applications Conference, 2006, vol. 2, pp. 189-194. R. J. Ellison and C. Woody, ‘Supply-Chain Risk Management: Incorporating Security into Software Development’, in Proceedings of 43rd Hawaii International Conference on System Sciences, 2010, pp. 1-10. S. Blom, M. Book, V. Gruhn, and R. Laue, ‘Switch or Struggle: Risk Assessment for Late Integration of COTS Components’, in Proceedings of Second International Workshop on the Incorporating COTS Software into Software Systems: Tools and Techniques, 2007, p. 1. L. Merola, ‘The COTS software obsolescence threat’, in Proceedings of Fifth International Conference on Commercial-off-the-Shelf (COTS)-Based Software Systems, 2006., p. 7 pp. V. Tran, B. Hummel, Dar-Biau Liu, Thu Anh Le, and J. Doan, ‘Understanding and managing the relationship between requirement changes and product constraints in component-based software projects’, in Proceedings of the Thirty-First Hawaii International Conference on the System Sciences, 1998, vol. 6, pp. 132-142 vol.6. Y. Yang, Jesal Bhuta, B. Boehm, and D. N. Port, ‘Value-based processes for COTS-based applications’, IEEE Software, vol. 22, no. 4, pp. 54-62, 2005. Jiangping Wan, Dejie Li, and K. Kuang, ‘Analysis of the Business Risks for the Software Outsourcing between Hongkong and Guangdong’, in Proceedings of 4th International Conference on Wireless Communications, Networking and Mobile Computing, 2008, pp. 1-4. 169

[25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41]

C. M. Lott, ‘Breathing new life into the waterfall model’, IEEE Software, vol. 14, no. 5, pp. 103-105, 1997. T. Moynihan, ‘How experienced project managers assess risk’, IEEE Software, vol. 14, no. 3, pp. 35-41, 1997. B. A. Aubert, S. Dussault, M. Patry, and S. Rivard, ‘Managing the risk of IT outsourcing’, in Proceedings of the 32nd Annual Hawaii International Conference on System Sciences, 1999, vol. Track7, p. 10 pp. L. M. Abdullah and J. M. Verner, ‘Outsourced strategic IT systems development risk’, in Proceedings of Third International Conference on the Research Challenges in Information Science, 2009, pp. 275-286. S. Serich, ‘Prototype stopping rules in software development projects’, IEEE Transactions on Engineering Management, vol. 52, no. 4, pp. 478-485, 2005. T. R. Huber, ‘Reducing business and legal risks in software reuse libraries’, in Proceedings of Third International Conference on Software Reuse: Advances in Software Reusability, 1994, pp. 110-117. Xiangnan Lu and Yali Ge, ‘Risk analysis in project of software development’, in Proceedings of the Engineering Management Conference, Managing Technologically Driven Organizations: The Human Side of Innovation and Change, 2003, pp. 72-75. J. A. McDermid, ‘COTS: the expensive solution?’, in Proceedings of IEE Colloquium on the COTS and Safety Critical Systems (Digest No. 1997/013), 1997, pp. 1/1-1/4. M. Jorgensen, ‘How to Avoid Selecting Bids Based on Overoptimistic Cost Estimates’, IEEE Software, vol. 26, no. 3, pp. 79-84, 2009. H. M. Edwards, G. M. Mallalieu, and J. B. Thompson, ‘Some insights into the maintenance of legacy systems within small manufacturing and distribution organisations in the UK’, in Proceedings of The Twenty-Third Annual International of Computer Software and Applications Conference, 1999, pp. 13-20. A. A. Kvale, J. Li, and R. Conradi, ‘A case study on building COTS-based system using aspect-oriented programming’, Santa Fe, New Mexico, 2005, pp. 1491-1498. P. Vitharana, ‘Risks and challenges of component-based software development’, Communication of the ACM, vol. 46, no. 8, pp. 67-72, 2003. A. Stuckenholz and A. Osterloh, ‘Safe component updates’, Portland, Oregon, USA, 2006, pp. 39-48. G. Brændeland and K. Stølen, ‘Using model-based security analysis in component-oriented system development’, Alexandria, Virginia, USA, 2006, pp. 11-18. S. Mahmood and A. Khan, ‘An industrial study on the importance of software component documentation: A system integrator’s perspective’, Information Processing Letters, vol. 111, no. 12, pp. 583-590, Jun. 2011. F. Redmill, ‘Analysis of the COTS debate’, Safety Science, vol. 42, no. 5, pp. 355-367, Jun. 2004. M. Hepner, R. Gamble, M. Kelkar, L. Davis, and D. Flagg, ‘Patterns of conflict among software components’, Journal of Systems and Software, vol. 79, no. 4, pp. 537-551, Apr. 2006. 170

[42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55]

C. Schmidt, P. Dart, L. Johnston, L. Sterling, and P. Thorne, ‘Disincentives for communicating risk: a risk paradox’, Information and Software Technology, vol. 41, no. 7, pp. 403-411, May 1999. S. A. Sherer, ‘Purchasing software systems : Managing the risk’, Information & Management, vol. 24, no. 5, pp. 257-266, 1993. J. Miller and H. C. Yeoh, ‘COTS acquisition process: incorporating business factors into COTS vendor evaluation taxonomies’, Software Process Improvement Practice, vol. 11, no. 6, pp. 601-626, 2006. Y. Yang and B. Boehm, ‘Improving process decisions in COTS-based development via risk-based prioritization’, Software Process Improvement Practice, vol. 12, no. 5, pp. 449-460, 2007. A. Mohamed, G. Ruhe, and A. Eberlein, ‘Optimized mismatch resolution for COTS selection’, Software Process Improvement Practice., vol. 13, no. 2, pp. 157-169, 2008. H. Saiedian, ‘Practical recommendations to minimize software capability evaluation risks’, Software Process Improvement Practice vol. 8, no. 3, pp. 145-156, 2003. G. Brændeland and K. Stølen, ‘A Semantic Paradigm for Component-Based Specification Integrating a Notion of Security Risk’, in Formal Aspects in Security and Trust, vol. 4691, T. Dimitrakos, F. Martinelli, P. Ryan, and S. Schneider, Eds. Springer Berlin / Heidelberg, 2007, pp. 31-46. J. Li, F. Bjørnson, R. Conradi, and V. Kampenes, ‘An empirical study of variations in COTS-based software development processes in the Norwegian IT industry’, Empirical Software Engineering, vol. 11, pp. 433-461, 2006. N. Admodisastro and G. Kotonya, ‘Architectural Analysis Approaches: A Component-Based System Development Perspective’, in High Confidence Software Reuse in Large Systems, vol. 5030, H. Mei, Ed. Springer Berlin / Heidelberg, 2008, pp. 26-38. D. Port and S. Chen, ‘Assessing COTS Assessment: How Much Is Enough?’, in COTS-Based Software Systems, vol. 2959, R. Kazman and D. Port, Eds. Springer Berlin / Heidelberg, 2004, pp. 183-198. D. Port and Y. Yang, ‘Empirical Analysis of COTS Activity Effort Sequences’, in COTS-Based Software Systems, vol. 2959, R. Kazman and D. Port, Eds. Springer Berlin / Heidelberg, 2004, pp. 169-182. B. Boehm, D. Port, Y. Yang, and J. Bhuta, ‘Not All CBS Are Created Equally: COTS-Intensive Project Types’, in COTS-Based Software Systems, vol. 2580, H. Erdogmus and T. Weng, Eds. Springer Berlin / Heidelberg, 2003, pp. 36-50. J. Li, R. Conradi, O. P. N. Slyngstad, M. Torchiano, M. Morisio, and C. Bunse, ‘Preliminary Results from a State-of-the-Practice Survey on Risk Management in Off-the-Shelf Component-Based Development’, in COTS-Based Software Systems, vol. 3412, X. Franch and D. Port, Eds. Springer Berlin / Heidelberg, 2005, pp. 278-288. L. Rose, ‘Risk Management of COTS Based Systems Development’, in Component-Based Software Quality, vol. 2693, Springer Berlin / Heidelberg, 2003, pp. 352-373. 171

[56] [57] [58] [59] [60] [61]

K. Ballurio, B. Scalzo, and L. Rose, ‘Risk Reduction in COTS Software Selection with BASIS’, in COTS-Based Software Systems, vol. 2255, J. Dean and A. Gravel, Eds. Springer Berlin / Heidelberg, 2002, pp. 31-43. D. Wu and Y. Yang, ‘Towards an Approach for Security Risk Analysis in COTS Based Development’, in Software Process Change, vol. 3966, Q. Wang, D. Pfahl, D. Raffo, and P. Wernick, Eds. Springer Berlin / Heidelberg, 2006, pp. 124-131. M. Feather and S. Cornford, ‘Quantitative risk-based requirements reasoning’, Requirements Engineering, vol. 8, pp. 248-265, 2003. S. Elgazzar, A. Kark, E. Putrycz, and M. Vigder, ‘COTS Acquisition: Getting a Good Contract’, in COTS-Based Software Systems, vol. 3412, X. Franch and D. Port, Eds. Springer Berlin / Heidelberg, 2005, pp. 43-53. M. A. Serva, S. A. Sherer, and J. C. Sipior, ‘“When Do You ASP?” The Software Life Cycle Control Model’, Information Systems Frontiers, vol. 5, pp. 219-232, 2003. E. Rosendahl and T. Vullinghs, ‘Performing Initial Risk Assessments in Software Acquisition Projects’, in Software Quality — ECSQ 2002, vol. 2349, J. Kontio and R. Conradi, Eds. Springer Berlin / Heidelberg, 2006, pp. 146-155.

172

Appendix D: The framework inputs The participant of case 1Dev is the developer (the acquirer is the stakeholder of the participant), the participant of case 2AcqBank is the acquirer (the developer is the stakeholder of the participant), the participant of case 3AcqTelco is the acquirer (the developer is the stakeholder of the participant), the participant of case 4AcqGovt is the acquirer (the developer is the stakeholder of the participant). Input of the framework Risk Item Selection effort ill-estimated

Case 1Dev Developer RI RC Hi Hi Lo Hi (was Hi)

Case 2AcqBank Acquirer Developer Acquirer RI RC RI RC RI RC Lo Lo Hi Hi Hi Hi

Case 3AcqTelco Developer Acquirer RI RC RI RC Hi Hi Hi Hi

Case 4AcqGovt Developer Acquirer RI RC RI RC Hi Hi Hi Hi

Hi

Hi

Lo

Lo

Hi

Hi

Hi

Hi

Lo

Lo

Hi

Hi

Hi

Hi

Requirements not negotiable

Hi

Hi

Hi

Hi

Lo

Lo

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Lo

Hi (was Lo)

Complicated multi-OTS components arrangement

Hi

Lo

Hi

Hi

Hi

Lo

Hi

Lo

Lo

Lo

Hi

Hi

Lo

Hi

Lo

Lo

Hi

Hi (was Lo)

Hi

Hi

Hi

Hi

Hi

Lo

Hi

Hi

Lo

Lo

Hi

Hi

Lo

Hi (was Lo)

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Lo

Hi

Lo

Hi

Lo

Hi (was

Hi

Lo

Lo

Lo

Lo

Lo

Hi

Hi

Lo (was

Lo (was

Lo

Hi

Lo

Hi

Not adaptable to requirement changes

Insufficient OTS component documents Lack of OTS-driven requirements engineering process Maintenance planning unfeasible

173

Lo)

Hi)

Hi)

Upgrade unfeasible

Lo

Lo

Lo

Hi

Lo

Lo

Lo

Lo

Lo

Lo

Lo

Lo

Hi

Hi

Lo

Lack of information on provider

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Lo

Lack of support

Hi

Hi Lo (was Hi)

Hi (was Lo) Lo

Lo

Hi

Hi

Hi

Lo

Lo

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Hi

Reduced control of future evolution of the system

Hi

Lo

Hi

Hi

Lo

Lo

Lo

Lo

Hi

Hi

Hi

Hi

Lo

Hi

Hi (was Lo)

Lo

Legend: RI: risk impact, RC: risk control, Hi: high, Lo: low.

174

Appendix E: Cross-case analysis for data collection 1 and 2, excluded inputs to the framework and evaluation criteria for risk assessment Question Q1.2.1

Participant’s answers in Case 1Dev Shared understanding

Case 2AcqBank Shared understanding

Case 3AcqTelco Meeting the requirements

Shared responsibility Q1.2.2

Stakeholder commitment Discussion

Q1.2.3

Both developer and stakeholder

Q1.2.4

A project manager and system analyst

Q2.2.1

Brainstorming and discussion

Focusing and prioritising tasks No formal analysis

User Acceptance Testing (UAT) Proof of Concept (POC) before implementation Both project managers (of developer and acquirer) Member of Proof of Concept (POC) and operation Different approaches and perceptions between participant and its developer

175

Case 4AcqGovt Managing risk of development

Both developer and stakeholder Web development and coordinator of application implementation Collective risk planning and developer involvement in OTS component selection

Agreement based on a scheduled meeting of tasks of the project. Acquirer A project coordinator Scheduled meeting and no formal risk management

Q2.2.2 and Q2.2.3

Using past project data Risk identification: brainstorming and discussion Risk analysis and prioritization: discussion and using past project data to rank risk

Q2.2.4

Risk identification: brainstorming and discussion Risk analysis and prioritization: planning and review meeting

The one who has responsibility on risks

Stakeholder having knowledge (on a risk)

Defined in Scope of Work

Defined in the project contract

Risk outside responsibility of acquirer and developer must be discussed with respective stakeholder Q2.2.5

Documented in the project contract (not including detailed technical aspects)

Detailing Scope of Work (after using the framework)

Defined in software contract

176

Integrated with budget management Based on cost/revenue analysis Risk identification: brainstorming and discussion

Risk identification: brainstorming and discussion Risk analysis and prioritization: follow-up based on the discussion

Risk analysis: what-if scenario Risk prioritization: based on audit, loss revenue and cost Defined in the software contract

Technical and nontechnical understanding of risk

Acquirer organization's policy about stakeholder role Communication to achieve and responsibility on system common understanding development Resource and capacity Negotiation Terms of reference Describing level of each Common understanding of stakeholder and shared the definition of control. responsibility (to avoid tendency to determine Detailed communication responsibility to other instruments, e.g. meeting

stakeholder)

Q2.2.6

The use of the framework as risk expectation and assessment method

The use of the framework to understand risk control, impact and responsibility

Adding cost/benefit analysis and comparison The use of the framework to understand risk control, impact and responsibility The use of the framework as risk expectation and assessment method

and Terms of reference

Common understanding of the definition of control and impact (of processes and risks)

The use of the framework to check relationships between risks

Q2.2.7

After Design Review Meeting

The beginning of the project

In the project tender (The use of the framework in the project tender is better than after design review meeting) Q2.2.8.a

Cannot predict what the other stakeholder’s opinions about risk control and impact

Cannot predict what the other stakeholder’s opinions about risk control and impact because of different interests 177

The expectation to modify the input of the framework The beginning of the project (requirement and OTS selection) To integrate to existing risk management, the framework should be added with other attribute (e.g. cost) Cannot predict what the other stakeholder’s opinions about risk control and impact

It can be integrated to software process

Cannot predict what the other stakeholder’s opinions about risk control and impact unless both

Q2.2.8.b

This framework (in Chapter 6) is better compared with the method used in Chapter 5 Using the framework, stakeholder should be able to determine risk control

This framework (in Chapter 6) is better compared with the method used in Chapter 5 because the framework compared actual risk perceptions

178

This framework (in Chapter 6) is better compared with the method used in Chapter 5 because the framework compared actual risk perceptions and use simple level of risk control and impact

stakeholders have a common understanding This framework (in Chapter 6) is better compared with the method used in Chapter 5 because the framework provided structured, methodical and measured approach to analyse risk control and impact