Software Release Decisions

The NPVI Method to Support Market Entry Strategies for Software Dr. Hans Sassenburg ([email protected])

© SE-CURE AG

1

Question Would you fly to Mars with the software of your organization?

© SE-CURE AG

2

Contents 1. 2. 3. 4. 5. 6. 7. 8.

Introduction Overview Methodology Release Definition Release Information Release Decision Release Implementation Methodogy out into Practice Conclusions © SE-CURE AG

3

1. Introduction Society‘s increasing software dependence: – Software-based systems replace older technologies in safety- or mission-critical applications. – Software moves from an auxiliary to a primary role in providing critical services. – Software becomes the only way of performing some function which is not perceived as critical but whose failures would deeply affect individuals or groups. – Software-provided services become increasingly an accepted part of everyday life without any special scrutiny. – Software-based systems are increasingly integrated and interacting, often without effective human control. (source: Littlewood and Strigini 2000) © SE-CURE AG

4

Some Facts • Software size and amount/variety of software applications increases exponentially • Increased size leads to – higher defect potentials – Lower defect removal efficiencies – and thus: Higher defect densities at release time

• Conclusion: – Importance of software release decisions is likely to increase

• Trade-off: – between an early release, to capture the benefits of an earlier market introduction, and the deferral of product release, to enhance functionality, or improve quality © SE-CURE AG

5

Defect Density as a Function of Size

Source: Jones 2002 © SE-CURE AG

6

Recalls in Automotive Industry 160

Number of official recalls

140 120 100 Germany

80

UK

60 40 20 0 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 Year

(source: KBA 2003) © SE-CURE AG

7

Cost of Quality Design change six months before SOP Long-term quality problems Production costs 10% over target Six months late reaching full production capacity 10% of customers lost to competitors Market introduction delayed six months 0

2

4

6

8

10

100 Mio. Euro © SE-CURE AG

8

Cost Distributions 20% Total costs

Requirements

Analyse

Design

10%

Defects

Realisation and Testing 15%

30% + 45%

Operations / Maintenance

Product Release

= Defect repairs

+

Sources: Boehm, Jones, Pfleeger, Fjeldstad...

80% Total costs

20%

(Learning software takes: >= 50%)

New enhancements

New Requirements

Product Release

80%

End-of-Life

Case Studies (1)

Important: you cannot test quality into a product! © SE-CURE AG

10

Case Studies (2)

Conclusion (Weinberg): a maintenance programmer is more like a brain surgeon than a janitor. © SE-CURE AG

11

When to Stop Testing? • A ‘sufficient’ percentage of test cases run successfully • Statistics are gathered about what code is exercised during the execution of a test suite • Defects are classified and numbers and trends are analysed • Real users conduct beta testing and report problems that are analysed • Developers analyse the number of reported problems in a certain period of time. When the number stabilizes, or remains below a certain threshold, the software is considered ‘good enough’.

This is reality: is it good for safety-critical systems? Alternative? © SE-CURE AG

Confidence • Depends on system’s purpose, user expectations and marketing environment – Software function • The level of confidence depends on how critical the software is to an organisation

– User expectations • Users may have low expectations of certain kinds of software

– Marketing environment • Getting a product to market early may be more important than finding defects in the program

© SE-CURE AG

Good Enough Quality • •

Defined by James Bach (IEEE Computer, 1997, vol. 30, no. 8, pp. 96-98) To claim that any given thing is good enough is to agree with allof the following propositions: – – – –

• •

It has sufficient benefits It has no critical problems The benefits sufficiently outweigh the problems In the present situation, and all things considered, further improvement would be more harmful than helpful

Aiming to Good Enough Quality does not necessarily allow us to do less quality practices We must find the critical problems and at least be able to estimate the risks of the remaining (unknown) problems

© SE-CURE AG

2. Overview Methodology Release Decision

Release Information The optimal information level is sought, recognizing that information has a price in time and cost

imperfect information

Presence of different preferences regarding the decision outcome may exist; bargaining techniques are used to arrive at a choice

accepted release decision

environmental constraints congruence?

Release Definition Managerial objectives are set, monitored, and used to compare and evaluate alternatives, hereby reducing uncertainty

Release Implementation expected outcome

actual outcome

© SE-CURE AG

Decision success is ensured through follow-up and control; an appraisal is conducted to increase organizational learning

15

3. Release Definition • Method developed using – life-cycle model semiconductor industry – extended with operational cost

• NPV as capital budgeting method – to include time-value of money

© SE-CURE AG

16

Saturation

W Maturity

Growth

Introduction

Cmax

W

W © SE-CURE AG

Decline

C

Simple Product Life-Cycle

2W

Time 17

Cost

Extended Model with Cost Functions

Imax Mmax

Peak development cost

Peak operational cost

Tr

Tr + W

© SE-CURE AG

Tr + 2W

Time

18

Asset value, cost

Resulting Profit Model

Break Even

Profit

Total Cost Asset value

Tr

Tr + W © SE-CURE AG

Tr + 2W

Time 19

C

Delayed Market Entry

Peak asset value

Cmax'

Market rise

Market fall

Cmax'

D Tr

Tr+W

© SE-CURE AG

Tr+2W

Time

20

Cost

Delayed Market Entry

Imax' Mmax'

Tr

Tr + D

Tr + W

© SE-CURE AG

Tr + 2W

Time

21

Break Even

Revenue, Cost

Resulting Profit Model

Profit

Total Cost Asset value

Tr

Tr + D

Tr + W © SE-CURE AG

Tr + 2W

Time 22

Example Tr = 50 weeks W = 50 weeks

D = 0 wk

D = 2.5 wk

D = 5 wk

D = 7.5 wk

D = 10 wk

Asset value (C)

-

-7%

-14%

-21%

-28%

Development cost (I)

-

10%

21%

32%

44%

Operational cost (M)

-

-7%

-14%

-21%

-28%

Net asset value (NAV) (Cm = 10, Im = 5, Mm = 5)

-

-25%

-50%

-75%

-100%

© SE-CURE AG

23

NPVI Method 0

T

end of life Operations

Development

I

C, M

NPV

=

-I + (C – M) / (1 + r) T

• NPVI = Net Present Value Incentive – Calculates difference in two NPVs from underlying metrics: premiums and advantages © SE-CURE AG

24

How it Works 1. Determine Base strategy b •

Cb, Ib, Mb, Tb, rb

2. Determine Test strategy a, relative to Base strategy b •

Ia, Ta, ra

3. Calculate Ca and Ma from: • • • • •

Early Entry Premium Product Functionality Premium Product Reliability Premium Short-term Maintenance Premium Long-term Maintenance Premium

© SE-CURE AG

(EEP) (PFP) (PRP) (SMP) (LMP)

25

© SE-CURE AG

PVI Present Value Incentive

OCA Operational Cost Advantage

( = log Mb - log Ma )

DCA Development Cost Advantage ( = log Ib - log Ia )

LMP Long-term Maintenance Premium

AVA Asset Value Advantage ( = log Ca - log Cb )

SMP Short-term Maintenance Premium

PRP Product Reliability Premium

PFP Product Functionality Premium

EEP Early Entry Premium

DTA Development Time Advantage ( = log Tb - log Ta )

NPVI Net Present Value Incentive

DCI Development Cost Incentive

NAVA Net Asset Value Advantage

26

4. Release Information • From case studies – High presence of uncertainty – both for routine and nonroutine market entry decisions

• Reactive behaviour regarding information collection – during testing phase: too late

• Result – market window: unknown, but high time pressure – reliability: probably ‘good enough‘ – maintainability: unknown

• Conclusions – EEP, PFP, PRP, SMP, LMP unknown – limited value of NPVI method

© SE-CURE AG

27

Value, Cost, Yield, Time

Information has a Price in Cost/Time T(i) C(i)

V(i) V* Y*

T*

Y(i)

C* 0

I*

100% 100 Information perfection

© SE-CURE AG

28

Sources of Information 1. Market

C(t)

2.a Product Reliability

I(t)

2.b Product Maintainability

M(t)

Information

3. Market <-> Product

© SE-CURE AG

C(t) = f (I(t), M(t))

29

Yield

Zone of Cost Effectiveness

Economic perspective

+ +

Uncertainty pre-release development cost post-release maintenance cost

Low maturity

+ -

High maturity

zone of cost effectiveness © SE-CURE AG

Information perfection

30

5. Release Decision • Decision-making – Principle of bounded rationality leads to simplifications of the real world (source: Simon 1957) – Presence of sources of group conflicts

• Result – Decision-makers in a collective decision-making process are likely to have different initial positions (aspiration levels) © SE-CURE AG

31

Decision-making Model quality of decision inputs

+

quality of decisionmaking process

Uncertainty - imperfect information - cognitive limitations

Sociological aspects - negotiated decision strategy - interacting group type - criteria for effective groups - consensus as decision rule

Group conflict - diverse goals - diverse heuristic values

Political aspects - management of meaning: high - challenge: low

© SE-CURE AG

=

quality of decision outcome

Satisficing decision behaviour - adopt alternative that is acceptable to all stakeholders - ensure that decision is understood by stakeholders - involve stakeholders responsible for implementation

32

6. Release Implementation “A decision brilliantly conceived can prove worthless without effective implementation.” (source: Shull et al. 1970)

• Key issues for decision implementation success – Involve Maintenance as a stakeholder from the proposal phase throughout product development – Discharge Development not before the released product functions correctly in its operational environment © SE-CURE AG

33

7. Methodology put into Practice Release Definition

Release Implementation Project Objectives Project Appraisal Project Discharge

H

Project Control Uncertainty Management

M

Product Rollout

Selection of Alternatives

L

Maintenance Budget

Verification Definition

Decision Choice

Verification Implementation

Stakeholder Involvement

Artefact Definition

Aspiration Levels

Artefact Implementation

Information Perfection

Release Decision

Release Information

© SE-CURE AG

34

7. Conclusions 1. Importance of software release decisions likely to increase 2. Release Definition • NPVI method powerful to compare and evaluate different market entry strategies 4. Release Information • Three main sources of information (C, I, M) • Information level must be within zone of cost effectiveness • High maturity organizations are able to obtain more valuable information against less time and cost 5. Release Decision • Decision success determined by information level and stakeholder positions 6. Release Implementation • Strong co-operation between Development and Maintenance – –

Maintenance is stakeholder from project beginning Development will be discharged after successful release implementation

7. Methodology • Has predictive character © SE-CURE AG

35

Further Information Please contact: Hans Sassenburg E: [email protected] W: www.se-cure.ch

© SE-CURE AG

36