Ch21
This document was uploaded by user and they confirmed that they have the permission to share it. If you are author or own the copyright of this book, please report to us by using this DMCA report form. Report DMCA
Overview
Download & View Ch21 as PDF for free.
More details
- Words: 3,659
- Pages: 50
Critical Systems Validation Validating the reliability, safety and security of computerbased systems
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 1
Validation perspectives ●
Reliability validation • •
●
Safety validation •
●
Does the measured reliability of the system meet its specification? Is the reliability of the system good enough to satisfy users? Does the system always operate in such a way that accidents do not occur or that accident consequences are minimised?
Security validation •
Is the system and its data secure against external attack?
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 2
Validation techniques ●
Static techniques • •
●
Dynamic techniques • • •
●
Design reviews and program inspections Mathematical arguments and proof Statistical testing Scenariobased testing Runtime checking
Process validation •
Design development processes that minimise the chances of process errors that might compromise the dependability of the system
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 3
Static validation techniques ●
●
●
Static validation is concerned with analyses of the system documentation (requirements, design, code, test data). It is concerned with finding errors in the system and identifying potential problems that may arise during system execution. Documents may be prepared (structured arguments, mathematical proofs, etc.) to support the static validation
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 4
Static techniques for safety validation ●
●
Demonstrating safety by testing is difficult because testing is intended to demonstrate what the system does in a particular situation. Testing all possible operational situations is impossible Normal reviews for correctness may be supplemented by specific techniques that are intended to focus on checking that unsafe situations never arise
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 5
Safety reviews ● ●
●
●
●
Review for correct intended function Review for maintainable, understandable structure Review to verify algorithm and data structure design against specification Review to check code consistency with algorithm and data structure design Review adequacy of system testing
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 6
Review guidance ● ●
●
●
Make software as simple as possible Use simple techniques for software development avoiding errorprone constructs such as pointers and recursion Use information hiding to localise the effect of any data corruption Make appropriate use of faulttolerant techniques but do not be seduced into thinking that fault tolerant software is necessarily safe
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 7
Hazarddriven analysis ●
●
Effective safety assurance relies on hazard identification (covered in previous lectures) Safety can be assured by • • •
●
Hazard avoidance Accident avoidance Protection systems
Safety reviews should demonstrate that one or more of these techniques have been applied to all identified hazards
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 8
The system safety case ●
●
●
It is now normal practice for a formal safety case to be required for all safetycritical computerbased systems e.g. railway signalling, air traffic control, etc. A safety case presents a list of arguments, based on identified hazards, why there is an acceptably low probability that these hazards will not result in an accident Arguments can be based on formal proof, design rationale, safety proofs, etc. Process factors may also be included ©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 9
Formal methods and critical systems ●
●
●
The development of critical systems is one of the ‘success’ stories for formal methods Formal methods are mandated in Britain for the development of some types of safetycritical software for defence applications There is not currently general agreement on the value of formal methods in critical systems development
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 10
Formal methods and validation ●
Specification validation • •
●
Developing a formal model of a system requirements specification forces a detailed analysis of that specification and this usually reveals errors and omissions Mathematical analysis of the formal specification is possible and this also discovers specification problems
Formal verification •
Mathematical arguments (at varying degrees of rigour) are used to demonstrate that a program or a design is consistent with its formal specification
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 11
Problems with formal validation ●
The formal model of the specification is not understandable by domain experts • •
●
It is difficult or impossible to check if the formal model is an accurate representation of the specification for most systems A consistently wrong specification is not useful!
Verification does not scaleup •
Verification is complex, errorprone and requires the use of systems such as theorem provers. The cost of verification increases exponentially as the system size increases.
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 12
Formal methods conclusion ●
Formal specification and checking of critical system components is, in my view, useful •
●
While formality does not provide any guarantees, it helps to increase confidence in the system by demonstrating that some classes of error are not present
Formal verification is only likely to be used for very small, critical, system components •
About 56000 lines of code seems to be the upper limit for practical verification
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 13
Safety proofs ●
●
●
Safety proofs are intended to show that the system cannot reach in unsafe state Weaker than correctness proofs which must show that the system code conforms to its specification Generally based on proof by contradiction • •
●
Assume that an unsafe state can be reached Show that this is contradicted by the program code
May be displayed graphically
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 14
Construction of a safety proof ●
●
● ●
Establish the safe exit conditions for a component or a program Starting from the END of the code, work backwards until you have identified all paths that lead to the exit of the code Assume that the exit condition is false Show that, for each path leading to the exit that the assignments made in that path contradict the assumption of an unsafe exit from the component
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 15
Gas warning system ●
●
System to warn of poisonous gas. Consists of a sensor, a controller and an alarm Two levels of gas are hazardous • •
●
Warning level no immediate danger but take action to reduce level Evacuate level immediate danger. Evacuate the area
The controller takes air samples, computes the gas level and then decides whether or not the alarm should be activated
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 16
Gas sensor control Gas_level: GL_TYPE ; loop Take 100 samples of air Gas_level := 0.000 ; for i in 1..100 loop Gas_level := Gas_level + Gas_sensor.Read ; end loop ; Gas_level := Gas_level / 100 ; if Gas_level > Warning and Gas_level < Danger then Alarm := Warning ; Wait_for_reset ; elsif Gas_level > Danger then Alarm := Evacuate ; Wait_for_reset ; else Alarm := off ; end if ; end loop ; ©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 17
Graphical argument Unsafe state
Gas_level > Warning and Alarm = off
Path 1
or
Gas_level > Warning and Gas_level < Danger
Alarm = Warning
contradiction
©Ian Sommerville 2000
or Path 2
or Path 3
Gas_level > Danger
Alarm = Evacuate
Alarm = off
contradiction
CS 365 Critical Systems Validation
Slide 18
Condition checking Gas_level < Warning Gas_level = Warning Gas_level > Warning and Gas_level < Danger Gas_level = Danger Gas_level > Danger
Path 3 Path 3 Path 1 Path 3 Path 2
Alarm = off (Contradiction) Alarm = off (Contradiction) Alarm = Warning (Contradiction) Alarm = off Alarm = Evacuate (Contradiction)
Code is incorrect. Gas_level = Danger does not cause the alarm to be on
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 19
Key points ●
●
●
Safetyrelated systems should be developed to be as simple as possible using ‘safe’ development techniques Safety assurance may depend on ‘trusted’ development processes and specific development techniques such as the use of formal methods and safety proofs Safety proofs are easier than proofs of consistency or correctness. They must demonstrate that the system cannot reach an unsafe state. Usually proofs by contradiction
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 20
Dynamic validation techniques ●
These are techniques that are concerned with validating the system in execution • •
Testing techniques analysing the system outside of its operational environment Runtime checking checking during execution that the system is operating within a dependability ‘envelope’
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 21
Reliability validation ●
●
●
Reliability validation involves exercising the program to assess whether or not it has reached the required level of reliability Cannot be included as part of a normal defect testing process because data for defect testing is (usually) atypical of actual usage data Statistical testing must be used where a statistically significant data sample based on simulated usage is used to assess the reliability
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 22
Statistical testing ● ●
●
Testing software for reliability rather than fault detection Measuring the number of errors allows the reliability of the software to be predicted. Note that, for statistical reasons, more errors than are allowed for in the reliability specification must be induced An acceptable level of reliability should be specified and the software tested and amended until that level of reliability is reached
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 23
Reliability validation process ● ●
●
●
Establish the operational profile for the system Construct test data reflecting the operational profile Test the system and observe the number of failures and the times of these failures Compute the reliability after a statistically significant number of failures have been observed
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 24
Operational profiles ●
●
An operational profile is a set of test data whose frequency matches the actual frequency of these inputs from ‘normal’ usage of the system. A close match with actual usage is necessary otherwise the measured reliability will not be reflected in the actual usage of the system Can be generated from real data collected from an existing system or (more often) depends on assumptions made about the pattern of usage of a system
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 25
N u m b e r of inputs
An operational profile
©Ian Sommerville 2000
Icln tase p u
CS 365 Critical Systems Validation
Slide 26
Operational profile generation ●
●
●
Should be generated automatically whenever possible Automatic profile generation is difficult for interactive systems May be straightforward for ‘normal’ inputs but it is difficult to predict ‘unlikely’ inputs and to create test data for them
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 27
Reliability modelling ●
●
A reliability growth model is a mathematical model of the system reliability change as it is tested and faults are removed Used as a means of reliability prediction by extrapolating from current data •
●
Simplifies test planning and customer negotiations
Depends on the use of statistical testing to measure the reliability of a system version
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 28
e(O R liaC bO litF y) t1t2t3T t 4 t 5 im e
Equalstep reliability growth
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 29
Observed reliability growth ●
●
●
●
Simple equalstep model but does not reflect reality Reliability does not necessarily increase with change as the change can introduce new faults The rate of reliability growth tends to slow down with time as frequently occurring faults are discovered and removed from the software A randomgrowth model may be more accurate
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 30
N o t e d i f e r n t r l i a b l t y prvm sF e(O R liaC bO liF ty) m a(inu ldct recspairR seO dC srO linF eab )liftayult w t1t2t3T t 4 t 5 im e
Randomstep reliability growth
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 31
Growth model selection ●
● ●
●
Many different reliability growth models have been proposed No universally applicable growth model Reliability should be measured and observed data should be fitted to several models Bestfit model should be used for reliability prediction
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 32
R eliablity = M easured reliablity F i t e d r e l i a b l i t y m o c u v e erlqaubirled R ty tim saec hotfim E avrlted T i m e bnilty
Reliability prediction
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 33
Reliability validation problems ●
Operational profile uncertainty •
●
High costs of test data generation •
●
Is the operational profile an accurate reflection of the real use of the system Very expensive to generate and check the large number of test cases that are required
Statistical uncertainty for highreliability systems •
It may be impossible to generate enough failures to draw statistically valid conclusions
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 34
Security validation ●
●
●
Security validation has something in common with safety validation It is intended to demonstrate that the system cannot enter some state (an unsafe or an insecure state) rather than to demonstrate that the system can do something However, there are differences • •
Safety problems are accidental; security problems are deliberate Security problems are more generic; Safety problems are related to the application domain
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 35
Security validation ●
Experiencebased validation •
●
Toolbased validation •
●
The system is reviewed and analysed against the types of attack that are known to the validation team Various security tools such as password checkers are used to analyse the system in operation
Tiger teams •
A team is established whose goal is to breach the security of the system by simulating attacks on the system.
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 36
Key points ●
●
●
Statistical testing supplements the defect testing process and is intended to measure the reliability of a system Reliability validation relies on exercising the system using an operational profile a simulated input set which matches the actual usage of the system Reliability growth modelling is concerned with modelling how the reliability of a software system improves as it is tested and faults are removed
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 37
The portable insulin pump Validating the safety of the insulin pump system
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 38
Safety validation ●
Design validation •
●
Code validation •
●
Checking the design to ensure that hazards do not arise or that they can be handled without causing an accident. Testing the system to check the conformance of the code to its specification and to check that the code is a true implementation of the design.
Runtime validation •
Designing safety checks while the system is in operation to ensure that it does not reach an unsafe state.
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 39
Insulin system hazards ● ● ●
●
●
●
insulin overdose or underdose (biological) power failure (electrical) machine interferes electrically with other medical equipment such as a heart pacemaker (electrical) parts of machine break off in patient’s body(physical) infection caused by introduction of machine (biol.) allergic reaction to the materials or insulin used in the machine (biol).
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 40
Fault tree for software hazards r edcotse iadnIm snucilo inn istered or
Iungcaorr leevcetl sm easured or
C oelriveecrt eddo aset d w rong tim e
D lsitveem ry sfey ailure or
S Spuugtaartion fT Inpsuutlain iim erer com Puim po srig ntals faeinluso rer com t i o n a l u n c e c eror incorect or or A lgeo riethrm lgeo riethrm rroitrhm A oertic A rroitrhm A oretic ©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 41
Safety proofs ●
●
●
Safety proofs are intended to show that the system cannot reach in unsafe state Weaker than correctness proofs which must show that the system code conforms to its specification Generally based on proof by contradiction • •
Assume that an unsafe state can be reached Show that this is contradicted by the program code
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 42
Insulin delivery system ●
Safe state is a shutdown state where no insulin is delivered •
●
●
If hazard arises,shutting down the system will prevent an accident
Software may be included to detect and prevent hazards such as power failure Consider only hazards arising from software failure • •
Arithmetic error The insulin dose is computed incorrectly because of some failure of the computer arithmetic Algorithmic error The dose computation algorithm is incorrect
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 43
Arithmetic errors ●
●
●
● ●
Use language exception handling mechanisms to trap errors as they arise Use explicit error checks for all errors which are identified Avoid errorprone arithmetic operations (multiply and divide). Replace with add and subtract Never use floatingpoint numbers Shut down system if exception detected (safe state)
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 44
Algorithmic errors ●
●
●
●
Harder to detect than arithmetic errors. System should always err on the side of safety Use reasonableness checks for the dose delivered based on previous dose and rate of dose change Set maximum delivery level in any specified time period If computed dose is very high, medical intervention may be necessary anyway because the patient may be ill
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 45
Insulin delivery code // The insulin dose to be delivered is a function of blood sugar level, the previous dose // delivered and the time of delivery of the previous dose currentDose = computeInsulin () ; // Safety check adjust currentDose if necessary if (previousDose == 0) // if statement 1 { if (currentDose > 16) currentDose = 16 ; } else if (currentDose > (previousDose * 2) ) currentDose = previousDose * 2 ; if ( currentDose < minimumDose ) // if statement 2 currentDose = 0 ; // then branch else if ( currentDose > maxDose ) // else branch currentDose = maxDose ; administerInsulin (currentDose) ; ©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 46
O v e r d o s e a d m i n i s t r d A d m i n s t e r i n s u l i n I n s u l i n _ d o s e > P r e c o n d i t o n M a x m u m _ d o s e f o u s a f e s t a e o r C o n t r a d i c t i o n I n s u ln _ d o s e > = M i n i m u m _ d o s e a n d i < a x i u m _ d o s e C o n t r a d i c t i o n C o n t r a d i c t i o n I n s u l i n _ d s e = I n s u l i n _ d o s e = 0 M a x m u m _ d o s e i f s t a e m e n t 2I I n s u l i n _ d o s e : = n s u l i n _ d o s e : 0 = n o x c u e d M a x m u m _ d o s e i f s t a e m e n t 2 i f s t a e m e n t 2 h n p a r e l i f p a r e x c u td x c u td Informal safety proof
See Portrait slide
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 47
System testing ●
●
●
●
System testing of the software has to rely on simulators for the sensor and the insulin delivery components. Test for normal operation using an operational profile. Can be constructed using data gathered from existing diabetics Testing has to include situations where rate of change of glucose is very fast and very slow Test for exceptions using the simulator
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 48
Safety assertions ●
●
●
Predicates included in the program indicating conditions which should hold at that point May be based on precomputed limits e.g. number of insulin pump increments in maximum dose Used in formal program inspections or may be preprocessed into safety checks that are executed when the system is in operation
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 49
Safety assertions static void administerInsulin ( ) throws SafetyException { int maxIncrements = InsulinPump.maxDose / 8 ; int increments = InsulinPump.currentDose / 8 ; // assert currentDose <= InsulinPump.maxDose if (InsulinPump.currentDose > InsulinPump.maxDose) throw new SafetyException (Pump.doseHigh); else for (int i=1; i<= increments; i++) { generateSignal () ; if (i > maxIncrements) throw new SafetyException ( Pump.incorrectIncrements); } // for loop } //administerInsulin ©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 50
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 1
Validation perspectives ●
Reliability validation • •
●
Safety validation •
●
Does the measured reliability of the system meet its specification? Is the reliability of the system good enough to satisfy users? Does the system always operate in such a way that accidents do not occur or that accident consequences are minimised?
Security validation •
Is the system and its data secure against external attack?
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 2
Validation techniques ●
Static techniques • •
●
Dynamic techniques • • •
●
Design reviews and program inspections Mathematical arguments and proof Statistical testing Scenariobased testing Runtime checking
Process validation •
Design development processes that minimise the chances of process errors that might compromise the dependability of the system
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 3
Static validation techniques ●
●
●
Static validation is concerned with analyses of the system documentation (requirements, design, code, test data). It is concerned with finding errors in the system and identifying potential problems that may arise during system execution. Documents may be prepared (structured arguments, mathematical proofs, etc.) to support the static validation
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 4
Static techniques for safety validation ●
●
Demonstrating safety by testing is difficult because testing is intended to demonstrate what the system does in a particular situation. Testing all possible operational situations is impossible Normal reviews for correctness may be supplemented by specific techniques that are intended to focus on checking that unsafe situations never arise
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 5
Safety reviews ● ●
●
●
●
Review for correct intended function Review for maintainable, understandable structure Review to verify algorithm and data structure design against specification Review to check code consistency with algorithm and data structure design Review adequacy of system testing
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 6
Review guidance ● ●
●
●
Make software as simple as possible Use simple techniques for software development avoiding errorprone constructs such as pointers and recursion Use information hiding to localise the effect of any data corruption Make appropriate use of faulttolerant techniques but do not be seduced into thinking that fault tolerant software is necessarily safe
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 7
Hazarddriven analysis ●
●
Effective safety assurance relies on hazard identification (covered in previous lectures) Safety can be assured by • • •
●
Hazard avoidance Accident avoidance Protection systems
Safety reviews should demonstrate that one or more of these techniques have been applied to all identified hazards
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 8
The system safety case ●
●
●
It is now normal practice for a formal safety case to be required for all safetycritical computerbased systems e.g. railway signalling, air traffic control, etc. A safety case presents a list of arguments, based on identified hazards, why there is an acceptably low probability that these hazards will not result in an accident Arguments can be based on formal proof, design rationale, safety proofs, etc. Process factors may also be included ©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 9
Formal methods and critical systems ●
●
●
The development of critical systems is one of the ‘success’ stories for formal methods Formal methods are mandated in Britain for the development of some types of safetycritical software for defence applications There is not currently general agreement on the value of formal methods in critical systems development
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 10
Formal methods and validation ●
Specification validation • •
●
Developing a formal model of a system requirements specification forces a detailed analysis of that specification and this usually reveals errors and omissions Mathematical analysis of the formal specification is possible and this also discovers specification problems
Formal verification •
Mathematical arguments (at varying degrees of rigour) are used to demonstrate that a program or a design is consistent with its formal specification
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 11
Problems with formal validation ●
The formal model of the specification is not understandable by domain experts • •
●
It is difficult or impossible to check if the formal model is an accurate representation of the specification for most systems A consistently wrong specification is not useful!
Verification does not scaleup •
Verification is complex, errorprone and requires the use of systems such as theorem provers. The cost of verification increases exponentially as the system size increases.
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 12
Formal methods conclusion ●
Formal specification and checking of critical system components is, in my view, useful •
●
While formality does not provide any guarantees, it helps to increase confidence in the system by demonstrating that some classes of error are not present
Formal verification is only likely to be used for very small, critical, system components •
About 56000 lines of code seems to be the upper limit for practical verification
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 13
Safety proofs ●
●
●
Safety proofs are intended to show that the system cannot reach in unsafe state Weaker than correctness proofs which must show that the system code conforms to its specification Generally based on proof by contradiction • •
●
Assume that an unsafe state can be reached Show that this is contradicted by the program code
May be displayed graphically
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 14
Construction of a safety proof ●
●
● ●
Establish the safe exit conditions for a component or a program Starting from the END of the code, work backwards until you have identified all paths that lead to the exit of the code Assume that the exit condition is false Show that, for each path leading to the exit that the assignments made in that path contradict the assumption of an unsafe exit from the component
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 15
Gas warning system ●
●
System to warn of poisonous gas. Consists of a sensor, a controller and an alarm Two levels of gas are hazardous • •
●
Warning level no immediate danger but take action to reduce level Evacuate level immediate danger. Evacuate the area
The controller takes air samples, computes the gas level and then decides whether or not the alarm should be activated
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 16
Gas sensor control Gas_level: GL_TYPE ; loop Take 100 samples of air Gas_level := 0.000 ; for i in 1..100 loop Gas_level := Gas_level + Gas_sensor.Read ; end loop ; Gas_level := Gas_level / 100 ; if Gas_level > Warning and Gas_level < Danger then Alarm := Warning ; Wait_for_reset ; elsif Gas_level > Danger then Alarm := Evacuate ; Wait_for_reset ; else Alarm := off ; end if ; end loop ; ©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 17
Graphical argument Unsafe state
Gas_level > Warning and Alarm = off
Path 1
or
Gas_level > Warning and Gas_level < Danger
Alarm = Warning
contradiction
©Ian Sommerville 2000
or Path 2
or Path 3
Gas_level > Danger
Alarm = Evacuate
Alarm = off
contradiction
CS 365 Critical Systems Validation
Slide 18
Condition checking Gas_level < Warning Gas_level = Warning Gas_level > Warning and Gas_level < Danger Gas_level = Danger Gas_level > Danger
Path 3 Path 3 Path 1 Path 3 Path 2
Alarm = off (Contradiction) Alarm = off (Contradiction) Alarm = Warning (Contradiction) Alarm = off Alarm = Evacuate (Contradiction)
Code is incorrect. Gas_level = Danger does not cause the alarm to be on
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 19
Key points ●
●
●
Safetyrelated systems should be developed to be as simple as possible using ‘safe’ development techniques Safety assurance may depend on ‘trusted’ development processes and specific development techniques such as the use of formal methods and safety proofs Safety proofs are easier than proofs of consistency or correctness. They must demonstrate that the system cannot reach an unsafe state. Usually proofs by contradiction
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 20
Dynamic validation techniques ●
These are techniques that are concerned with validating the system in execution • •
Testing techniques analysing the system outside of its operational environment Runtime checking checking during execution that the system is operating within a dependability ‘envelope’
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 21
Reliability validation ●
●
●
Reliability validation involves exercising the program to assess whether or not it has reached the required level of reliability Cannot be included as part of a normal defect testing process because data for defect testing is (usually) atypical of actual usage data Statistical testing must be used where a statistically significant data sample based on simulated usage is used to assess the reliability
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 22
Statistical testing ● ●
●
Testing software for reliability rather than fault detection Measuring the number of errors allows the reliability of the software to be predicted. Note that, for statistical reasons, more errors than are allowed for in the reliability specification must be induced An acceptable level of reliability should be specified and the software tested and amended until that level of reliability is reached
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 23
Reliability validation process ● ●
●
●
Establish the operational profile for the system Construct test data reflecting the operational profile Test the system and observe the number of failures and the times of these failures Compute the reliability after a statistically significant number of failures have been observed
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 24
Operational profiles ●
●
An operational profile is a set of test data whose frequency matches the actual frequency of these inputs from ‘normal’ usage of the system. A close match with actual usage is necessary otherwise the measured reliability will not be reflected in the actual usage of the system Can be generated from real data collected from an existing system or (more often) depends on assumptions made about the pattern of usage of a system
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 25
N u m b e r of inputs
An operational profile
©Ian Sommerville 2000
Icln tase p u
CS 365 Critical Systems Validation
Slide 26
Operational profile generation ●
●
●
Should be generated automatically whenever possible Automatic profile generation is difficult for interactive systems May be straightforward for ‘normal’ inputs but it is difficult to predict ‘unlikely’ inputs and to create test data for them
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 27
Reliability modelling ●
●
A reliability growth model is a mathematical model of the system reliability change as it is tested and faults are removed Used as a means of reliability prediction by extrapolating from current data •
●
Simplifies test planning and customer negotiations
Depends on the use of statistical testing to measure the reliability of a system version
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 28
e(O R liaC bO litF y) t1t2t3T t 4 t 5 im e
Equalstep reliability growth
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 29
Observed reliability growth ●
●
●
●
Simple equalstep model but does not reflect reality Reliability does not necessarily increase with change as the change can introduce new faults The rate of reliability growth tends to slow down with time as frequently occurring faults are discovered and removed from the software A randomgrowth model may be more accurate
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 30
N o t e d i f e r n t r l i a b l t y prvm sF e(O R liaC bO liF ty) m a(inu ldct recspairR seO dC srO linF eab )liftayult w t1t2t3T t 4 t 5 im e
Randomstep reliability growth
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 31
Growth model selection ●
● ●
●
Many different reliability growth models have been proposed No universally applicable growth model Reliability should be measured and observed data should be fitted to several models Bestfit model should be used for reliability prediction
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 32
R eliablity = M easured reliablity F i t e d r e l i a b l i t y m o c u v e erlqaubirled R ty tim saec hotfim E avrlted T i m e bnilty
Reliability prediction
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 33
Reliability validation problems ●
Operational profile uncertainty •
●
High costs of test data generation •
●
Is the operational profile an accurate reflection of the real use of the system Very expensive to generate and check the large number of test cases that are required
Statistical uncertainty for highreliability systems •
It may be impossible to generate enough failures to draw statistically valid conclusions
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 34
Security validation ●
●
●
Security validation has something in common with safety validation It is intended to demonstrate that the system cannot enter some state (an unsafe or an insecure state) rather than to demonstrate that the system can do something However, there are differences • •
Safety problems are accidental; security problems are deliberate Security problems are more generic; Safety problems are related to the application domain
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 35
Security validation ●
Experiencebased validation •
●
Toolbased validation •
●
The system is reviewed and analysed against the types of attack that are known to the validation team Various security tools such as password checkers are used to analyse the system in operation
Tiger teams •
A team is established whose goal is to breach the security of the system by simulating attacks on the system.
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 36
Key points ●
●
●
Statistical testing supplements the defect testing process and is intended to measure the reliability of a system Reliability validation relies on exercising the system using an operational profile a simulated input set which matches the actual usage of the system Reliability growth modelling is concerned with modelling how the reliability of a software system improves as it is tested and faults are removed
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 37
The portable insulin pump Validating the safety of the insulin pump system
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 38
Safety validation ●
Design validation •
●
Code validation •
●
Checking the design to ensure that hazards do not arise or that they can be handled without causing an accident. Testing the system to check the conformance of the code to its specification and to check that the code is a true implementation of the design.
Runtime validation •
Designing safety checks while the system is in operation to ensure that it does not reach an unsafe state.
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 39
Insulin system hazards ● ● ●
●
●
●
insulin overdose or underdose (biological) power failure (electrical) machine interferes electrically with other medical equipment such as a heart pacemaker (electrical) parts of machine break off in patient’s body(physical) infection caused by introduction of machine (biol.) allergic reaction to the materials or insulin used in the machine (biol).
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 40
Fault tree for software hazards r edcotse iadnIm snucilo inn istered or
Iungcaorr leevcetl sm easured or
C oelriveecrt eddo aset d w rong tim e
D lsitveem ry sfey ailure or
S Spuugtaartion fT Inpsuutlain iim erer com Puim po srig ntals faeinluso rer com t i o n a l u n c e c eror incorect or or A lgeo riethrm lgeo riethrm rroitrhm A oertic A rroitrhm A oretic ©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 41
Safety proofs ●
●
●
Safety proofs are intended to show that the system cannot reach in unsafe state Weaker than correctness proofs which must show that the system code conforms to its specification Generally based on proof by contradiction • •
Assume that an unsafe state can be reached Show that this is contradicted by the program code
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 42
Insulin delivery system ●
Safe state is a shutdown state where no insulin is delivered •
●
●
If hazard arises,shutting down the system will prevent an accident
Software may be included to detect and prevent hazards such as power failure Consider only hazards arising from software failure • •
Arithmetic error The insulin dose is computed incorrectly because of some failure of the computer arithmetic Algorithmic error The dose computation algorithm is incorrect
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 43
Arithmetic errors ●
●
●
● ●
Use language exception handling mechanisms to trap errors as they arise Use explicit error checks for all errors which are identified Avoid errorprone arithmetic operations (multiply and divide). Replace with add and subtract Never use floatingpoint numbers Shut down system if exception detected (safe state)
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 44
Algorithmic errors ●
●
●
●
Harder to detect than arithmetic errors. System should always err on the side of safety Use reasonableness checks for the dose delivered based on previous dose and rate of dose change Set maximum delivery level in any specified time period If computed dose is very high, medical intervention may be necessary anyway because the patient may be ill
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 45
Insulin delivery code // The insulin dose to be delivered is a function of blood sugar level, the previous dose // delivered and the time of delivery of the previous dose currentDose = computeInsulin () ; // Safety check adjust currentDose if necessary if (previousDose == 0) // if statement 1 { if (currentDose > 16) currentDose = 16 ; } else if (currentDose > (previousDose * 2) ) currentDose = previousDose * 2 ; if ( currentDose < minimumDose ) // if statement 2 currentDose = 0 ; // then branch else if ( currentDose > maxDose ) // else branch currentDose = maxDose ; administerInsulin (currentDose) ; ©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 46
O v e r d o s e a d m i n i s t r d A d m i n s t e r i n s u l i n I n s u l i n _ d o s e > P r e c o n d i t o n M a x m u m _ d o s e f o u s a f e s t a e o r C o n t r a d i c t i o n I n s u ln _ d o s e > = M i n i m u m _ d o s e a n d i < a x i u m _ d o s e C o n t r a d i c t i o n C o n t r a d i c t i o n I n s u l i n _ d s e = I n s u l i n _ d o s e = 0 M a x m u m _ d o s e i f s t a e m e n t 2I I n s u l i n _ d o s e : = n s u l i n _ d o s e : 0 = n o x c u e d M a x m u m _ d o s e i f s t a e m e n t 2 i f s t a e m e n t 2 h n p a r e l i f p a r e x c u td x c u td Informal safety proof
See Portrait slide
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 47
System testing ●
●
●
●
System testing of the software has to rely on simulators for the sensor and the insulin delivery components. Test for normal operation using an operational profile. Can be constructed using data gathered from existing diabetics Testing has to include situations where rate of change of glucose is very fast and very slow Test for exceptions using the simulator
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 48
Safety assertions ●
●
●
Predicates included in the program indicating conditions which should hold at that point May be based on precomputed limits e.g. number of insulin pump increments in maximum dose Used in formal program inspections or may be preprocessed into safety checks that are executed when the system is in operation
©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 49
Safety assertions static void administerInsulin ( ) throws SafetyException { int maxIncrements = InsulinPump.maxDose / 8 ; int increments = InsulinPump.currentDose / 8 ; // assert currentDose <= InsulinPump.maxDose if (InsulinPump.currentDose > InsulinPump.maxDose) throw new SafetyException (Pump.doseHigh); else for (int i=1; i<= increments; i++) { generateSignal () ; if (i > maxIncrements) throw new SafetyException ( Pump.incorrectIncrements); } // for loop } //administerInsulin ©Ian Sommerville 2000
CS 365 Critical Systems Validation
Slide 50
