Conformance Testing

Conformance Testing Lynne S. Rosenthal Software Diagnostic and Conformance Testing Division (301)975-3353 [email protected]

Outline  Software

Standards and Testing  Conformity assessment  Conformance Testing – what is it – why bother – how it works

Software Standards  Promote

portability & interoperability  Success depends on: – Efficiency of standard development process – Quality of resulting standard – Ability to ensure implementations conform to the standard

Software Testing  Testing

- process of trying to find errors by means of experimentation  Testing of realistic system not exhaustive – can show presence of errors not their absence  Testing

is either

– structural testing (white box) – functional testing (black box) • conformance testing

Conformance Testing  Determines

which areas of the specification* are implemented correctly. – tests capture the technical description of the specification – testing is bound in scope by the specification – internal structure of product-under-test not accessible

*publicly available standard

Why: conformity assessment  ISO

Guide 2: “any activity concerned with determining directly or indirectly that relevant requirements are fulfilled.”

 communication

between buyer and seller

– buyers increased confidence – sellers substantiate claims

Why:  Improves

quality of software  Objective method for evaluating products – ensures presence of required characteristics  Promote

portability and interoperability

How: building tests  Falsification

Testing

– find errors by means of experimentation – show presence of errors not their absence – prove non-conformance  Tests

based on:

– required functionality – logical errors by misunderstanding requirements – errors from boundary conditions and divergence – common programming errors

Requirements

Software

Conformance Tests Test programs

When:  Develop

in parallel with specification

– feedback to specification for clarification, correction, etc.  Develop

in partnership with product developers

Summary: Conformance tests  Determine

if a product has required

characteristics  Increases

product confidence

 Communication  Improve

between buyer and seller

quality of software