System test planning and the usefulness of a safety checklist
This presentation is the property of its rightful owner.
Sponsored Links
1 / 34

System Test Planning and the usefulness of a “Safety Checklist” PowerPoint PPT Presentation


  • 128 Views
  • Uploaded on
  • Presentation posted in: General

System Test Planning and the usefulness of a “Safety Checklist”. ECEN5033. State Transition Diagram for “split” routine. Present StateInput or EventActionOutputNext State ST1. Idle card insertedrequest for PINWaiting for PIN

Download Presentation

System Test Planning and the usefulness of a “Safety Checklist”

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


System test planning and the usefulness of a safety checklist

System Test Planningand the usefulness of a “Safety Checklist”

ECEN5033

R. Dameron, University of Colorado, ECEN5033, System Test Planning


State transition diagram for split routine

State Transition Diagram for “split” routine

R. Dameron, University of Colorado, ECEN5033, System Test Planning


System test planning and the usefulness of a safety checklist

Present StateInput or EventActionOutputNext State

ST1. Idlecard insertedrequest for PINWaiting for PIN

ST2. Waiting for PINPIN entereddisplay asterisksValidating PIN

ST3. Waiting for PINcanceldisplay msgEjecting

ST4. Validating PINindicates “valid”display choicesWaiting for customer

transaction choice

ST5. Validating PINindicates “stolen”display “stolen” confiscating

ST6. Validating PINindicates “invalid”display “invalid” Waiting for PIN

ST7. Waiting for customer transaction choice Canceldisplay “cancel” Ejecting

ST8. Waiting for customer transaction choice Balance Query selectedProcessing query

continued on next slide

R. Dameron, University of Colorado, ECEN5033, System Test Planning


System test planning and the usefulness of a safety checklist

ST9. Waiting for customer transaction choice

Withdrawal selectedProcessing w/d

ST10. confiscatingCard confiscatedterminating

ST11. Processing queryRejected for this userdisplay “rejected” Ejecting

ST12. Processing queryQuery OKdisplay printing printing

ST13. Processing withdrawal ok amountdisplay ok msgdispensing

ST14. Processing withdrawal not ok amountdisplay refusalEjecting

ST15. Printingtransaction completeprint receiptejecting

ST16. Dispensingsufficient cash in ATMcashprinting

ST17. Dispensinginsufficient cash in ATMdisp “insufficient cash”ejecting

ST18. Ejectingcard ejection starteddisplay msg to

take cardterminating

ST19. terminatingcard ejection completedisplay ending msg idle

R. Dameron, University of Colorado, ECEN5033, System Test Planning


State transition diagram incomplete

State Transition Diagram - incomplete

card inserted/

PIN inserted/

waiting for PIN

Idle

validating PIN

“invalid”

card ej

complete

“cancel”

“stolen”

ejecting

“valid”

terminat-ing

confis-cating

waiting for cust xaction

card confis’d

“cancel”

R. Dameron, University of Colorado, ECEN5033, System Test Planning


2 dimensional event table

2-dimensional event table

Action;action = sequential actions. Action, action = concurrent actions. X = impossible. --- = no action required.

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Checklist regarding robustness

Checklist regarding Robustness

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Robustness

Robustness

11. If performance degradation is the chosen response, is the degradation predictable?

12. Are there sufficient delays incorporated into the error-recovery responses, e.g. don’t return to normal state prior to managing the error

13. Are feedback loops specified where appropriate to compare the actual effects of outputs on the system with the predicted effects?

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Robustness continued

Robustness, continued

  • Are all modes and modules reachable (used in some path through the code)? Superfluous? Other logic error?

  • If a hazards analysis has been done, does every path from a hazardous state (a failure-mode) lead to a low-risk state?

  • Are the inputs identified which, if not received, can lead to a hazardous state or can prevent recovery (single-point failures)?

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Usefulness

Usefulness?

  • Safety checklist has been demonstrated to “ask the right questions”

  • Not sufficient to preclude introducing errors

  • Necessary although not sufficient

R. Dameron, University of Colorado, ECEN5033, System Test Planning


May i have the envelope please

#1

#2

#4

May I have the envelope, please …

Not every hazardous state led to a low-risk state.

Error-recovery responses incorporated

insufficient delays.

Input arrived when it shouldn’t and no response

was specified; response defaulted to

unintended behavior.

Response not specified for out-of-range values

that were possible for some inputs

#5 Output produced too fast for interfacing module

#3

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Allows tailoring

Allows tailoring

  • Focuses on historically troublesome aspects of safety-critical, embedded software

  • Avoids over-specification of well-understood or low-risk requirements

  • Tailor to level of technical or historical risk

R. Dameron, University of Colorado, ECEN5033, System Test Planning


First step toward safety constraints

First step toward safety constraints

  • Many items that it identifies are system hazards

  • Can be used to identify safety constraints

  • Not yet ready for formal prediction

    • How use for informal prediction of error prone factors

R. Dameron, University of Colorado, ECEN5033, System Test Planning


After requirements are improved

After Requirements Are Improved …

  • How do we ensure that requirements are implemented and maintained?

    • After code is written (new code or bug fixes); note: difficult to unit test these issues

    • After new requirements are added

    • After old requirements are modified

  • Role of reviews

  • Code the invariants where appropriate

  • System tests to test use cases and the safety checklist

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Create a system test plan ieee std 829

Create a system test plan – IEEE Std. 829

  • Test the Success Scenario and conditions that lead to alternative paths of use cases

  • If possible, test to verify the relevant safety checklist items – “safety” may not be main concern but correct interfaces and robustness are.

  • If any resources are shared among processes, review and test for correctness of mutual exclusion. (SW Eng of Multi-program Sys)

  • If “cooperating processes”, verify suspension happens correctly, a suspended process restored when appropriate, restoration correct.

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Ieee 829 standard test plan outline

IEEE 829 Standard Test Plan Outline

1.0Introduction

2.0Test Items

3.0Tested Features

4.0Features Not Tested (per cycle)

5.0Testing Strategy and Approach

5.1Syntax

5.2Description of functionality

5.3Arguments for Tests

5.4Expected Output

5.5Specific Exclusions

5.6Dependencies

5.7Test Case Success/Failure Criteria

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Ieee 829 standard test plan outline 1

1.0Introduction

2.0Test Items

3.0Tested Features

4.0Features Not Tested (per cycle)

[Repeat 5.0 for each system level test.]

5.0 Testing Strategy and Approach

5.1Syntax

5.2Description of functionality

5.3Arguments for Test

5.4Expected Output

5.5Specific Exclusions

5.6Dependencies

5.7Test Case Success/Failure Criteria

IEEE 829 Standard Test Plan Outline - 1

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Ieee 829 standard test plan outline 2

IEEE 829 Standard Test Plan Outline - 2

6.0Pass/Fail Criteria for the Complete Test Cycle

7.0Entrance Criteria/Exit Criteria

8.0Test-Suspension Criteria and Resumption Req’s

9.0Test Deliverables/Status Communications Vehicles

10.0Testing Tasks

11.0Hardware and Software Requirements (for testing)

12.0Problem Determination and Correction Responsibilities

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Ieee 829 standard test plan outline 3

IEEE 829 Standard Test Plan outline - 3

13.0Staffing and Training Needs/Assignments

14.0Test Schedules

15.0Risks and Contingencies

16.0Approvals

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Glass box briefly need implementation details

Glass-box – briefly (Need implementation details)

  • Test module/process/object-cluster interfaces (process level can be system test)

  • Test object/object-cluster contracts

  • Create test data to force certain code paths

  • Note: if team is doing incremental development, you can begin glass-box testing early

R. Dameron, University of Colorado, ECEN5033, System Test Planning


More to consider

More to consider

  • If the system is too large to test thoroughly, what tests should you emphasize?

  • Stay tuned …

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Hazop

HAZOP

System Safety: HAZOP and Software HAZOP, by Felix Redmill, Morris Chudleigh, James Catmur, John Wiley & Sons, 1999

R. Dameron, University of Colorado, ECEN5033, System Test Planning


What is hazop

What is HAZOP?

  • Technique for identifying and analyzing the hazards and operational concerns of a system.

  • Central activity – a methodical investigation of a system description (design representation).

R. Dameron, University of Colorado, ECEN5033, System Test Planning


What this presentation does not cover

What this presentation does not cover:

  • The book puts a LOT of emphasis on

    • Selecting the study initiator

    • Selecting the study leader

    • Planning the study

    • Roles during the study

    • Questions vs. follow-up

    • Completion criteria

      (P.S. It also tells how to conduct the study itself :-)

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Reasonable limits for this class

Reasonable Limits for this class

  • This is a human-intensive activity

  • As such, the details on the previous page are of extreme importance – authors are experienced and therefore recognize this

  • You won’t be able to conduct a HAZOP study on the basis of these slides

  • Goal: Understand what it is – set the bar higher

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Study process itself in a nutshell

Study process itself in a nutshell

Introductions

Presentation of design notation

Examine design methodically one unit at a time

Is it possible to deviate from design intent here?

YES

Examine both consequences and causes of the possible deviation

NO

NO

Document results

Define follow-up work

YES

Time up?

Agree on documentation

R. Dameron, University of Colorado, ECEN5033, System Test Planning

Sign off


Examine design methodically each unit in turn

Examine design methodically each unit in turn

  • Suppose the design representation is a collection of state transition tables:

  • Units are states, transitions, event/action pairs

  • For EACH, list the recommended attributes (see table from the Hazop book)

  • For each attribute, use the guide words to trigger the questions about ways to deviate

R. Dameron, University of Colorado, ECEN5033, System Test Planning


The suggested guide words

The suggested guide words

  • No: negation of design intention; no part of design intention is achieved but nothing else happens

  • More: Quantitative increase

  • Less: Quantitative decrease

  • As well as: Qualitative increase where all design intention is achieved plus additional activity

  • Part of: Qualitative decrease where only part of the design intention is achieved

  • Reverse: logical opposite of the intention

  • Other than: complete substituion, where no part of the original intention is achieved but something quite different happens

R. Dameron, University of Colorado, ECEN5033, System Test Planning


When timing matters

When timing matters

  • Add the following guide words:

    • Early: something happens earlier in time than intended

    • Late: something happens later in time than intended

    • Before: something happens earlier in a sequence than intended

    • After: something happens later in a sequence than intended

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Guide words chosen

Guide words chosen

  • Match the system being examined to appropriate table or modify the closest

  • Match the design representation

  • Note: not all guide words apply to all attributes

    • For attribute “speed” of an electric motor, omit guide word “as well as” and “part of”

    • For attribute “data flow” on a dfd, “less” is not used because meaning covered by “part of”

  • Generally, study leader selects from the guide words, provides interpretations based on chosen design representation and context, distributes to team in advance of the study

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Applications

Applications

  • Originally developed for chemical plants

  • Book has detailed examples for

    • Software using data flow diagrams

    • Software using state transition diagrams

      • Includes timing attributes of response time and repetition time

    • Software using various OO models

    • Digital electronics

    • Communication systems

    • Electromechanical systems

  • Same guide words, different interpretations

R. Dameron, University of Colorado, ECEN5033, System Test Planning


See book excerpts

More detailed outline of the HAZOP process – Figure 9.2

For all entities

For all attributes

For each guide word

Is deviation credible?

Example matrices

See book excerpts

R. Dameron, University of Colorado, ECEN5033, System Test Planning


Fig 9 2

Fig 9.2

HAZOP meeting process

R. Dameron, University of Colorado, ECEN5033, System Test Planning


System test planning and the usefulness of a safety checklist

R. Dameron, University of Colorado, ECEN5033, System Test Planning


  • Login