1 / 34

Testing Techniques

Testing Techniques. Testing with Finite State Machines. Ed Brinksma course 2004. This Lecture : Overview. Testing with formal methods: Generic framework Testing based on Labelled Transition Systems - ioco Testing based on Finite State Machines (FSM) Now: FSM State based testing

angus
Download Presentation

Testing Techniques

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Testing Techniques Testing withFinite State Machines Ed Brinksma course 2004

  2. This Lecture : Overview • Testing with formal methods: • Generic framework • Testing based on Labelled Transition Systems - ioco • Testing based on Finite State Machines (FSM) • Now: FSM • State based testing • H. Ural, Formal methods for test sequence generation,Computer Communications, 15(5), 1992. • Other literature:D. Lee and M. Yannakakis,Principles and methods of testing finite state machines - A survey.The Proceedings of the IEEE 84, August 1996. TT 2004, FSM 2

  3. State Machines • Many systems can be specified / modelled as state machines • State machines as the basis for testing : • FSM : Finite State Machine • black box • specification based • reactive systems : • communication protocols • control systems • embedded systems TT 2004, FSM 3

  4. state input output new state State Machines • States • Transitions • Inputs ( “triggers”) • Outputs TT 2004, FSM 4

  5. Coffee Machine : State Graph stui? / - 0 5 koffie? / - koffie? / - stui? / - dub? / - dub? / stui! koffie? / koffie! 10 stui? / stui! dub? / dub! TT 2004, FSM 5

  6. Coffee Machine : State Table • conventions (from Z): • plain name: state • name?: input • name!: output • Table gives new state and output as function of state and input State 0 5 10 Input stui? 5 / - 10 / - 10 / stui! dub? 10 /- 10 / stui! 10 / dub! koffie? 0 / - 5 / - 0 / koffie! TT 2004, FSM 6

  7. State Machine : FSM Model • FSM - Finite State Machine - or Mealy Machine is 5-tuple • M = ( S, I, O, ,  ) • S finite set of states • I finite set of inputs • O finite set of outputs •  : S x I S transfer function •  : S x I  O output function • usually we also indicate an initial state • Natural extension to sequences :  : S x I*  S : S x I*  O* TT 2004, FSM 7

  8. State Machine : FSM Model • FSM restrictions: • deterministic: : S x I S and  : S x I  O are functions • completely specified: : S x I S and  : S x I  O are complete functions( empty output is allowed; sometimes implicit completeness ) • strongly connected:from any state any other state can be reached,or any state can be reached from the initial state • reduced:there are no equivalent states TT 2004, FSM 8

  9. Coffee Machine FSM Model stui? / - 0 5 koffie? / - koffie? / - stui? / - dub? / - dub? / stui! koffie? / koffie! 10 dub? / dub! stui? / stui! TT 2004, FSM 9

  10. Testing with FSM • Given: • a specification FSM MS • a ( black box ) implementation FSM MI • determine whether MI conforms to MS • i.e., MI behaves in accordance with MS • i.e., whether outputs of MI are the same as of MS • i.e., whether the reduced MI is equivalent to MS • Possible errors: • extra or missing states • output fault • transition fault • to other state • to new state TT 2004, FSM 10

  11. FSM Testing • Test with paths of the (specification) FSM • Path = sequence of inputs with expected outputs( cf. path testing as white-box technique) • Infinitely many paths : how to select ? • Different strategies : • test every state : state coverage (of specification !) • test every transition : transition coverage • test output of every transition • test output + resulting state of every transition TT 2004, FSM 11

  12. stui? / - stui? / - koffie? / - koffie? / - 0 0 koffie? / - koffie? / - 5 5 stui? / - stui? / - dub? / - dub? / - dub? / stui! dub? / stui! koffie? / koffie! koffie? / koffie! 10 10 stui? / stui! stui? / stui! dub? / dub! dub? / dub! FSM State Testing • Make State Tour that covers every state Test sequence : stui? dub? koffie? TT 2004, FSM 12

  13. stui? / - stui? / - koffie? / - koffie? / - 0 0 koffie? / - koffie? / - 5 5 stui? / - stui? / - dub? / - dub? / - dub? / stui! dub? / stui! koffie? / koffie! koffie? / koffie! 10 10 stui? / stui! stui? / stui! dub? / dub! dub? / dub! FSM Transition Tour • Make Transition Tour that covers every transition Test sequence : koffie? stui? koffie? stui? stui? dub? koffie? dub? koffie? stui? dub? koffie? TT 2004, FSM 13

  14. a? / x! S1 S2 FSM Transition Testing • One big tour as test case not always desirable( too long, too complex, difficult to analyse, not specific ) • Make test case for every transition separately: • Test transition : • Go to state S1 • Apply input a? • Check output x! • Verify state S2 ( optionally ) • Test purpose: “Test whether the system, when in state S1, produces output x! on input a? and goes to state S2” TT 2004, FSM 14

  15. FSM Transition Testing • Go to state S1 : • synchronizing sequence brings machine to particular state, say S0, from any state ( but synchronizing sequence may not exist ) • or: use reset transition if available • go from S0 to S1( always possible because of determinism and completeness ) TT 2004, FSM 15

  16. stui? / - stui? / - koffie? / - koffie? / - 0 0 koffie? / - koffie? / - 5 5 stui? / - stui? / - stui? / - dub? / - dub? / - koffie? / - 0 stui? / - dub? / stui! dub? / stui! 5 koffie? / - koffie? / koffie! koffie? / koffie! stui? / - koffie? / - 0 koffie? / - 10 10 5 dub? / - stui? / - stui? / stui! stui? / stui! dub? / dub! dub? / dub! dub? / stui! dub? / - koffie? / koffie! 10 dub? / stui! koffie? / koffie! stui? / - stui? / stui! dub? / dub! 10 koffie? / - 0 5 koffie? / - stui? / stui! stui? / - dub? / dub! dub? / - dub? / stui! koffie? / koffie! 10 stui? / stui! dub? / dub! FSM Transition Testing synchronizing sequence : dub? koffie? TT 2004, FSM 16

  17. stui? / - stui? / - koffie? / - 0 koffie? / - koffie? / - 0 5 koffie? / - 5 stui? / - stui? / - dub? / - dub? / - dub? / stui! dub? / stui! koffie? / koffie! koffie? / koffie! 10 10 stui? / stui! dub? / dub! stui? / stui! dub? / dub! FSM Transition Testing synchronizing sequence : dub? koffie? To test dub? / stui! : go to state 5 by : dub? koffie? stui? TT 2004, FSM 17

  18. stui? / - koffie? / - 0 koffie? / - 5 stui? / - dub? / - dub? / stui! koffie? / koffie! 10 stui? / stui! dub? / dub! FSM Transition Testing • To test dub? / stui! : • go to state 5 by : dub? koffie? stui? • give input dub? • check output stui! • verify that machine is in state 10 TT 2004, FSM 18

  19. FSM Transition Testing • State identification and verification : • Apply sequence of inputs in the current state of the FSMsuch that from the outputs we can • identify that state where we started; or • verify that we were in a particular start state • Different kinds of sequences • UIO sequences ( Unique Input Output sequence, SIOS) • Distinguishing sequence ( DS ) • W - set ( characterizing set of sequences ) • UIOv • SUIO Single UIO • MUIO Multiple UIO • Overlapping UIO TT 2004, FSM 19

  20. FSM Transition Testing • State verification : • UIO sequences • sequence x that distinguishes state s from all other states :for all t s :  ( s, x )   ( t, x ) • each state has its own UIO sequence • UIO sequences may not exist • Distinguishing sequence • sequence x that produces different output for each state :for all pairs t, s with t s :  ( s, x )   ( t, x ) • a distinguishing sequence may not exist • W - set of sequences • set of sequences W which can distinguish any pair of states :for all pairs t s there is x  W :  ( s, x )   ( t, x ) • W - set always exists for reduced FSM TT 2004, FSM 20

  21. stui? / - koffie? / - 0 koffie? / - 5 stui? / - dub? / - dub? / stui! koffie? / koffie! 10 stui? / stui! dub? / dub! FSM Transition Testing UIO sequences state 0 : stui? / -koffie? / - state 5 : dub? / stui! state 10 : koffie? / koffie! TT 2004, FSM 21

  22. stui? / - koffie? / - 0 koffie? / - 5 stui? / - dub? / - dub? / stui! koffie? / koffie! 10 stui? / stui! dub? / dub! FSM Transition Testing DS sequence DS sequence : dub? output state 0 : - output state 5 : stui! output state 10 : dub! TT 2004, FSM 22

  23. stui? / - koffie? / - 0 koffie? / - 5 stui? / - dub? / - dub? / stui! koffie? / koffie! 10 dub? / dub! stui? / stui! FSM Transition Testing go to state 5 : dub? koffie? stui? give input dub?check output stui! Apply UIO of state 10 : koffie? / koffie! • To test dub? / stui! : Test case : dub? / * koffie? / * stui? / -dub? / stui!koffie? / koffie! TT 2004, FSM 23

  24. stui? / - koffie? / - 0 koffie? / - 5 stui? / - dub? / - dub? / stui! koffie? / koffie! 10 stui? / stui! dub? / dub! FSM Transition Testing - 9 transitions / test cases for coffee machine - if end-state of one corresponds with start-state of next then concatenate - different ways to optimize and remove overlapping / redundant parts - there are (academic) tools to support this TT 2004, FSM 24

  25. FSM Transition Testing • Test transition : • Go to state S1 • Apply input a? • Check output x! • Verify state S2 • Checks every output fault and transfer fault (to existing state) • If we assume that the number of states of the implementation machine MI is less than or equal to the number of states of the specification machine to MS.then testing all transitions in this wayleads to equivalence of reduced machines,i.e., complete conformance TT 2004, FSM 25

  26. FSM: variations on this theme • there exists many variations on this theme: • Moore machines:output determined by state instead of transition • Infinite state machines:infinite number of states (e.g. state contains variable) • Non-deterministic FSM:transition relation instead of transition function Labelled Transition Systems - ioco • ... TT 2004, FSM 26

  27. FSM: Moore machines • Mealy Machine is 5-tuple: M = ( S, I, O, ,  ) • S finite set of states • I finite set of inputs • O finite set of outputs •  : S x I S transfer function •  : S x I  O output function • Moore Machine is 5-tuple: M = ( S, I, O, ,  ) • S finite set of states • I finite set of inputs • O finite set of outputs •  : S x I S transfer function •  : S  O output function, not dependent of input • usually we add an initial state TT 2004, FSM 27

  28. Coffee Machine: Mealy model stui? / - 0 5 koffie? / - koffie? / - stui? / - dub? / - dub? / stui! koffie? / koffie! 10 dub? / dub! stui? / stui! TT 2004, FSM 28

  29. Coffee Machine: Moore model stui? koffie? koffie? stui? dub? 105/5! 0/- 5/- stui? stui? stui? dub? koffie? dub? stui? dub? koffie? dub? K/koffie! 100/- 1010/10! dub? koffie? koffie? TT 2004, FSM 29

  30. Moore Coffee Machine : State Table State 0 5 100 105 1010 K output - - - 5! 10! koffie! Input stui? 5 100 105 105 105 5 dub? 100 105 1010 1010 1010 100 koffie? 0 1010 K K K 0 • Compared to Mealy machine: • more states • simpler output function (not dependent on input) • changes hardly anything for testing TT 2004, FSM 30

  31. state input output new state Infinite - State Machines • States : data structures / data bases • Inputs ( “triggers”) : operations on data bases • Transitions : new data base states • Outputs : results of queries TT 2004, FSM 31

  32. State Machine : FSM Model • FSM - Finite State Machine - or Mealy Machine is 5-tuple • M = ( S, I, O, ,  ) • S finite set of states • I finite set of inputs • O finite set of outputs •  : S x I S transfer function •  : S x I  O output function • Natural extension to sequences :  : S x I*  S : S x I*  O* TT 2004, FSM 32

  33. state input output new state Infinite State Machine Testing • Not all transitions can be tested ….. • But principle remains the same : test transitions • Go to start state of transition • Apply input • Check output • Verify result state • Use selection techniques to select transitions / start states • equivalence partitioning • boundary value analysis • ……... TT 2004, FSM 33

  34. FSM Testing vs. InfSM Testing • Restrictions on FSM: • deterministic • completeness • FSM has always alternation between input and output • Difficult to specify interleaving in FSM • FSM is not compositional • FSM has “more intuitive” theory • FSM test suite is complete-- but only w.r.t. assumption on number of states • FSM test theory has been around for a number of years TT 2004, FSM 34

More Related