1 / 27

Automatic test case generation from a formal model

Automatic test case generation from a formal model. LTG technology evaluation based on a B model of a part of ATV onboard software. Julien ORSAT – Stagiaire from the University of Franche-Comté (France) 08/03 – 23/07/2004 Supervisor: Philippe CHEVALLEY. Outline.

lovie
Download Presentation

Automatic test case generation from a formal model

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automatic test case generationfrom a formal model LTG technology evaluation based on a B model of a part of ATV onboard software Julien ORSAT – Stagiaire from the University of Franche-Comté (France) 08/03 – 23/07/2004 Supervisor: Philippe CHEVALLEY

  2. Outline • B modelling for Leirios Test Generator • ATV software case study: Gyra functional unit • Gyra B formal model • LTG automatic test case generation • Conclusion

  3. Outline • B modelling for Leirios Test Generator • B formal notation • LTG requirement traceability • LTG syntax restrictions • ATV software case study: Gyra functional Unit • Gyra B formal model • LTG automatic test case generation • Conclusion

  4. B formal notation • Used as input language for LTG • Other supported input format : Statecharts/Statemate, UML • Part of the B method • Code generation from the model • Proof obligations to ensure correctness • Several refinement steps from the model to the target code • Created in the 90’s by J-R Abrial • Applied in railway transports and smart card industry • LTG input: 1st step of the B method (“abstract machine”) • Text format • Statecharts and UML are graphically used

  5. Some B concepts • Sets ; could be considered as base types • GYRO_ID = {G1, G2, G3, G4} • GYRO_MODE = {COARSE, FINE} • Relations ; to define sets of ordered pairs • gyra_mode : GYRO_ID --> GYRO_MODE • Substitutions ; to modify variable values • gyra_mode := gyra_mode <+ {(G3 |-> FINE)} • Operations ; to describe system behaviour • Used by LTG to compute test cases

  6. cmd_report <-- setGyroMode(gyro, mode) = PRE/* precondition */ gyro : GYRO_ID & mode : GYRO_MODE & … THEN/* body */ IFconditionTHEN cmd_report := REJECTED ELSE cmd_report := ACCEPTED || gyra_mode := gyra_mode <+ { (gyro |-> mode) } END END Profile B reserved word Commentary Boolean expression Substitution B operation example • A B operation is considered atomic • <+ is the relation overriding operator

  7. LTG requirement traceability • Provided by LTG • not part of B notation • Uses tags in the model • Based on B commentaries • IF (...) THEN cmd_report := REJECTED /*@REQ: R52 @*/ELSE ... • Allows automatic traceability report generation • Test case -> covered requirements • Requirement -> covering test cases  Requirement coverage criteriafor generated test case evaluation

  8. LTG restrictions to the B notation • Syntax • Real numbers • Difficulty to compute expected results • Sequences • list-like data structures • “Such as” set description • { element | predicates } • Single B machine only • No machine inclusion, promotion …

  9. Outline • B modelling for Leirios Test Generator • ATV software case study: Gyra functional unit • Gyra Functional Unit • ATV software structure • Case study choice explanation • Gyra B formal model • LTG automatic test case generation • Conclusion

  10. Gyra Functional Unit • High level services to use a 4 GYRoscope Assembly (GYRA) • Hardware control • Monitoring algorithms • Processing algorithms • Designed and developed by EADS • 89 requirements • 13 Ada83 packages • 7 000 code lines Managed by an internalfinite state machine (FSM)

  11. ATV software structure • Gyra Functional Unit is part of software layer 3 LN 3:… LN 3:… LN 3: Gyra FU 28 FUs Applications Software layers LN 2 library Communications LN 1 library Hardware interface Device#1 Device#n Hardware … … …

  12. Case study choice explanation • ATV spacecraft • Uses current space software techniques(software under development) • Gyra application • Easy access to Gyra specification documents • Focus on the FSM • Heavily used in ATV: 28 FSMs, 989 transitions • Gyra FSM has an average size (38 transitions) • FSMs are difficult/expensive to test • “Transition testing is one of the most, if not the most, demanding tasks of the product validation” EADS test plan  Would be valuable to automate test case generation

  13. Outline • B modelling for Leirios Test Generator • ATV software case study: Gyra functional Unit • Gyra B formal model • Modelled Finite State Machine • Requirement traceability • Automatic test case generation • Conclusion

  14. Modelled Finite State Machine ON_14 • Transitions are activated by commands • Redundant commands are accepted (requirement 16241) • FSM modes are groups of FSM states (OFF mode, ON mode) ON_1 ON_4 OFF_N ON_12 ON_34 ON_2 ON_N ON_3 ON_13 ON_24 ON_23

  15. Example: GYRA_TR_MODE ON_x (11 states) ON_x (11 states) ON_x (11 states) According to EADS:1 transitionModelling choice:11 different transitions Modelled FSM key figures • 12 states • 22 graphically represented transitions • + 14 different reflexive transitions • would be 102 more transitionson the picture • distinct fromcommand redundancy transitions  750 line model • NB: separated model for Survival modes(3 states, 2 transitions – 150 lines)

  16. Requirement traceability • Traceable Gyra requirements: 43 / 49 • Non traceable requirements (6) : • Initial values of variables (3), • No LTG tag for INITIALISATION part of the model • High level requirements, refined by others (2) • FSM mode shall be computed from FSM state (1) • That is to say not stored in memory – code inspection is required

  17. Outline • B modelling for Leirios Test Generator • ATV software case study: Gyra functional unit • Gyra B formal model • LTG automatic test case generation • Test generation criteria • Generated test cases • Ada test program generation • Conclusion

  18. Test generation criteria • Model coverage criteria: 4 possibilities • Decision coverage (DC) • Condition/Decision coverage (C/DC) • Modified condition/Decision coverage (MC/DC) • Multiple condition coverage (MCC) • An LTG test case is made of • 1. Preambule • 2. Body + Identification • 3. Postambule • With Gyra, test programs may end in any FSM state no need for a postambule Softwarecriticality

  19. Generated test cases • 179 functional abstract test cases generated • = 179 operation activation sequences • XML format (HTML and RTF translators) • Comparison to EADS test plan • Chaining of several tests (8 large test sequences) difficult to compare • Traceable requirements covered by LTG: 37/43 • Difficulty with generic requirements (5) • Multiple tags for one requirement • Were those requirements fully covered ? • 1 requirement was not covered: • some behaviours, related to 1 B operation, were not reached • different generation criteria should solve the problem. HTML example

  20. EADS test plan coverage • Gyra FSM state and transition coverage ON_14 ON_1 ON_4 OFF_N ON_12 ON_34 ON_2 ON_N ON_3 ON_13 ON_24 ON_23

  21. Example: GYRA_TR_MODE ON_x (11 states) In EADS test plan:1 test caseOur choice:11 test cases Coverage comparison • State coverage • EADS test plan: 9/12 • LTG: 12/12 • Graphically represented transition coverage • EADS test plan: 14/22 • LTG : 22/22 • Reflexive transition coverage • EADS: 13/14, tested on one state • LTG: 13/14, tested on every state • uncovered transition is relatedto uncovered B operation

  22. Ada test program generation • Required to generate test programs from test cases: • Tagged test program pattern • Relation mapping between B operations and Ada services • Test program generation on a simple model • Test program example • Not enough time to generate test programs

  23. Outline • B modelling for Leirios Test Generator • ATV software case study: Gyra functional unit • Gyra B formal model • LTG automatic test case generation • Conclusion

  24. LEIRIOS Test Generator™ 24 B Model (750 lines) Functional Modeling Model Validation LTG Model Animator 49 Gyra functional requirements Test Generation Criteria LTG Test Case Generator done to do Test Case Generation 179 Test Cases LTG eXecutable Script Builder Test Driver Schemas Test Execution Environment Test Driver Generation

  25. Conclusion • Technology evaluation: B notation • Particularly suited to model finite state machines • Difficulty to model real time aspects like scheduling • Added requirement traceability • Some restrictions on the notation • Automatically generated test cases on the case-study • 15 hour computation on a desktop PC, for 179 test cases • Coverage is similar or better compared to EADS test plan • LTG product stability, documentation quality to be raised • Transition from research prototypeto commercial product is in progress

  26. Perspectives • Short-term perspectives • Generate and execute Gyra FSM test programs • 2 possibilities: by Estec or by EADS • Structural coverage study • Fault injection in the Gyra implementation to evaluate test efficiency • Generate tests for Gyra algorithms • Long-term perspectives • LTG input format comparison for space projects • B notation • Statecharts • UML

  27. Automatic test case generationfrom a formal model LTG technology evaluation based on a B model of a part of ATV onboard software Julien ORSAT – Stagiaire from the University of Franche-Comté (France) 08/03 – 23/07/2004 Supervisor: Philippe CHEVALLEY

More Related