1 / 36

Integration Testing of Components Guided by Incremental State Machine Learning

Keqin Li, CNRS LSR-IMAG Roland Groz, INPG LSR-IMAG Muzammil Shahbaz, France Telecom August 2006. Integration Testing of Components Guided by Incremental State Machine Learning. Content. Motivation Background Framework Scenario Testing Unit Testing Integration Testing Conclusion.

kaoru
Download Presentation

Integration Testing of Components Guided by Incremental State Machine Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Keqin Li, CNRS LSR-IMAG Roland Groz, INPG LSR-IMAG Muzammil Shahbaz, France Telecom August 2006 Integration Testing of Components Guided by Incremental StateMachine Learning

  2. Content Motivation Background Framework Scenario Testing Unit Testing Integration Testing Conclusion

  3. Motivation Software development is more and more based on component integration Unknown Internal structures and insufficient documents Only partial and informal information is available Provide a required system assembling COTS of which he has a limited knowledge Unit testing and integration testing Have to rely on intuitions and little support is available Integrator: Component:

  4. Our Approach Infer formal models of components from behaviours observed during tests Perform integration testing based on the formal models Iteratively improve the formal models

  5. Background - Angluin’s Algorithm [1987] Infer formal model (conjecture) from observed behavior (observation table) For deterministic finite-state acceptor Teacher answers Membership query Equivalence query

  6. Process Add strings to Observation Table No No Closed & Cons. Empty Observation Table Membership Query Yes Equivalence Query Yes Yes? End String Accepted Or Not Yes Or Counterexample Conjecture Teacher

  7. Assumptions Black boxes with known interfaces Only external interfaces are controllable Slow environment Test scenarios

  8. Framework Scenario Testing Unit Testing Integration Testing Consult Domain Expert Initial Observation Tables Models of Components

  9. Models C.E. C.E. I. T. I. T. I. T. U. T.: Unit Testing I. T.: Integration Testing C.E.: Counter- example M1(1) M1(2) M1(3) U. T. U. T. U. T. M2(1) M2(2) M2(3) U. T. U. T. U. T. … … … Mn(1) Mn(2) Mn(3) U. T. U. T. U. T.

  10. Scenario Testing A test scenario is an input/output sequence containing external inputs and outputs A test case is constructed for each test scenario Replace problematic component Record observed behaviour of each component in its observation table

  11. Unit Testing Each Component is tested individually For each component Complete the initial observation table Using extended learning algorithm to construct closed and consistent observation table Make conjectured model

  12. Integration Testing Unit Testing Yes Coverage Criterion Satisfied? Discre- pency? Test Generation Test Execution No Yes No

  13. Test Generation based on Composition of FSMs M N [Petrenko 98] An algorithm to compute M N [Lima 97] Assign colour to “global” transitions An algorithm to generate tests covering colours [Kim 02]

  14. References [Petrenko 98] A. Petrenko, N. Yevtushenko, “Solving Asynchronous Equations,” Proceedings of FORTE’98, 1998 [Lima 97] L. Lima Jr., A. Cavalli, “A Pragmatic Approach to Generating Test Sequences for Embedded Systems,” Proceedings of IWTCS’97, 1997 [Kim 02] M. Kim, C. Besse, A. Cavalli, and F. Zaïdi, “Two Methods for Interoperability Tests Generation: An Application to the TCP/IP Protocol,” Proceedings of TestCom 2002, 2002.

  15. Dealing with Discrepancy Observe the behaviour of each component, and compare with its model Discrepancy Unexpected output sequence New input symbol

  16. Refine (relearn) model Counterexample or new input symbol has been found Add corresponding strings into observation table Perform learning algorithm to make the observation table closed and consistent again Make conjectured model

  17. Consulting Domain Expert Models of components and test results Domain expert Identify unexpected behaviours Propose additional test scenarios Iteration

  18. Results of Integration Testing For each component, we have a model The joint behaviour of these components has been systematically tested Faults might be revealed during testing

  19. Conclusion Combines machine learning and model-based testing for the integration of components Reuses Angluin’s algorithm Iteratively refines models

  20. The Models Just a “by-product” Our goal is not to build a complete model of the system Just used as basis for the testing process to: Generate integration tests of components Assess the coverage, hence the quality of the integration testing process

  21. The Next Steps Move to Parameterized FSM presentation by Muzammil Shahbaz Identify the type of approximation needed Experiment and case study Revisit model-based test generation strategies

  22. Thank you

  23. Backup

  24. Observation Table ε a ε 1 0 a 0 1 b 0 0 bb 1 0 aa 1 0 ab 0 0 ba 0 0 bba 0 1 bbb 0 0 E S S.A

  25. Closed ε a ε 1 0 a 0 1 b 0 0 bb 1 0 aa 1 0 ab 0 0 ba 0 0 bba 0 1 bbb 0 0 E tSA, sS, s.t., row(t)=row(s) S SA

  26. Consistent ε a ε 1 0 a 0 1 b 0 0 bb 1 0 aa 1 0 ab 0 0 ba 0 0 bba 0 1 bbb 0 0 E s1S, s2S, row(s1)=row(s2), Then, aA, row(s1a)=row(s2a) S SA

  27. Conjecture ε a ε 1 0 a 0 1 b 0 0 bb 1 0 aa 1 0 ab 0 0 ba 0 0 bba 0 1 bbb 0 0 E q0 S a b a b b q2 q1 SA a

  28. Observation Table a b ε x x a x y b y x aa x y ab y x ba x y bb y x row(ε) a/x b/x b/y row(a) row(b) a/x b/x a/y

  29. Example – Component Hotel Reservation OK/CS1 OK/CD1 No/F q3 q5 No/TL T/PD1 Dbl/F q2 Sgl/F T/PS1 No/HL q4 q6 No/TL H1/TL No/F N/HL q0 q1 H2/TL No/TL No/F q8 q10 T/PD2 Dbl/F q7 Sgl/F T/PS2 q9 q11 No/TL OK/CD2 No/F OK/CS2

  30. Initial Observation Table N H1 H2 Dbl Sgl T OK No ε HL Ω Ω Ω Ω Ω Ω Ω N Ω TL TL Ω Ω Ω Ω Ω H1 HL Ω Ω Ω Ω Ω Ω Ω H2 HL Ω Ω Ω Ω Ω Ω Ω Dbl HL Ω Ω Ω Ω Ω Ω Ω Sgl HL Ω Ω Ω Ω Ω Ω Ω T HL Ω Ω Ω Ω Ω Ω Ω OK HL Ω Ω Ω Ω Ω Ω Ω No HL Ω Ω Ω Ω Ω Ω Ω

  31. Add N to S N H1 H2 Dbl Sgl T OK No ε HL Ω Ω Ω Ω Ω Ω Ω N Ω TL TL Ω Ω Ω Ω Ω H1 HL Ω Ω Ω Ω Ω Ω Ω … … … … … … … … … No HL Ω Ω Ω Ω Ω Ω Ω N/N Ω TL TL Ω Ω Ω Ω Ω N/H1 Ω Ω Ω F F Ω Ω HL N/H2 Ω Ω Ω F F Ω Ω HL N/Dbl Ω TL TL Ω Ω Ω Ω Ω … … … … … … … … … N/No Ω TL TL Ω Ω Ω Ω Ω

  32. Conjecture OK/CD1 No/F row(N-H1-D) row(N-H1-D-T) No/TL T/PD1 No/HL D/F row(N-H1) N/HL H1/TL row(ε) row(N) S/F row(N-H1-S-T) T/PS1 row(N-H1-S) H2/TL No/TL No/F OK/CS1

  33. Example – Component User Interface UI_Dbl/Dbl UI_Sgl/Sgl UI_No/No UI_T/T UI_No/No PD1/UO_PD1 PS1/UO_PS1 PD2/UO_PD2 PS2/UO_PS2 UI_H1/H1 UI_H2/H2 UI_OK/OK UI_No/No UI_Name/N HL/UO_HL F/UO_F TL/UO_TL q0 q1 q2 q3 q4 TL/UO_TL F/UO_F HL/UO_HL CD1/UO_CD1 CS1/UO_CS1 CD2/UO_CD2 CS2/UO_CS2

  34. Integration Testing Environment oMN cM User Interface (M) Hotel Reservation (N) oM oNM

  35. Counterexample UI_Name/UI_H2/UI_Dbl/UI_T The real input sequence of Hotel Reservation is: N/H2/Dbl/T The corresponding output sequence is: HL/TL/F/PD2 According to the model, the output should be: HL/TL/F/PD1

  36. Comparing with Angluin’s Algorithm Add strings to Observation Table Yes No Closed & Cons. No Counter- example? Initial Observation Table Unit Testing Yes Integration Testing End Test Output Output Test Component Component A Component B Scenario Testing

More Related