Property-Based Test Generation Li Tan, Oleg Sokolsky, and Insup Lee University of Pennsylvania

1. Property-Based Test GenerationLi Tan, Oleg Sokolsky, and Insup LeeUniversity of Pennsylvania

2. The Overview of Our Approach LTL formulae F1, …, Fn Quasi linear (Lasso-shape) proof structure Feature conflict detection Model Checker Test trace generator Behavior specification LTL formulae F1, …, Fn Temporal Property Translator Traces {r1,…,rn} Criteria not being covered Specification Model (CHARON) Simulation-based randomized test generator Traces Coverage Criteria Testing result System Modeling Environ. Modeling Interface Definition Informal System Specification Hardware Specification/limitation Test Harness Implementation Test result

3. Goal: Using model-checking technique to make test generation more efficient, flexible, and centered on the system-specific properties (features). Step I. Preparing specifications • Properties (feature specification) as linear temporal logic formula • (optional) System specification system as CHARON (for hybrid systems) and EFSM (for discrete systems) Step II. Test generation using model checkers. • (For hybrid systems) Simulation-based test generation with the assistance of predicate abstraction reachability analysis. • (For Discrete system) • (Option A) Using the proof structures of evidence-ready model checkers. • (Option B) Reducing the test generation for LTL formula to safety check • (For temporal specification only) Functional test. • Generating non-trivial test traces for temporal specification (feature specification) • Detecting conflicting in temporal specification. Step III: Realizing test harness. Model-checking based Test generator Temporal (feature) Spec. + Model (optional) Test suite (Finite set of finite traces)

4. I. From infinite length to finite: Synthesizing test suites for 9LTL property From Property and Model to Test suite: Property-based test generation

5. A infinite Lasso-shaped test suite can be checked adequately by finite steps if the implementation is bounded. Turn=1, :c1, : c2 Turn=1, c1, : c2 + Turn=2, c1, : c2 Turn=2, c1, c2 Turn=1, c1, c2 Estimating the number of relevant implementation states using slicing Turn=1, c1, c2 A quasi-linear proof skeleton A finite test suite

6. Test Generation using Model Checkers Option A: Modifying model checkers and retaining proofs. Option B: Using the idea of reducing LTL model checking to reachability analysis [A. Biere etc], but enhancing the observer to retain proof SMV model SMV model + Extended Observer Model + Repetition information Linear Temporal Logic Specification Extracted Proof Generated test trace

7. II. From infinite numbers of traces to finite: selecting interesting traces System properties are required to be held on all the paths, we will select only nontrivial paths, whose characteristics are caught by ELTL formula systematically deriving from the properties. LTL f => ELTL formulae a2e(f)={f(f’ !ð(f’))|f’ Áf} F= G(req -> F(cancelÇresponse)) FÆ (: G(req ! F(cancelÇ false))) = FÆ F(req Æ G(: cancel)) Test the case that no cancel follows a request (hence a reponse must be placed) FÆ (: G(req ! F(false Ç response))) = FÆ F(req Æ G(: response)) Test the case that no response follows a request (hence a cancel must be placed) FÆ (: G(true ! F(cancel Ç response))) = FÆ FG(: (cancel Ç response)) Test the case that no cancel or reponses occurs after time t, (hence should not a request occur). FÆ (: G(req ! false)) = FÆ F(req) Test the case that there is request

8. So, what if only behavior (feature) specification is available …… From only Property to Test suite: Functional test generation LTL formulae F Nontrivial ELTL formulae Derived from F Y=a2e(F) f02FÆ 2 Y f12FÆ 2 Y fn2FÆ 2 Y ……. Buchi automaton B0 Buchi automaton B1 Buchi automaton Bn Check nonemptiness Check nonemptiness A trace satisfies f0 A trace satisfies f1 A trace satisfies fn

9. Testing Hybrid system: simulation-based test generator with predicate-abstract reachability analysis System Modeling Coverage Criteria No Bad set Reachability Checker CHARON (Model) Flatten hybrid model Predicate set NO w/ more predicates Simulation /refinement Yes w/ Trace YES Test Suite Concretize Implementation

10. An implementation of simulation-based test generator a. CHARON simulator with test generator b. Progress report of test generator c. Visual display of generated test traces.

11. Charon model for CARA Charon model For patient Closed Charon Model for CARA Test harness as I/O Interface CARA simulator /model-generated code Standalone executable program Realizing Test Harness Test trace Variable back_EMF Value Time 60.0 0.001 70.0 0.002 …… Simulation-based test generator Coverage criteriae Test Result

12. Conclusion • Applying model-checking technique to traditional domain of test generation is appealing. • Test generation is centralized on system-specific properties • State-of-art model checkers may be adapted as general purpose test generator (and think properties as programs ). • Techniques in model checking may help find interesting test traces and provide new angle to view and think test generation. • Property-based test generation requires integrated efforts. • Test generation ¹ witness generation. • Proof is necessary to generate partial test suite and perform optimization. • Proof is also needed to extend the notion of “testable” properties. • Model-based code generation may help build test harness.