Presentation Transcript

1. CitLab: a Laboratory for CombinatorialInteraction Testing

   Workshop on Combinatorial Testing (CT) @ ICST 2012 April 17, 2012 Angelo Gargantini Università di Bergamo - Italy http://cs.unibg.it/gargantini Joint work with Paolo Vavassori In collaboration with Andrea CalvagnaUniversità di Catania
2. Some open issues in CTuser perspective Still missing A common abstract language with a precise semantics for concepts like constraints, seeds … A standard concrete syntax/grammar or exchange format for tools: to guarantee interoperability Limited assistance in writing the models generation tools do not offer any editing capabilities (only a grammar and a parser) and are rarely integrated in any IDE for programming or design. Angelo Gargantini - CitLab: a Laboratory for Combinatorial
3. Some open issues in CT researcher perspective No standard language  Comparison among tools and algorithms is difficult Conceptual point of view: some features may be supported in an approach but not in another or different meanings Practical: every comparison is quite unreliable since every user must redefine in its own language and tool the examples No help in defining new algorithms and tools In order to experiment a new test generation algorithm, a researcher should define a proper grammar and a parser, develop the libraries to manipulate the model data, and translate the benchmarks Angelo Gargantini - CitLab: a Laboratory for Combinatorial
4. CitLabin brief (Domain Specific) Language A rich abstract language with a precise formal semantics A concrete syntax with a well defined grammar by XTEXT A textual editorintegrated in the eclipse IDE A Framework based on the Eclipse Modeling Framework (EMF) which provides tools and run-time support to manipulate combinatorial problems in Java A rich collection of Java utility classes and methods For introducing new test generation algorithms For introducing code translators for importing and exporting models and tests Angelo Gargantini - CitLab: a Laboratory for Combinatorial
5. A DSL for CIT problems Domain-specific languages (DSLs) are languages tailored to a particular problem domain. They offer substantial gains in expressiveness and ease of use compared with general-purpose programming languages CIT can be considered a specific application domain for which developing a domain-specific language is worthwhile. DSL development is supported by several tools and technologies. Xtext Angelo Gargantini - CitLab: a Laboratory for Combinatorial Xtext is a framework for development of domain specific languages. It covers all aspects of a complete language infrastructure, from parsers, over linker, compiler or interpreter to fully-blown top-notch Eclipse IDE integration.
6. Language Model .xtxt Grammar Manual (required) Manual (optional) Generator Generated Angelo Gargantini - CitLab: a Laboratory for Combinatorial Language project Editor Project AntlrParser .ecore (metamodel) Editor Syntaxcoloring Outline Content Assist Template Constraints Java API … Formatting
7. A Domain Specific language for CIT problems Angelo Gargantini - CitLab: a Laboratory for Combinatorial
8. CitLabLanguage in a glance Types ModelModel Types: ... end Parameters: ... end Constraints: # ... # end Seeds: # ... # end TestGoals: # ... # end Parameters Angelo Gargantini - CitLab: a Laboratory for Combinatorial Constraints Seeds Test goals
9. Parameters and their types To describe a combinatorial problem would be sufficient to specify the number of variables and their cardinality. CITLAB language forces the designer to name parameters and to specify their types by listing all the values in their domain. Angelo Gargantini - CitLab: a Laboratory for Combinatorial Enumerative for parameters that can take a value in a set of symbolicconstants. Boolean for parameters that can be either true or false. Numerical value for parameters that take any value in an integerrange. Parameters: BooleanemailViewer; RangetextLines [ 25 .. 30 ]; Enumerativedisplay { 16MC 8MC BW }; end Choice:explicitparameternames to facilitate the modeling of real systems and to ease the specification of constraints and seeds From ACTS
10. Named Types Types can be defined with their name in the Types section to be used in parameters declaration Instead of an “anonymous” type. Angelo Gargantini - CitLab: a Laboratory for Combinatorial Types: EnumerativeTypecameraType { 2MP 1MP NOC }; end Parameters: EnumerativerearCamera : cameraType; EnumerativefrontCamera : cameraType; ... end Advantages: the use of namedparametertypes to make more compact and more maintainable the models in case manyparameters share the same domain.
11. Constraints In CITLAB, we adopt the language of propositional logic with equality and arithmetic to express constraints General Form (GF) constraints propositional calculus and Boolean operators aor b => c and d equality and inequality emailViewer = trueorfrontCamera ~= NOC arithmetic over the integers relational and arithmetic operators for numeric terms textLines >= threshold + 10 A test t satisfies a constraint C or it is a model of C iffC is true by substituting each parameter pi in C with the value of piin t Angelo Gargantini - CitLab: a Laboratory for Combinatorial
12. Constraints Semantics A test t is valid if it is a model of all the constraints , i.e., or equivalently Challenge: finding “valid” “complete” test suites Because of constraints, some combinations are not coverable A test suite must cover “only” all the coverable combinations Finding a (if any) model is an NP problem = SAT problem Checking if a test satisfies a constraint is however linear Checking if a combination violates the constraints is linear Challenge: assuring constraints consistency constraints are consistent if there exits at least a test that satisfies them, i.e., Consistency checking is again a SAT problem FUTURE WORK: use a SAT/SMT solver for this Angelo Gargantini - CitLab: a Laboratory for Combinatorial
13. GF vs forbidden combinations Most CIT that support constraints, allow only the use of forbidden combinations or tuples A forbidden combination would be translated in our model as a not statement. a forbidden pair x = a; y = b would be represented # not (x = a and y = b) # Angelo Gargantini - CitLab: a Laboratory for Combinatorial Advantages: Much more compact than forbidden tuples Example: if b {x1 … xn } “a = x => b = xk” would need n-1 forbidden tuples More expressive (whilst in a finite domain …) More similar to the natural language No need to distinguish between implicit and explicit constraints
14. Seeds The testers can also force the inclusion of their favorite test cases by specifying them as seed tests. They can be tests generated by other criteria Critical complete combinations The seeds must be included in the generated test set without modification CITLAB considers only complete seeds, i.e., seeds that assign a valid value to each parameter # frontCamera==1MP, display==16MC, emailViewer=true, rearCamera==2MP, textLines=30# Partial seeds? Critical partial combination the tester would like to be included in the test suite Angelo Gargantini - CitLab: a Laboratory for Combinatorial
15. Test Goals Critical situation that must be tested Predicates that must be covered by some tests Test goals can be again in GF (as the constraints): // at least one camera is missing and the display has at least threshold lines ( rearCamera==NOC orfrontCamera==NOC) andtextLines>=threshold Partial combinations // both rearCamera and frontCamera are NOC and display is 16MC rearCamera==NOC andfrontCamera==NOC and display==16MC Remarks: Checking feasibility of test goals is an NP-complete problem, since it can be reduced to a SATisfiabilityproblem Checking if a test covers a test goal is however linear Angelo Gargantini - CitLab: a Laboratory for Combinatorial
16. CitLabEditor Angelo Gargantini - CitLab: a Laboratory for Combinatorial DEMO
17. Editor features Syntax Coloring Content Assist Template Proposals Rich Hover Rename Refactoring Quick Fixes Outline Folding Hyperlinks for all Cross References Find References Toggle Comment Mark Occurrences Formatting Angelo Gargantini - CitLab: a Laboratory for Combinatorial
18. Example, model validation XTEXT provides several levels of validation for the defined language the user can specify additional constraints for the model by providing validation fragments Angelo Gargantini - CitLab: a Laboratory for Combinatorial Example: No seed can violate any constraint Note: checking if a seed violates a constraints is decidable in a linear time
19. AN EXTENSIBLE FRAMEWORK FOR CIT Angelo Gargantini - CitLab: a Laboratory for Combinatorial
20. CitLab: as a extensible framework For the definition and implementation of test generators and a set of exporters/importers to and from other languages Based on the eclipse plugin development environment Goals to foster tools interoperability to favor reuse of artifacts to allow fair comparison generator exporter Angelo Gargantini - CitLab: a Laboratory for Combinatorial Random generator Casa exporter CitLab framework
21. Extension points and extensions CitLab accepts new contributions as plugins by defining extension points Contributors add new plugins by implementing extensions Declare the extension in .xml Implement an Interface Angelo Gargantini - CitLab: a Laboratory for Combinatorial
22. CitLab:extension points Angelo Gargantini - CitLab: a Laboratory for Combinatorial
23. Using Model to Text or Model to Model MDE approach: apossible way to define importers and exporters is to use Model to Text (M2T) or Model to Model (M2M) transformations For instance, Xtend2 for CASA exporter deftoCasaCode(Model sm,int n){ ''' «n» «sm.parameters.size» «FORparam :sm.parameters»«param.getSize»«ENDFOR» ''' } Angelo Gargantini - CitLab: a Laboratory for Combinatorial
24. Test Suite meta-model and Java utils a simple meta-model for tests and test suites We have developed several classes and methods capable to perform routine tasks like generating all the test requirements for a given strength, checking if a test or a seed violates some constraints checking if a test goal is covered by a given test suite … Angelo Gargantini - CitLab: a Laboratory for Combinatorial
25. CitLab site and Licenceissues All the code is available at http://code.google.com/a/eclipselabs.org/p/citlab/ CitLabis released under the Eclipse Public License (EPL) an open source software license used by the Eclipse Foundation a business-friendly free software license copyleft but weaker than other, like GNU General Public License (GPL) can use, modify, copy and distribute the work and modified versions, in some cases being obligated to release their own changes CitLab Plugins can be commercial (distribute as binaries) Any modification of the CitLab framework must be open sourced The EPL is approved by the Open Source Initiative (OSI) and is listed as a "free software license" by the Free Software Foundation (FSF). Angelo Gargantini - CitLab: a Laboratory for Combinatorial
26. Status and Future work Adding new plugins is easy (as student projects) Paolo Vavassori has a part-time contract for 1 year Already implemented Generator Random with seeds support IPO and AETG work in progress Model exporter To CASA (constraints still in progress) Test suite exporter To excel /CSV file New planned features Language, e.g. output for oracles, arrays, … Meta-algorithms (like: run 50 times and take the best) Test suite postprocessor, e.g. remove unnecessary test cases Angelo Gargantini - CitLab: a Laboratory for Combinatorial
27. Objectives and conclusions CitLab Language & editor Establish a CIT “language” with a precise semantics New concepts and constructs? Exchange models and benchmarks CIT “knowledge base” A user friendly textual editor integrated with the eclipse IDE CitLab plugin-based environment defining new generators to foster contributions from new parties making comparison easier and more reliable Angelo Gargantini - CitLab: a Laboratory for Combinatorial