a conformance test suite for ttcn 3 tools n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
A conformance test suite for TTCN-3 tools PowerPoint Presentation
Download Presentation
A conformance test suite for TTCN-3 tools

Loading in 2 Seconds...

play fullscreen
1 / 18

A conformance test suite for TTCN-3 tools - PowerPoint PPT Presentation


  • 168 Views
  • Uploaded on

A conformance test suite for TTCN-3 tools. Benjamin Zeiss, T-Systems Andras Kovacs, Broadbit Bogdan Stanca-Kaposta, Testing Technologies. Agenda. Motivation STF 409 Methodology Discussion Challenges Conclusion. Motivation (1/2). Many TTCN-3 compilers (~7 commercial, 2 internal)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

A conformance test suite for TTCN-3 tools


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
    Presentation Transcript
    1. A conformance test suite for TTCN-3 tools Benjamin Zeiss, T-Systems Andras Kovacs, Broadbit Bogdan Stanca-Kaposta, Testing Technologies

    2. Agenda • Motivation • STF 409 • Methodology • Discussion • Challenges • Conclusion A conformance test suite for TTCN-3 tools

    3. Motivation (1/2) • Many TTCN-3 compilers (~7 commercial, 2 internal) • High interest in the language • Complex standard: hundreds of pages • Estimation of ~5000 requirements • No way to check the compliance of TTCN-3 tools • TTCN-3 is based on ETSI standards • A conformance test suite for TTCN-3 tools is needed! • Quality of a test tool is critical • Errors in the test tool may influece test results. A conformance test suite for TTCN-3 tools

    4. Motivation (2/2) • Relevance: • For users: • Is my tool standards compliant? • Can I switch tools easily / vendor lock-in? • For tool vendors: • Can I improve the standard compliance of my tool? • Possibly later: compliance certification • For standard developers: • Where is need for clarification? A conformance test suite for TTCN-3 tools

    5. STF 409 (1/2) • Specialist Task Force initiated by ETSI MTS • Manpower: 3 persons w. 20 days each (60 days total) • Benjamin Zeiss (T-Systems/University of Göttingen) • Andras Kovacs (Broadbit) • Bogdan Stanca-Kaposta (Testing Technologies) • Dates: October 2010 – Early 2011 • Work is based on a framework given by TTCN-3 STF. • Task: • Devel. of a conformance test suite for TTCN-3 tools. • Creation of a thorough documentation for the ATS. A conformance test suite for TTCN-3 tools

    6. STF 409 (2/2) • Three deliverables: • ETSI TS 102 950-1 V1.1.1 (2011-04) – ICS • ETSI TS 102 950-1 V1.1.1 (2011-04) – TSS & TP • ETSI TS 102 950-2 V1.1.1 (2011-04) – ATS & IXIT • Approved by MTS in April 2011. • Downloadable from the ETSI download area right now! • Time spent: • 10 days each (30 work days) • Development of conformance tests. • 10 days each + extra time (30+ work days): • Validation, fixing of problems, documentation, tool development. A conformance test suite for TTCN-3 tools

    7. Methodology (1/7) • Five main steps: • Create clause priority list. Discuss with MTS. • Develop conformance tests. • Create documentation framework. • Generate documentation content. • Validate and fix. • Step 1 (creation of clause priority list): • Not all clauses can be covered. Selection has to be made. • How to prioritize? • Vendor feedback + most basic functionality. • There is no perfect selection! A conformance test suite for TTCN-3 tools

    8. Methodology (2/7) • Step 2 (development of conformance tests): • The ATS is not the test automation, but the test input! • In most cases: 1 module = 1 conformance test. • Two steps: • Specification of the test purpose: @purpose + @verdict • Implementation of the test in TTCN-3. • Conformance verdict: • Expected tool output is specified in the document tags (@verdict). • Purpose: • Also specified in the document tags (@purpose). • Test automation is not provided by the ATS! • Can be scripted easily by extracting the document tags. • Example Java source code can be provided! • Test suite structure: organized according to clauses. A conformance test suite for TTCN-3 tools

    9. Methodology (3/7) • Step 2 (development of conformance tests) cont.: • Expected output for verdict determination: • @verdict pass accept/reject [expectedoutput] • Examples: • @verdict pass reject • @verdict pass accept, ttcn3verdict:pass • @verdict pass accept, noexecution • Purpose specification: • @purpose documentreference, description • Documentreference: part:clause • Examples: • @purpose 1:5, Ensure that when the IUT loads a module containing some definitions before the module declaration then the module is rejected. A conformance test suite for TTCN-3 tools

    10. Methodology (4/7) • Step 2 (development of conformance tests) cont.: • Complete TTCN-3 test input example: /*************************************************** ** @purpose 1:5.2.2, Ensure that the IUT correctly handles the uniqueness of variable names in its scope. ** @verdict pass accept, ttcn3verdict:pass ***************************************************/ module Sem_050202_Uniqueness_001 { importfrom Sem_050202_Uniqueness_001_import { constall; } typecomponent GeneralComp {} function f_funcScope() { varboolean repeatedIdentifier := true; if(repeatedIdentifier==true) { setverdict(pass); } } testcase TC_Sem_050202_Uniqueness_001() runson GeneralComp { f_funcScope(); } control { execute(TC_Sem_050202_Uniqueness_001()); } } module Sem_050202_Uniqueness_001_import { // repeated indentifier // from imported module // is allowed constinteger repeatedIdentifier := 0; } A conformance test suite for TTCN-3 tools

    11. Methodology (5/7) • Step 3 (creation of documentation framework): • Already provided by STF393 (Proforma). • Split single document into three documents. • Step 4 (generate documentation content): • Only 60 days manpower, little time for documentation. • Working on a single source saves time. • Solution: • Write test purposes and expected tool outputs directly into the test inputs. • Generate tables for the documentation from the ATS: • Test purpose tables. • Test suite structure. • Much easier to maintain! • Documentation generation tools developed as part of STF409. A conformance test suite for TTCN-3 tools

    12. Methodology (6/7) • Step 5 (validation and fixing): • Validation and fixing took more than half of the development time! • STF internal validation with 4 tools: • TTworkbench: compile-time and execution. • TestCast: compile-time and execution. • IBM: some initial compile-time validation. • MTP: as reports are sent in. • Lots of feedback from Elvior and MTP! A conformance test suite for TTCN-3 tools

    13. Methodology (7/7) • Executing the ATS: TTCN-3 File TTCN-3 File TTCN-3 File Sem_060101_TopLevel_001.ttcn ATS ** @verdict pass accept, ttcn3verdict:pass Test input Sem_060101_TopLevel_001.ttcn NegSem_060303_component_types_001.ttcn … Expected output TTCN-3 Tool (test subject) Tool output Rejection as invalid Rejection to execute Execution with results (e.g., pass/inconc/fail) Tool output evaluation pass fail A conformance test suite for TTCN-3 tools

    14. Discussion pass TTCN-3 Conformance ATS TTCN-3 Tool Meaning? • Compliance does not imply safety from a vendor lock-in! • Testing is never complete! • With v1.1.1 of the ATS, about 32% of the core standard clauses are covered with at least one test. • Ideally, we should have many tests for each clause. • Compliance means less likelihood for vendor lock-in. • With better coverage and higher number of test cases, we improve the quality of this statement. • Compliance reduces the likelihood for major tool errors • Tests range from simple arithmetic operations to communication operations and verdict handling. • ATS checks syntax and semantics. • ATS does not check the correctness of every single tool message, but of the overall output. A conformance test suite for TTCN-3 tools

    15. Challenges (1/2) • Various standard clarifications were necessary: • 9 CRs for TTCN-3 core languages have been written. • Unclarified test cases have been excluded from the delivery ("on-hold"). • Tools sometimes handle the standard differently. • Tests need to be expressed as tool independent as possible. • Time management: • Proper validation and fixing is extremely time consuming. • More than 10 voluntary days have been actually spent. • Responsibilities: • Everyone mixes up STF393 and STF409 responsibilities (proforma). A conformance test suite for TTCN-3 tools

    16. Challenges (2/2) • Version numbers: • To which core language standard version does the conformance test suite relate to? • How to maintain compatibility across different language versions? • Against which version should the conformance test suite be developed in the future? • Target quality: • The target quality of the test suite should be fixed right in the beginning (Terms of Reference). A conformance test suite for TTCN-3 tools

    17. Conclusion • The conformance test suite is available now! • ATS Metrics: • Total number of conformance tests: 733 • Number of positive syntactic conformance tests: 110 • Number of negative syntactic conformance tests: 34 • Number of positive semantic conformance tests: 335 • Number of negative semantic conformance tests: 254 • Total number of clauses in the TTCN-3 standard: 342 • Total number of clauses (at least partially) covered: 110 • About 140 pages of test purposes. • Free to use for tool developers. Use it to improve your tool! • Bug reports: http://t-ort.etsi.org A conformance test suite for TTCN-3 tools

    18. Special thanks: Tool validation and comments Elvior, MTP, Conformiq, Ericsson, ISPRAS Jens Grabowski / University of Göttingen Questions? Feedback? A conformance test suite for TTCN-3 tools