1 / 23

Conformance

Conformance. Lynne S. Rosenthal Mary Brady National Institute of Standards and Technology Lynne.rosenthal@nist.gov Mary.brady@nist.gov. Outline. Terminology Testing Methodologies Benefits of Conformance Testing Test Suites for XML Technologies XML, DOM, XSLT, XSL FO, XML Schema.

mcathcart
Download Presentation

Conformance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Conformance Lynne S. Rosenthal Mary Brady National Institute of Standards and Technology Lynne.rosenthal@nist.gov Mary.brady@nist.gov

  2. Outline • Terminology • Testing Methodologies • Benefits of Conformance Testing • Test Suites for XML Technologies • XML, DOM, XSLT, • XSL FO, XML Schema Conformance clause Conformance Testing Test suites Test tools Certification Validation Certificates Branding Procedures, reporting

  3. Unscrambling the Terminology Conformance Specification Conformance Clause Conformance Testing test suite, test tools Validation process, reporting Certification certificate, brand

  4. Terminology - Conformance • CONFORMANCE - the fulfillment of a product, process or service of specified requirements (ISO Guide 2) • These requirements are specified in a standard or specification as part of a conformance clause or in the body of the specification • CONFORMANCE CLAUSE - a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance

  5. Terminology - Conformance Testing • CONFORMANCE TESTING – a way to verify implementations of a specification to determine whether or not deviations from the specifications exist (through the use of test suites) • tests capture the technical requirements within the specification • testing is bound in scope by the specification • internal structure of product-under-test not accessible (black box testing) • Necessary, but not sufficient, for interoperability Note: referred to as conformity assessment in ISO Guide 2

  6. Input Output Testing Methodology • Falsification Testing • find errors by means of experimentation • show presence of errors not their absence • prove non-conformance; can never prove conformance • Tests Based on: • required functionality • logical errors by misunderstanding requirements • errors from boundary conditions and divergence • common programming errors

  7. Specification Requirements Implementation Conformance Tests Test programs

  8. Terminology - Testing Process • VALIDATION - process necessary to perform conformance testing in accordance with a prescribed procedure and official test suite • ensures that testing can be repeatable and reproducible • ensures that conclusions are consistent with facts presented in the evaluation • CERTIFICATION - acknowledgement that a validation was completed and the criteria established by CIO for issuing certificates (brands) was met.

  9. Testing Ingredients • Certification • qualified bodies to do the testing and certification • control board - advisory and arbiter Validation Process - policy and procedures for testing • Conformance Testing • Test assertions • Test suite • (test software, test scripts, test criteria) • Standard • Conformance clause

  10. Benefits of Conformance Testing • Increased probability that products are implemented correctly • Contains required functionality • Behaves as expected • Performs functions in a known manner • Increased likelihood of portability and interoperability • Portability - move software or applications among different systems • Interoperability – ability of 2+ systems to exchange and use information

  11. Benefits of Conformance Testing cont. • For Developers of Specifications • Feedback loop • For Software Developers • Help to improve quality - identify areas in which they conform or not conform • For Buyers • Increases buyer’s confidence in a product and its ability to meet their needs • Substantiate seller’s claims • Not locked into purchasing from a single vendor

  12. Test Harness Test Cases Test Scenario Test Assertions, Axioms, Semantic Requirements Specification Components of a Test Suite

  13. OASIS XML Conformance • XML Conformance Committee • Develop tests to improve the quality of XML processors • Provide a forum to discuss issues related to interoperability • Provide input to W3C regarding errors/ambiguities • Deliverables • Test Suite containing over 2000 tests, Second Edition • Associated test report • Built using XML/XSLT

  14. XML Test Suite • Types of XML Tests • Binary • Validating processors must check validity constraints. • Non-validating processors do not have to read external entities. Accordingly, they may not report errors which would have been detected had those entities been read. • Output Tests • Requirements on XML processors to ensure that they pass the necessary information on to applications.

  15. XML Test Matrix

  16. XML Test Suite • Test Report • XSL Stylesheets • IE5, HTML renditions Testcases.dtd Conformance.xml • Test Profile • Test URI • Unique ID • Spec Section • Type • Entities • Description • NIST / OASIS • SUN Microsystems • James Clark • FUJI Xerox • IBM • XML.COM • Java application • Tests XML processors • Dave Brownell Test Case DTD XML Test Description Files Outputs <TESTCASES PROFILE=“SUN Microsystems XML Tests”> <TEST URI=“sun/invalid/e101.xml” ID=“e101” SECTIONS=“3 [39]” TYPE=“invalid” ENTITIES=“none”> Tests the Element Valid VC by including an undeclared child element.</TEST> </TESTCASES>

  17. XML Test Suite Results, v1 • XML.COM Conformance Update – Dave Brownell • Validating Parsers • Tested JAXP 1.0, IBM Xerces/Java 1.0.3, Layered Validator (distributed with Aelfred 2), MSXML3 Technology Preview, Oracle V2.0.2.7 • Pass Rate 82% - 99% • Non-validating Parsers • Tested all of the above plus XP 0.5 • Pass rate 86% - 99% • Microsoft Update – July 2000 MSXML 3.0 Beta – Chris Lovett • Non-validating: DOM: 95%, SAX: 97% • Validating: 97%

  18. Test Harness – automatically runs available tests and presents user with a color-coded table of results; gives access to spec, assertions, and source code Test Scenario Load appropriate XML or HTML file, exercise DOM calls Test Assertions – Fundamental, Extended, HTML Interfaces DOM L1 Specification NIST DOM Test Suite Test Cases 200 Java – fundamental, extended 800 ECMAScript fundamental, extended, HTML

  19. DOM Test Suite • Test Results: • Run tests against IE5, Mozilla (some fundamental interfaces), Xerces, JAXP, and Oracle; Pass rate: 82 – 98% • Implementations differ greatly in the way that parsers handle white-space and entity expansion issues – differences in DOM tree • DOM Level 2 • Working with W3C to identify resources • Modified Java tests to be consistent with SUN contribution • Will update NIST tests to L2 status

  20. XSLT Test Suite • OASIS Technical Committee, chaired by G. Ken Holman • Member contributions include Lotus, NIST, Sun • Expands on XML Test Description file • Dave Marston, XSLT/XPATH Conformance Wed, 4pm, Delaware Room • Public Information Meeting, Wed, 8pm, Suite 8228

  21. XSL FO Test Suite • W3C WG to test Candidate Recommendation • Develop test assertions for all basic formatting objects and properties • Develop test DTD and associated harness • Work with vendors to categorize available tests • Research translating available CSS and XSLT tests • Develop tests for additional spec components • Expected March 2001

  22. XML Schema Test Suite • Test Control • Test Types • # schemas • # instances Test Generator Schema Tests • Constraints • Structure • Limits • Special info

  23. Summary • Interesting URL’s • http://www.nist.gov/xml/ • http://www.w3.org/ • http://www.oasis-open.org/ • Questions???

More Related