1 / 23

Conformance - PowerPoint PPT Presentation

  • Updated On :

Conformance. Lynne S. Rosenthal Mary Brady National Institute of Standards and Technology [email protected] [email protected] Outline. Terminology Testing Methodologies Benefits of Conformance Testing Test Suites for XML Technologies XML, DOM, XSLT, XSL FO, XML Schema.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Conformance' - dericia

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript


Lynne S. Rosenthal

Mary Brady

National Institute of Standards and Technology

[email protected]

[email protected]


  • Terminology

  • Testing Methodologies

  • Benefits of Conformance Testing

  • Test Suites for XML Technologies

    • XML, DOM, XSLT,

    • XSL FO, XML Schema



Conformance Testing

Test suites

Test tools





Procedures, reporting

Unscrambling the terminology
Unscrambling the Terminology



Conformance Clause

Conformance Testing

test suite, test tools


process, reporting


certificate, brand

Terminology conformance
Terminology - Conformance

  • CONFORMANCE - the fulfillment of a product, process or service of specified requirements (ISO Guide 2)

    • These requirements are specified in a standard or specification as part of a conformance clause or in the body of the specification

  • CONFORMANCE CLAUSE - a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance

Terminology conformance testing
Terminology - Conformance Testing

  • CONFORMANCE TESTING – a way to verify implementations of a specification to determine whether or not deviations from the specifications exist (through the use of test suites)

    • tests capture the technical requirements within the specification

    • testing is bound in scope by the specification

    • internal structure of product-under-test not accessible (black box testing)

  • Necessary, but not sufficient, for interoperability

    Note: referred to as conformity assessment in ISO Guide 2

Testing methodology



Testing Methodology

  • Falsification Testing

    • find errors by means of experimentation

    • show presence of errors not their absence

    • prove non-conformance; can never prove conformance

  • Tests Based on:

    • required functionality

    • logical errors by misunderstanding requirements

    • errors from boundary conditions and divergence

    • common programming errors




Conformance Tests

Test programs

Terminology testing process
Terminology - Testing Process

  • VALIDATION - process necessary to perform conformance testing in accordance with a prescribed procedure and official test suite

    • ensures that testing can be repeatable and reproducible

    • ensures that conclusions are consistent with facts presented in the evaluation

  • CERTIFICATION - acknowledgement that a validation was completed and the criteria established by CIO for issuing certificates (brands) was met.

Testing ingredients
Testing Ingredients

  • Certification

    • qualified bodies to do the testing and certification

    • control board - advisory and arbiter


Process - policy and procedures for testing

  • Conformance Testing

    • Test assertions

    • Test suite

      • (test software, test scripts, test criteria)

  • Standard

    • Conformance clause

Benefits of conformance testing
Benefits of Conformance Testing

  • Increased probability that products are implemented correctly

    • Contains required functionality

    • Behaves as expected

    • Performs functions in a known manner

  • Increased likelihood of portability and interoperability

    • Portability - move software or applications among different systems

    • Interoperability – ability of 2+ systems to exchange and use information

Benefits of conformance testing cont
Benefits of Conformance Testing cont.

  • For Developers of Specifications

    • Feedback loop

  • For Software Developers

    • Help to improve quality - identify areas in which they conform or not conform

  • For Buyers

    • Increases buyer’s confidence in a product and its ability to meet their needs

    • Substantiate seller’s claims

    • Not locked into purchasing from a single vendor

Components of a test suite

Test Harness

Test Cases

Test Scenario

Test Assertions, Axioms,

Semantic Requirements


Components of a Test Suite

Oasis xml conformance
OASIS XML Conformance

  • XML Conformance Committee

    • Develop tests to improve the quality of XML processors

    • Provide a forum to discuss issues related to interoperability

    • Provide input to W3C regarding errors/ambiguities

  • Deliverables

    • Test Suite containing over 2000 tests, Second Edition

    • Associated test report

    • Built using XML/XSLT

Xml test suite
XML Test Suite

  • Types of XML Tests

    • Binary

      • Validating processors must check validity constraints.

      • Non-validating processors do not have to read external entities. Accordingly, they may not report errors which would have been detected had those entities been read.

    • Output Tests

      • Requirements on XML processors to ensure that they pass the necessary information on to applications.

Xml test suite1
XML Test Suite

  • Test Report

  • XSL Stylesheets

  • IE5, HTML renditions



  • Test Profile

  • Test URI

  • Unique ID

  • Spec Section

  • Type

  • Entities

  • Description


  • SUN Microsystems

  • James Clark

  • FUJI Xerox

  • IBM


  • Java application

  • Tests XML processors

  • Dave Brownell

Test Case DTD

XML Test Description Files


<TESTCASES PROFILE=“SUN Microsystems XML Tests”>

<TEST URI=“sun/invalid/e101.xml” ID=“e101” SECTIONS=“3 [39]” TYPE=“invalid” ENTITIES=“none”>

Tests the Element Valid VC by including an undeclared child element.</TEST>


Xml test suite results v1
XML Test Suite Results, v1

  • XML.COM Conformance Update – Dave Brownell

  • Validating Parsers

    • Tested JAXP 1.0, IBM Xerces/Java 1.0.3, Layered Validator (distributed with Aelfred 2), MSXML3 Technology Preview, Oracle V2.0.2.7

    • Pass Rate 82% - 99%

  • Non-validating Parsers

    • Tested all of the above plus XP 0.5

    • Pass rate 86% - 99%

  • Microsoft Update – July 2000 MSXML 3.0 Beta – Chris Lovett

    • Non-validating: DOM: 95%, SAX: 97%

    • Validating: 97%

Nist dom test suite

Test Harness – automatically runs available tests and presents user with

a color-coded table of results; gives access to spec, assertions, and source code

Test Scenario

Load appropriate XML or HTML file, exercise DOM calls

Test Assertions – Fundamental, Extended,

HTML Interfaces

DOM L1 Specification

NIST DOM Test Suite

Test Cases

200 Java – fundamental,


800 ECMAScript

fundamental, extended, HTML

Dom test suite
DOM Test Suite

  • Test Results:

    • Run tests against IE5, Mozilla (some fundamental interfaces), Xerces, JAXP, and Oracle; Pass rate: 82 – 98%

    • Implementations differ greatly in the way that parsers handle white-space and entity expansion issues – differences in DOM tree

  • DOM Level 2

    • Working with W3C to identify resources

    • Modified Java tests to be consistent with SUN contribution

    • Will update NIST tests to L2 status

Xslt test suite
XSLT Test Suite

  • OASIS Technical Committee, chaired by G. Ken Holman

  • Member contributions include Lotus, NIST, Sun

  • Expands on XML Test Description file

  • Dave Marston, XSLT/XPATH Conformance Wed, 4pm, Delaware Room

  • Public Information Meeting, Wed, 8pm, Suite 8228

Xsl fo test suite
XSL FO Test Suite

  • W3C WG to test Candidate Recommendation

  • Develop test assertions for all basic formatting objects and properties

  • Develop test DTD and associated harness

  • Work with vendors to categorize available tests

  • Research translating available CSS and XSLT tests

  • Develop tests for additional spec components

  • Expected March 2001

Xml schema test suite
XML Schema Test Suite

  • Test Control

  • Test Types

  • # schemas

  • # instances

Test Generator



  • Constraints

  • Structure

  • Limits

  • Special info


  • Interesting URL’s




  • Questions???