Conformance
Download
1 / 23

Conformance - PowerPoint PPT Presentation


  • 171 Views
  • Updated On :

Conformance. Lynne S. Rosenthal Mary Brady National Institute of Standards and Technology [email protected] [email protected] Outline. Terminology Testing Methodologies Benefits of Conformance Testing Test Suites for XML Technologies XML, DOM, XSLT, XSL FO, XML Schema.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Conformance' - dericia


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Conformance

Conformance

Lynne S. Rosenthal

Mary Brady

National Institute of Standards and Technology

[email protected]

[email protected]


Outline
Outline

  • Terminology

  • Testing Methodologies

  • Benefits of Conformance Testing

  • Test Suites for XML Technologies

    • XML, DOM, XSLT,

    • XSL FO, XML Schema

Conformance

clause

Conformance Testing

Test suites

Test tools

Certification

Validation

Certificates

Branding

Procedures, reporting


Unscrambling the terminology
Unscrambling the Terminology

Conformance

Specification

Conformance Clause

Conformance Testing

test suite, test tools

Validation

process, reporting

Certification

certificate, brand


Terminology conformance
Terminology - Conformance

  • CONFORMANCE - the fulfillment of a product, process or service of specified requirements (ISO Guide 2)

    • These requirements are specified in a standard or specification as part of a conformance clause or in the body of the specification

  • CONFORMANCE CLAUSE - a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance


Terminology conformance testing
Terminology - Conformance Testing

  • CONFORMANCE TESTING – a way to verify implementations of a specification to determine whether or not deviations from the specifications exist (through the use of test suites)

    • tests capture the technical requirements within the specification

    • testing is bound in scope by the specification

    • internal structure of product-under-test not accessible (black box testing)

  • Necessary, but not sufficient, for interoperability

    Note: referred to as conformity assessment in ISO Guide 2


Testing methodology

Input

Output

Testing Methodology

  • Falsification Testing

    • find errors by means of experimentation

    • show presence of errors not their absence

    • prove non-conformance; can never prove conformance

  • Tests Based on:

    • required functionality

    • logical errors by misunderstanding requirements

    • errors from boundary conditions and divergence

    • common programming errors


Specification

Requirements

Implementation

Conformance Tests

Test programs


Terminology testing process
Terminology - Testing Process

  • VALIDATION - process necessary to perform conformance testing in accordance with a prescribed procedure and official test suite

    • ensures that testing can be repeatable and reproducible

    • ensures that conclusions are consistent with facts presented in the evaluation

  • CERTIFICATION - acknowledgement that a validation was completed and the criteria established by CIO for issuing certificates (brands) was met.


Testing ingredients
Testing Ingredients

  • Certification

    • qualified bodies to do the testing and certification

    • control board - advisory and arbiter

Validation

Process - policy and procedures for testing

  • Conformance Testing

    • Test assertions

    • Test suite

      • (test software, test scripts, test criteria)

  • Standard

    • Conformance clause


Benefits of conformance testing
Benefits of Conformance Testing

  • Increased probability that products are implemented correctly

    • Contains required functionality

    • Behaves as expected

    • Performs functions in a known manner

  • Increased likelihood of portability and interoperability

    • Portability - move software or applications among different systems

    • Interoperability – ability of 2+ systems to exchange and use information


Benefits of conformance testing cont
Benefits of Conformance Testing cont.

  • For Developers of Specifications

    • Feedback loop

  • For Software Developers

    • Help to improve quality - identify areas in which they conform or not conform

  • For Buyers

    • Increases buyer’s confidence in a product and its ability to meet their needs

    • Substantiate seller’s claims

    • Not locked into purchasing from a single vendor


Components of a test suite

Test Harness

Test Cases

Test Scenario

Test Assertions, Axioms,

Semantic Requirements

Specification

Components of a Test Suite


Oasis xml conformance
OASIS XML Conformance

  • XML Conformance Committee

    • Develop tests to improve the quality of XML processors

    • Provide a forum to discuss issues related to interoperability

    • Provide input to W3C regarding errors/ambiguities

  • Deliverables

    • Test Suite containing over 2000 tests, Second Edition

    • Associated test report

    • Built using XML/XSLT


Xml test suite
XML Test Suite

  • Types of XML Tests

    • Binary

      • Validating processors must check validity constraints.

      • Non-validating processors do not have to read external entities. Accordingly, they may not report errors which would have been detected had those entities been read.

    • Output Tests

      • Requirements on XML processors to ensure that they pass the necessary information on to applications.



Xml test suite1
XML Test Suite

  • Test Report

  • XSL Stylesheets

  • IE5, HTML renditions

Testcases.dtd

Conformance.xml

  • Test Profile

  • Test URI

  • Unique ID

  • Spec Section

  • Type

  • Entities

  • Description

  • NIST / OASIS

  • SUN Microsystems

  • James Clark

  • FUJI Xerox

  • IBM

  • XML.COM

  • Java application

  • Tests XML processors

  • Dave Brownell

Test Case DTD

XML Test Description Files

Outputs

<TESTCASES PROFILE=“SUN Microsystems XML Tests”>

<TEST URI=“sun/invalid/e101.xml” ID=“e101” SECTIONS=“3 [39]” TYPE=“invalid” ENTITIES=“none”>

Tests the Element Valid VC by including an undeclared child element.</TEST>

</TESTCASES>


Xml test suite results v1
XML Test Suite Results, v1

  • XML.COM Conformance Update – Dave Brownell

  • Validating Parsers

    • Tested JAXP 1.0, IBM Xerces/Java 1.0.3, Layered Validator (distributed with Aelfred 2), MSXML3 Technology Preview, Oracle V2.0.2.7

    • Pass Rate 82% - 99%

  • Non-validating Parsers

    • Tested all of the above plus XP 0.5

    • Pass rate 86% - 99%

  • Microsoft Update – July 2000 MSXML 3.0 Beta – Chris Lovett

    • Non-validating: DOM: 95%, SAX: 97%

    • Validating: 97%


Nist dom test suite

Test Harness – automatically runs available tests and presents user with

a color-coded table of results; gives access to spec, assertions, and source code

Test Scenario

Load appropriate XML or HTML file, exercise DOM calls

Test Assertions – Fundamental, Extended,

HTML Interfaces

DOM L1 Specification

NIST DOM Test Suite

Test Cases

200 Java – fundamental,

extended

800 ECMAScript

fundamental, extended, HTML


Dom test suite
DOM Test Suite

  • Test Results:

    • Run tests against IE5, Mozilla (some fundamental interfaces), Xerces, JAXP, and Oracle; Pass rate: 82 – 98%

    • Implementations differ greatly in the way that parsers handle white-space and entity expansion issues – differences in DOM tree

  • DOM Level 2

    • Working with W3C to identify resources

    • Modified Java tests to be consistent with SUN contribution

    • Will update NIST tests to L2 status


Xslt test suite
XSLT Test Suite

  • OASIS Technical Committee, chaired by G. Ken Holman

  • Member contributions include Lotus, NIST, Sun

  • Expands on XML Test Description file

  • Dave Marston, XSLT/XPATH Conformance Wed, 4pm, Delaware Room

  • Public Information Meeting, Wed, 8pm, Suite 8228


Xsl fo test suite
XSL FO Test Suite

  • W3C WG to test Candidate Recommendation

  • Develop test assertions for all basic formatting objects and properties

  • Develop test DTD and associated harness

  • Work with vendors to categorize available tests

  • Research translating available CSS and XSLT tests

  • Develop tests for additional spec components

  • Expected March 2001


Xml schema test suite
XML Schema Test Suite

  • Test Control

  • Test Types

  • # schemas

  • # instances

Test Generator

Schema

Tests

  • Constraints

  • Structure

  • Limits

  • Special info


Summary
Summary

  • Interesting URL’s

    • http://www.nist.gov/xml/

    • http://www.w3.org/

    • http://www.oasis-open.org/

  • Questions???


ad