SOA Data Integration - The Unsolved, Unspoken Problem
This presentation is the property of its rightful owner.
Sponsored Links
1 / 46

SOA Data Integration - The Unsolved, Unspoken Problem PowerPoint PPT Presentation


  • 87 Views
  • Uploaded on
  • Presentation posted in: General

SOA Data Integration - The Unsolved, Unspoken Problem. SOA Data Integration - The Unsolved, Unspoken Problem. David Webber SOA Architect Chair OASIS CAM TC November, 2007. [email protected] Agenda. SOA information integration needs Selection of illustrative field examples

Download Presentation

SOA Data Integration - The Unsolved, Unspoken Problem

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Soa data integration the unsolved unspoken problem

SOA Data Integration - The Unsolved, Unspoken Problem


Soa data integration the unsolved unspoken problem

SOA Data Integration - The Unsolved, Unspoken Problem

David Webber

SOA Architect

Chair OASIS CAM TC

November, 2007

[email protected]


Agenda

Agenda

  • SOA information integration needs

  • Selection of illustrative field examples

  • Solution and technology approach

  • Demonstration

  • Summary / Next Steps


Agenda1

Agenda

  • SOA information integration needs

  • Selection of illustrative field examples

  • Solution and technology approach

  • Demonstration

  • Summary / Next Steps


Ceo enterprise strategic vision

CEO - Enterprise Strategic Vision

Modernize

& Integrate

Systems

Improve

Program

Integrity

Reduce

Costs

Improve

Human

Capital

Management

Improve

Products &

Services

- Reconcile current business demands, economics and future business strategies with present state of legacy systems, technology and human resource skills

-Strengthen financial management and internal controls

-Provide effective oversight

-Ensure the accuracy of all data

-Implement an integrated set of information systems

- Enable cost reduction and re-engineered business processes

- Establish metrics to measure performance and productivity trends across the enterprise over time

-Ensure enterprise has the right people in the right jobs with the right skill sets

-Create and foster an environment for collaborative problem solving and decision making

-Listen to the customer to understand and prioritize needs and/or desired products and solutions

-Inform business solutions by incorporating industry best practices


Some soa claims and justifications

Some SOA claims and justifications

Service Oriented Architecture means:

  • Reusable components exposed through services shared with business partners and enterprise systems enabling low-cost, low-impact incremental and organic adaptation


Soa characteristics

SOA Characteristics

Open

Standards

Modular

Approach

Supports

Strategic

Initiatives

and Goals

Swappable

Agile

Components

Questions?

- SOA implements the functionality of the system using formal open standards for information exchange, transport and delivery.

- SOA is an attempt to modularize large complex systems in such a way that they are composed of independent software components that offer services to one another through well-defined interfaces.

  • How can we swap things if the information interfaces are not clearly defined and easily verifiable?

  • How can we leverage inconsistent legacy interfaces?

- Reconcile current business demands, economics and future business strategies with present state of legacy systems, technology and human resource skills

- This supports the notion that any of the components could be ‘swapped’ for a better one, when it becomes available.

7


What are soa services

What are SOA Services?

  • The ability to orchestrate service logic with business processes

  • Not just static point information service sinks or sources

  • Ability to associate behavior with information exchanges

    • Support for context and role

    • Support for business process actions and signals

    • Support for access profiles and partner agreements

    • Support for dynamic structure queries and responses

  • Use of open public standards to ensure predictable access to services

  • Independent of platform deployment and legacy systems constraints


What makes something agile

What makes something Agile?

  • Re-usable methods that can be applied to many areas

    • Based on open standards and approach; not proprietary

  • Context and role driven and aware

    • This allows tailoring to specific profiles and use pattern templates dynamically

  • Self-adaptive

    • When requirements change can be adjusted on-the-fly in real time

  • Fault tolerant and not brittle

    • Ability to ignore non-critical interchange items and especially not to fail for trivial reasons or slight version nuances

    • Leveraging XML capabilities to make self-describing transactions possible rather than static fixed legacy exchanges

  • Able to support new uses without extensive re-programming

    • Usage patterns set via external configuration allowing broad but controlled uses


How to support discovery

How to Support Discovery?

  • As an organization develops extended numbers of services the need to catalogue and share these collaborative increases.

  • Services need to:

    • Describe their means of connection and interfacing

    • Provide a reference to the domain of use

    • Be associated with a catalogue and classification system

  • The most powerful discovery mechanism is not simple text search but contextandprofileagreement parameters linked to templates

  • Associate with a domain vocabulary to facilitate shared meaning and core components

  • Align to interoperability of information exchanges and agile concepts

  • Bottom line – answer the questions:

    • Can I use this? (context)

    • How can I set it up? (templates)

    • Will it work & connect with my existing services (behaviors)?


Defining information services

Defining Information Services

1

XML

Share Results

XML

Create Samples

Publish

Structure + Vocabulary

Samples

4

2

Detail Use Rules

Structure

Rules

Context

Vocabulary

Templates

Analyst

Rules

Develop

Rules + Context

Templates

html

Enabling

Agile

Information

Exchanges

Test Rules on Samples

3

Rules

Editor

Results

Verify

Verify Template Outcomes

html


Partner conformance

Partner Conformance

Publish

Structure

Rules

Context

Templates

Templates

5

Validate

Partner

Creates

Results

XML

4

Test

Report

Test and

Certification

XML

6

html

Pass / Fail


Facilitation needs

Facilitation Needs

  • For business communities:

    • Ability to create sharable templates for communities of practice that need consistent XML transaction handling definitions that are open and public.

  • For business data analysts:

    • Printable rule documentation support and features.

    • Example domain templates within communities of practice

    • Genericode codelists implementation.

  • For programmers:

    • XML content manipulation support and rules ( xslt, XPath …)

    • Web services and SOA support


Select your technology set

Select your technology set

  • Many technologies are available, including application servers, distributed objects, and integration servers. Agile use of XML a prerequisite.

  • The choice of technology will likely be a mix of components that together meet your SOA needs.

    • Avoid vendor lock-in

    • Prefer Open Standards and Open Source solutions

    • Ensure proper use of layers to separate and manage complexity

    • Select simple components that integrate easily


Basic wsdl web service

Basic WSDL Web Service

Packaging / Encoding

Requirements

XML

Query / Response

Validation

UML diagrams

XSD

4

5

3

Delivery

Data Services

Process

Application

Logic

Security

Transport

1

2

WSDL

SOAP / http / https

Internet Infrastructure


Extended data services

Extended Data Services

Business Model

Service

Agreement

State & Context

Addressing / Envelope

Context

/ Roles

Transactions

Vocabulary / Semantics

Coordination

Business

Processes

Packaging / Encoding

Push / Pull

XML / edi

Query / Response

Validation / Assembly

Description

Mapping / Transform

4

5

3

Delivery

XSD, CAM, Schematron

Data Services

Msg Exchange

Profile

Process

Security

Transport

1

2

WSDL

SOAP / http / https

Internet Infrastructure


Soa orchestration

SOA Orchestration

Discovery

Search / Classification

Ownership Profiles

Business Model

Requests to Consume

Service

Agreement

Service Definitions

Management / Versioning

Context

/ Roles

State & Context

Vocabulary / Semantics

6

Addressing / Envelope

Registry

Vocabulary / Semantics

Business

Processes

Transactions

Packaging / Encoding

Coordination

XML / edi

Description

Push / Pull

Validation / Assembly

Mapping / Transform

4

5

3

Delivery

Msg Exchange

Profile

XSD, CAM, Schematron

Data Services

Process

Security

Transport

1

Other protocols

Security Credentials

B

2

B

REST

W

S

I

2

Quality of Service

Other services

WSDL

Reliable Messaging

SOAP / http / https

Internet Infrastructure


Agenda2

Agenda

  • SOA information integration needs

  • Selection of illustrative field examples

  • Solution and technology approach

  • Demonstration

  • Summary / Next Steps


Domain examples

Domain Examples

UBL

schema

and

CCTS

Fannie Mae

EDI-esque

XML

OASIS

EML

Grants.gov

form

based

schema

DOJ

NIEM/LEX

PESC

Dictionary

and

schema

  • -Transactions replacing EDI

  • Overloaded reuse of structure components

  • Financial reporting data

  • Financial transaction data

  • Extended use of code sets to label transaction content / purpose

-Developed by BAH and NG for form-based application submissions

-Translate previous paper form into multiple schema sections

-Massive schema with extended namespaces

  • Universal Business Language (UBL)

  • OASIS standard

  • Uses UML models and CCTS approach

  • Derived from xCBL and simplEDI approach

  • Supply chain schema with joint initiative EU / Asia / US

-Developed by GTRI for DOJ community

-Vocabulary based

-LEXS schema built using NIEM vocabulary

-Law enforcement and court applications

  • -Developed for Education Department

  • -form-style XML

  • -Older XSD techniques for simple flat schema model

  • No re-use of common structures

  • Student loans and transcripts

  • -Set of functional transactions and configuration templates for managing and processing elections

  • Common set of components and vocabulary

  • Widely differing use patterns and items by country localization


Challenges for users and implementers

Challenges for Users and Implementers

UBL

schema

and

CCTS

Fannie Mae

EDI-esque

XML

OASIS

EML

Grants.gov

form

based

schema

DOJ

NIEM/LEX

PESC

Dictionary

and

schema

  • -Simple structures with need to show transaction use patterns

  • -Extensive use of codes

  • -Calculations and numbers hard to validate using schema alone

  • -Extensive cross-field relations

-Extended form field edits and code lists

-Large structure and verbose syntax

-Excessive use of namespaces

-Extensive cross-field and form relations

-Related content edits (PDF documents)

  • Complex schemas that meet extended use cases

  • Less obvious how to create sparse subset for typical everyday uses (in State / in Country)

  • Verbose syntax and structures from CCTS models

-Complex schema with use model that does not necessarily map to domain information stores and search mechanisms

-Platform transport formats determine transaction packaging

- Limited interoperability of search methods

-Implementers have created baseline samples for typical student use cases

-Backend applications developed by solution providers

-Interoperability verified between providers solutions

-Documenting widely differing use patterns and items by country localization

-Understanding how to use flexible structure constructs for their own applications

-Verification and certification of exact usage and computations


Solution metrics

Solution Metrics

Can I create a standard simple open format to describe my message structures and data content rules?

Can my partners validate their transactions in test BEFORE they send them?

How do people know what I will send them?

I want something that’s simple and standards based – leverages existing XML components

Can I generate HTML documentation that is readable by business analysts?


Why not use xsd

Why not use XSD?

Today’s XML schemas have complex structures with no context awareness + no cross field association rules + no dynamic lookups

The XSD provides a model of ALL possible structure instances – not the particular instance

Excessive use of namespaces make for fragile XML transaction handling

Generating valid sparse transaction layout is tough

Documentation diagrams hard to read

How to create simple re-usable templates?


Xsd pesc gpa model example

XSD – PESC “GPA” model example

EVERYTHING

is optional! So

what do I

REALLY send

to you?


Conceptual view of pesc transactions

Conceptual View of PESC transactions

1

XML

Transaction

Share Details

Publish

Common Details

4

Address

Course

Rules

Structure

Rules

Context

2

Templates

Contact

3

Context

Student

Use Rules

Reports

Sponsor

html

Content Rules

Loan

html

Lookup Values

Transcript

Versioning


What about these needs

What about these needs?

  • Versioning

  • Content consistency

  • Use rules

  • Codelists

  • Associations (what uses which?)

  • Guidelines

  • Providing local contextual validation services


Versioning challenges

Versioning Challenges

If the schema version changes – how to ensure it does not break our in place validations?

How to rapidly adapt to rule changes in a production environment?

How to develop user context driven deep version control and re-use of sub-components?

Enhance and automate Test release cycle by improving transparency for bug fix process and expose change deltas to speed testing process? Support for regression testing?


Agenda3

Agenda

  • SOA information integration needs

  • Selection of illustrative field examples

  • Solution and technology approach

  • Demonstration

  • Summary / Next Steps


History and status of cam work

History and Status of CAM work

OASIS technical committee

Five years of combined development in UN/CEFACT and OASIS

OASIS v1.1 public standard and specification

jCAM open source implementation in Java

Creating simple XML-scripted open standard mechanisms for XML transaction assembly and processing

Re-use: leverages XPath and xslt, and extends schematron approach

Developing templates for common industry formats


Cam process architecture

CAM Process Architecture

XML Parser

/ DOM

CAM

XPath

handler

XML-aware

Built-in

Functions

EXTENSIONS

Rule Engine

Post-

Processing

/ Errors

SQL

persistence

Terms

Registry


Jcam functional components

jCAM Functional Components

xerces

XML Parser

/ DOM

jaxen

CAM

XPath

handler

cam

XML-aware

Built-in

Functions

EXTENSIONS

Rule Engine

Post-

Processing

/ Errors

SQL

persistence

Terms

Registry

X

planned using AJAX / ebXML

e.g. JRules,

Others…

Saxon - xslt

XML Data Mapping


Deployment options

Deployment Options

Web services

B2B

Templates

Templates

Structure

Rules

Context

Request

1

XML

2

Server

jCAM

engine

jCAM

engine

Validate

XML

Message

System

2

Java

API

Response

html

Java

API

Process

XML

3

1

Receive

Standalone

Template

XML

Process

EDITOR

html

XML

3

Report


How does cam work

How does CAM work?

CAM uses WYSIWYG approach to XML

Starting with your XML sample – creates structure template from that + default data content model

Next – add your structure use rules – optional / repeatable, date fields, allowed values, lookups

Then make context business rules – cross field use rules, exclude, include, variables

Save template – run against samples

Eclipse editor tool makes this all easy to do!

Deploy to production using jCAM processor


Eclipse cam editor

Eclipse CAM Editor

Available structures

1

Rule Details

3

4

Validation

Process

2

Structure Rule Viewer

5

Results Viewer


Agenda4

Agenda

  • SOA information integration needs

  • Selection of illustrative field examples

  • Solution and technology approach

  • Demonstration

  • Summary / Next Steps


Using jcam

Using jCAM :

Start with sample XML

Use Eclipse template editor

Load XML, generate CAM

Enhance base template

Test, refine and deploy

Generate documentation

Deliver business solution

1

Simple

XML instance

2

Build Simple

Template

3

Extending

Template

4

Verify Results

5

Document

Rule Details


Illustrative mismo requirements

Illustrative MISMO requirements

  • When Loan Amount > 700,000

    • Require LTV <= 90

  • When Loan Amount > 1,000,000

    • Require LTV <= 80

    • AND CLTV <= 90

  • When Documentation Type = Full AND Loan Purpose = Purchase AND Occupancy = Primary Residence

    • Require LTV <= 95

    • AND CLTV <= 95

  • Require Property State = CA


Cam rules syntax

CAM rules syntax

<as:BusinessUseContext>

<as:Rules>

<as:default>

<as:context>

<as:constraint condition="//Loan/Amount > 700000 and( //Loan/LTV > 90)" action="restrictValues(//Loan/LTV, ‘LTV must be less than or equal 90 for 700,000 loans') "/>

<as:constraint action="restrictValues(//Property/State, 'CA')"/>

</as:context>

</as:default>

</as:Rules>

</as:BusinessUseContext>


Summary of cam template features

Summary of CAM template features

Ability to use <as:include> to modularize complex content

Use of XPath references enables agile rule handling

Full contextual support including variables, rules and context sections

Code lists can be externally built and then referenced

Extensions section provides customization options

Integration with xslt provides extended results handling and error reporting capabilities

Template representation model and XML enables extended post-processing and documentation options


Cam functions summary

CAM Functions Summary

(FAQ – can I extend the functions?)

<as:BusinessUseContext>

<as:Rules>

<as:default>

<as:context>

<as:constraint action="makeRepeatable(//Items/Item)"/>

<as:constraint action="makeOptional(//Item/comment)"/>

<as:constraint action="setLength(//shipTo/state,2)"/>

<as:constraint action="setDateMask(//PurchaseOrder/shipDate,YYYY-MM-DD)"/>

<as:constraint action="makeOptional(//PurchaseOrder/comment)"/>

<as:constraint action="restrictValues([email protected],'US'| 'CA'| 'MX', 'US')"/>

<as:constraint action="setDateMask([email protected],YYYY-MM-DD)"/>

<as:constraint action="setNumberMask([email protected],###-###)"/>

<as:constraint action="setNumberMask(//Item/quantity,###)"/>

<as:constraint action="setNumberMask(//Item/price,####.##)"/>

<as:constraint condition="[email protected] = 123-678“

action="restrictValues(//shipTo/state,'WA')"><as:annotation>

<as:documentation

type="documentation">Can only ship item 123-678 to Washington State

</as:documentation></as:annotation>

</as:constraint>

<as:constraint condition="$QuickBooks = true“

action="excludeElement(//Item/comment)" />

</as:context>

</as:default> </as:Rules> </as:BusinessUseContext>

Example of Rules Use


Interactive documentation idoc

Interactive Documentation (iDoc)

CAM

Template

XSLT

iDoc

wiki

HTML


Agenda5

Agenda

  • SOA information integration needs

  • Selection of illustrative field examples

  • Solution and technology approach

  • Demonstration

  • Summary / Next Steps


Value proposition

Value Proposition

Making XML transaction handling simpler and predictable

Extends and clarifies your existing XSD schema structures

Quick and easy rule building from sample XML transaction

Enabling more robust fault tolerant processing + versioning

Providing open sharable templates and documentation

Re-use easier through support for includable components

Ability to integrate to business processes and context

Open source, open public standard toolkit – editor + engine


What s next call to action

What’s Next / Call to Action

Develop template sets for your business domain

Integrate into your messaging exchanges

OrionSMG supports jCAM

Web service test-bed

Publish to registry to facilitate adoption

Create document templates to generate registry content / guidelines

Use Wiki / SourceForge to facilitate sharing


Questions

Questions?


Soa data integration the unsolved unspoken problem

www.jcam.org.ukcamprocessor.sourceforge.netwww.drools.org (JRules)www.oasis-open.org/committees/camdocs.oasis-open.org/cam

Resources:


Soa data integration the unsolved unspoken problem

A special mention for our contributors to the CAM and jCAM work:UK- Martin Roberts and team from BTplcUS- Sidhartha Nagolu from AC-Tech / NIH

Credits:


  • Login