1 / 41

Key Patterns Converge: Data Integration and Data Virtualization for SOA (CON8514)

kerem
Download Presentation

Key Patterns Converge: Data Integration and Data Virtualization for SOA (CON8514)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

  2. Key Patterns Converge: Data Integration and Data Virtualization for SOA (CON8514) Alex Kotopoulis and Ayush GaneriwalData Integration Product Management

  3. Agenda • Oracle Data Integration Solutions Overview • Combining SOA and Oracle Data Integrator • Data Virtualization with Oracle Data Services • Q & A

  4. Oracle Data IntegrationOverview

  5. Oracle Data Integration Solutions • Thousands of customers including Fortune 500 companies, representing Financial Services, Communications, Healthcare, Public Sector, Retail, Utilities industries • 4 of top 5 largest commercial banks • 3 of top 3 busiest ATM networks • 5 of top 5 financial data services companies • 5 of top 5 telecommunications providers • 3 of top 5 largest food & drug stores

  6. Real-time replication; optimized for Database 12c and Oracle Exadata End-to-end integrated with simplified deployment Unified tooling for both structured data sources and Hadoop / NoSQL Flexible deployment on-premise or in the Cloud for heterogeneous systems Expanded support for 3rd party systems and Oracle Applications in real-time data integration and continuous availability solutions Oracle Data Integration 12c Delivering Real-Time Integration for Big Data and Cloud Cloud Oracle Data Integrator Apps Oracle GoldenGate Oracle Enterprise Data Quality Database Oracle Data Service Integrator Big Data

  7. Oracle Data IntegratorBulk Data Processing and Data Transformation Oracle Data Integrator • Certified for leading technologies to deliver fast time to value • High-performance, low cost of ownership E-LT architecture • Lightweight deployment • Flexible, easy to enrich functionality Cloud High Performance E-LT Apps Declarative Design Database Extensible Knowledge Modules CEP Data Services Big Data Capable of Integrating Structured & Unstructured Data

  8. Oracle GoldenGate Real-time Data Integration Oracle GoldenGate • High-performance, low-impact real-time data integration and replication • Timely data for improved business insight • Continuous availability for 24/7 operation Cloud Log-based Change Data Capture and Delivery Apps Active-Active Replication Heterogeneous Source and Targets Database Reliability and Transaction Integrity Big Data Java Integration, Real-time Events

  9. Oracle Enterprise Data Quality Data Quality for Customer and Product Data Oracle Enterprise Data Quality • Improves data accuracy, usability and ‘fitness for purpose • Unified interface for ease-of-use, lower TCO • Depth of capability in multiple data domains reduces project risk Cloud Profile, Explore, Audit Apps Parse, Standardize, Cleanse Match, Merge, Enrich Database Governance Big Data Case Management

  10. Oracle Data Services Integrator Data Federation and Virtualization Oracle Data Services Integrator • Easily develop data services and ad-hoc queries without infrastructure changes • Open and standards based—SOA, Web Services, Java, Service Data Object Cloud Federated Data Services Apps Data Virtualization Integrates with Oracle SOA Database Bidirectional Reads & Writes Big Data Integrates with Oracle Business Analytics

  11. Oracle Data IntegrationCombining SOA and Oracle Data Integrator

  12. Data Integration Services Client Applications: BPEL, OSB, Custom WSDL WSDL OGG Data Replication Services ODI ODSI Data Virtualization Services OEDQ Data Quality Services WSDL WSDL ELT Services Data Access Services Agent Replicate Data in Real Time Validate and Cleanse Data Access Transformed Data Move & Transform Bulk Data Virtualize Access to distributed sources Data Warehouse Other DB Oracle Web Service Application File JMS

  13. Trail Trail Real-time Data Warehousing with ODI, OGG and OEDQ ODI Agent OEDQ Validation & Cleansing Kick off OEDQ process Bulk load snapshot of Production DB into staging Waits for availability of incremental data Production Schema Initial Load Transform & Load Transform and load data into Data Warehouse DIM DIM EMP DEPT FACT EMP DEPT DIM DIM DBMS Logs ODI generates and kicks of OGG process to replicate incremental changes TCP/IP DW Schema Staging Schema OGG OGG Pump Extract Replicat Transform

  14. 15 ProductSuppliers ODI for Transforming Large Payloads Oracle BPEL PM or ESB 2 1 Reference Reference 3 8 7 Reference ODI Invoke SharedMetadataRepository Data Integrator Order DB 5 4 6 Shared Container/JVM Shared SOA Server ODI loads payload ODI transforms payload ODI sends payload wherever instructed ODI notifies BPEL/ESB that job is complete Core BPEL/ESB processing completes Large File arrives, detected by BPEL File Adapter Execution Starts (BPEL/ ESB) – and a step for transforming a large document payload occurs Pass XML payload, by reference, to ODI <yxz> <zyx> Transform

  15. 16 ODI Loads a BI Warehouse (with Error Hospital Human Workflow) c Human Workflow 1 Oracle BPEL PM or ESB Event 2 7 • An event occurs which prompts a data load • Execution starts within BPEL/ ESB process • Instruction to start ODI • ODI extracts & loads data • Some errors w/in batch • Notification to BPEL Human Workflow engine • Repair & approvals process • Repairs are committed • ODI looks for repaired data when “Recycle” option is on • ODI transforms the data • ODI confirms job complete • BPEL/ESB process continues any additional process execution steps d Instruction 3 6 Confirmation b ODI Invoke e SharedMetadataRepository Data Integrator Shared Container/JVM Shared SOA Server a 5 4 DW Server DB Server BusinessIntelligence Any DB Records B Records A Transform Transform

  16. 17 ODI for SOA-based ERP Integration • A business process for Migration (initial bulk data load) or Replication (ongoing synch) is invoked • BPEL/ESB sends instruction to ODI • ODI performs E-LT • ODI creates Unique ID for new ERP data objects • ODI updates Unique ID for existing objects • ODI confirms job • BPEL/ESB begins processing ERP business transactions • BPEL/ESB leverage same Unique IDs for canonical XML • All SOA and ETL jobs keep business data aligned with Unique IDs Messaging and bulk data may leverage the same unique object IDs – thereby ensuringuniform ERP data objects(eg: PK123 = Pkxyz) ERPApp 1 ERPApp 2 Oracle BPEL PM or ESB XML XML UI UI event 5 2 Instruction 1 App App Confirmation SharedCanonicalID XREFLookups 4 ODI Invoke Data Integrator c 3 Shared Container/JVM DDL DDL Data Data b Transform a

  17. 18 WLS3 WLS2 WLS1 ODI Agent ODI Agent ODI Agent ODI High AvailabilityPerformance and Reliability through Weblogic and Coherence Clients WebLogic Cluster ODI Repositories on Oracle RAC ODI SDK APIs ODI Master Repository ODI Public Web Services RepositoryExplorer Proxy / Load Balancer ODI Work Repository Coherence ODI Scheduler Sync ODI Studio ODI Work Repository ODI Work Repository Designer Operator Topology Security

  18. XML File processing XML Schema customer <customer id=‘4433’ name=‘Paul’> <order id=‘1’> <item id=‘101’ count=‘5’/> <item id=‘102’ count=‘3’/> <item id=‘103’ count=‘3’/> </order> <order id=‘2’> <item id=‘201’ count=‘5’/> <item id=‘202’ count=‘3’/> </order> </customer order ODI XML JDBC Driver item

  19. Complex File processing Native Format Builder UI Staging Tables customer order nXSD Schema C|4433|Paul O|1 I|101|5 I|102|3 I|103|3 O|2 I|201|5 I|202|3 I|203|3 item ODI Complex File JDBC Driver S_OPTY

  20. Oracle Data IntegrationData Virtualization with Oracle Services Integrator

  21. Consolidation: Move data from distributed sources to central hub; consuming applications query hub database Data Virtualization / Federation: Query and update distributed sources in-place and provide combined result Hybrid Approach: Federation and Consolidation are combined based on best approach for individual use cases Different Data Access Techniques Consumer Consumer Consumer ODSI Consolidated Hub Consolidated Hub ODSI ODI/OGG ODI/OGG Source A Source B Source C SourceB Source A Source A Source B Data Virtualization only Hybrid Consolidation + Data Virtualization Consolidation only

  22. When to Use Which Integration Technology

  23. Data Virtualization Use Cases • Data Services Layer • Provide a clean data abstraction for SOA and application developers • Shield consumers from changes in data topology • Augmenting Business Intelligence • Extend access to sources outside the data warehouse • Rapid Prototyping of Data Integration • Quick creation of an interface to integrated data without building a consolidated store • Cloud Data Integration • Combine onsite and offsite data resources in a unified interface

  24. Data Virtualization: Oracle Data Service IntegratorProduct Architecture Java Web Svc .NET JDBC OSB Fast Track Transport Monitoring/Auditing ODSI Studio IDE Security OER Admin Console Metadata ODSI SDK Cache Query Engine Fast Track Transport OSB Database Web Svc XML/Flat File API / Apps

  25. ODSI Core Strengths • Native SOA Data Integration • Native XML type system allows processing and passing of hierarchical & SOA data. • Based on open standards (XML, WS, JDBC/SQL, XQuery) • Tight integration in Oracle SOA: Fast-path with OSB, Governance with OER • Normalization and Reuse through Data Services • Multi-layer approach to normalizing and transforming source data • Highly Scalable, Industrial-Strength Performance • Linear scalability through WLS Clusters • Performance through caching, streaming and query optimization • Fine-Grained Security and Auditing • Row and column-based data security • Comprehensive Update Capabilities • Automated CRUD operations on data services • Distributed XA transactions through WLS Transaction Manager

  26. Data Service Design & ModelingLayered Approach

  27. ODSI Development Environment

  28. ODSI Logical Data Service

  29. ODSI Parameterized Data Service

  30. ODSI Query Plan

  31. ODSI Test Execution

  32. ODSI Update Mapping

  33. ODSI Administration Console

  34. Oracle Data IntegrationQ&A

  35. Questions & Answers

  36. Join the Data Integration Community LinkedIn“Oracle Data Integration” Oracle’s Data Integration blogblogs.oracle.com/dataintegration Oracle Data Integrator Bookshttp://www.packtpub.com/oracle-data-integrator-11g-getting-started/bookhttp://www.packtpub.com/oracle-data-integrator-11g-cookbook/book Twittertwitter.com/ORCLGoldenGate Facebookfacebook.com/OracleDataIntegration Data Integration YouTube channelyoutube.com/oraclegoldengate Oracle.com/goto/dataintegration

  37. Oracle OpenWorld – Key DIS Sessions Oracle Data Integrator

  38. Graphic Section Divider

More Related