1 / 46

Corporate Data Architecture in a Federated World

Corporate Data Architecture in a Federated World. Presented by Deborah Henderson, INERGI LP to IRMAC Business Intelligence & Data Warehouse SIG. May 23 2002. You Know You have a Problem When…... You Have a 'Dark Matter Schema'. We know that the data must be in the

lucindat
Download Presentation

Corporate Data Architecture in a Federated World

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Corporate Data Architecture in a Federated World Presented by Deborah Henderson, INERGI LP to IRMAC Business Intelligence & Data Warehouse SIG May 23 2002

  2. You Know You have a Problem When…... You Have a 'Dark Matter Schema' We know that the data must be in the warehouse somewhere, but we can’t find it.

  3. The 'Dark Matter Schema' Two subsets, according to differing theories: The WIMPs schema: we are just too overwhelmed by user requests to track down where in the data that particular element resides. The MACHOs schema: we are too busy being the all knowing DW experts to track the elements down.

  4. Agenda • Inergi and Architecture • Corporate Data Architecture • What supports this : the IT Business Model and Architectural Compliance • Data Architecture Process and Procedure • Models & modelling • On the Horizon

  5. INERGI LP • Subsidiary of Cap Gemini Ernst & Young Canada Inc. along with New Horizons Solutions, another energy sector affiliate reporting into CGE&Y • Created March 1 2002 • Multi-year deal for sustainment of Hydro One IT systems • Supply, Finance, Pay, IT, Call Centre, Customerbilling for Hydro One • We are IT and Business process outsourcing specialists for the energy sector with many years experience • We are open for business!

  6. Corporate Data Architecture Models Domains Policies Standards

  7. Enterprise IT Architecture • Why is it Important? • Reduces cost of operations, through reuse of standard pieces of technology, application and data, network Example: • Financials • disk farm participant (technology) • official one source for ledger data (data) • Java reporting environment (application) • using network standards

  8. Corporate Architecture

  9. Data Architecture Idea Data Architecture Set of principles that defines ‘organization-wide data resource of well-described, properly structured, high-quality data that are properly documented’. (Brackett, 1994) Metadata Architecture Set of principles that defines and describes the data resources in an organization. Physical DBMS Architecture Architecture component that defines physical data components.

  10. Data Architecture Constructs

  11. Data Architecture Should be Principles Based • Try to leverage the DW Project : Data, metadata, technical, ETL (extract transform and load) , EUT (end use tools), Physical architectures • Develop architecture changes and additions to the overall Enterprise Architecture • High re-use of data and • processes across the enterprise • for next initiatives

  12. A Data Architecture is Principles Based EXAMPLES: 1. System of record will be established for all data 2. Corporate definitions of data will be resolved and maintained 3. Data will reside on database servers not on application servers or mail servers

  13. Enterprise Data Warehouse as Driver for Data Architecture • First data architecture effort often constrained by the EDW project • Think enterprise scalable: Hardware, Software, Processes, Centre of Excellence in Data, Corporate Data Architecture compliance, stewardship & vitality process

  14. Data Architecture Implemented: Tools & Expertise • Modelling & Metadata storage • Data store • Data mining • Statistics • Business Reporting • Environment modelling • keep the number of tools to a minimum reduces • lifetime ownership costs to the Company • establish the role of Product Specialist • for all tools

  15. Sample DW Data Architecture Local Data OLAP & Details External Data & History ODS source Datamart Local model 10% 20% 30% 40% OLAP & Analysis Historical & Benchmark purchased data Operational Report server

  16. Local model OLAP & Analysis External data, history ODS Sample DW Data Architecture There are models for each component Sometimes!! the models are linked/related

  17. Metadata Architecture in Most Companies Today modeler OLTP Applications RDBMS Modelling CASE ETL BI End Use Tool Interfaces are usually proprietary End user = repository

  18. Metadata Arch The Object Management Group (OMG) was established in 1989 and is the world's largest software consortium with a membership of over 700 vendors, developers, and end users. In June 2000, OMG released an XML-based metadata standard. OMG showcased XML metadata interchange in March 2001 at the DAMAI conference in Anaheim

  19. Physical DB Architecture • ORACLE and DB2 outlook • Impact of DW features • partitions, instances and machines • Referential integrity • placement and enforcement • Multi-dimensional cubes • Metadata ‘repository’ through hooks or??

  20. Business Intelligence: The Delivery Maturity Model E-business portal Document- centric portal DW /BI support portal Infrastructure Development

  21. Processes and Procedures

  22. IT Compliance Business Model For every IT project • Technical architecture • Data architecture • Map to Business model pre-requisite is a Policy on data sharing

  23. Data Architecture Compliance Process Modelling principles and procedures Metadata principles Dimensional Logical Dimensional Policies Conceptual Implemented DB Logical Dimensional Data architecture principles Naming Conventions Dimensional Database design and implementation guidelines

  24. Data Architecture Compliance Process • Document processes that drive • procedures, standards* • Data Models are necessary deliverable • of project, should be noted in Charter as • a deliverable • Architectural compliance and • operational readiness gate • Vitality process - keeping current

  25. *Additional Documentation to Support the Corporate Data Architecture • Metadata repository (or facsimile!) • Business definitions • Standard naming conventions • Standard abbreviating procedures • Standard domain structuring • Standard translation schemes • Conformed dimensions and facts - EDW • Stewardship patterns • CDA roles and responsibilities

  26. Governance and Using the Architecture Governance framework and repository maintenance Corporate Data Architect, Lead Data Modeller and Corporate Data Modeller SME, Lead Data Modeller and Data Modeller Using the Architecture in a project

  27. Data Stewardship: the Business-side Responsibility • Data will be controlled and managed throughout its life cycle as a resource, in the same manner as any other asset (capital, material, and people). • Access to data will be facilitated, and/or controlled and limited, as required to provide the best performance at the least cost for all users while meeting functional and technological , regulatory and legal requirements.

  28. Data Stewardship: the Business side responsibility cont’d • Data will be shared except where exempted by Corporate Security Policy. • Data will be standardized to avoid duplication and facilitate integration.

  29. Building Models • 1. Assess subject areas involved in a project and publish for reuse : • Conceptual model • bubbles • Subject area models • where complete • Documentation • standards for models

  30. REFER TO PROCESS DOCUMENTATION ! Building Models 2. Build on these models and submit for review and approval 3. Develop conformed data objects where required (as you go) 3. Add new models to the Model Repository

  31. Related Data Models ARE Your Quality Control • Conceptual and Enterprise Data Models maintained by IT Architecture • Logical Models assists in understanding, official definitions, (OLTP physical = logical) • Enterprise Data Warehouse Model a dimensional model that gets implemented in Oracle high-performance ‘read-only’ model • Cube Designs - Problem centric Dimensional models implemented in OLAP Cubes • Source Data Models (OLTP) - informational for BI, source for ODS

  32. Models

  33. More Physics of Schemas :-) 'Black Hole Schema' : Systems where the query never returns 'Pulsar Schema' : Only returns results every few queries or so 'Milky Way Schema' : A central warehouse with many dozens of offsprings that no one can keep track of 'SuperStrings Schema' : Many measures, all built on top of each other, relating to each other and that give the same result

  34. Conceptual Data Model • High-level model • Depiction of major Functional Areas in the Company • Each Functional Area defined

  35. Enterprise Data Model • Limited number of high-level data sets (Subject Areas) • Global relationship cardinalities are shown • Data Sets fully defined • Definition is formalized at the Corporate level as official • Data Stewardship is established for each Subject Area

  36. Logical Data Models • Logical Data Model developed per Project basis • LDM fully synchronized using the Corporate Data Architecture Principles • Objects fully defined and attributed • Re-usable domains implemented • Re-usable rules identified, documented and implemented consistently

  37. DW Physical Data Models • Implemented • Limited use of RI (load only) to keep the data integrity • Business rules implemented through ETL procedures • Model in-sync with the database • CASE tool used for model/database synchronization • Colors extensively used for better readability

  38. Time for…...More Physics of Schemas :-) 'Binary System Schema' : Two datamarts that do the same thing and try to suck each other into themselves ’Chance Theory Schema' : The results are always uncertain and questionable as it changes every time you run the report ‘Event Horizon Schema' : Has many dimensions, but you if you ask for more than a certain number at a time, it converts to a Black Hole Schema

  39. 'Big Bang Schema' : A data warehouse that we miraculously brought into existence and the user does not know why or how or how it’s useful 'Tachyon Schema' : Miraculously fast and becomes faster as you add data. But cannot be implemented as it is theoretical. Most demos fall into this space.

  40. OLTP Enterprise Data Model OLAP Enterprise Data Warehouse Model 1 Subject area EXTRACTS 2 Logical Data models 1 2 3 n DATA MARTS

  41. Enterprise Data Model Enterprise Data Warehouse Model Corporate Data Architect, Lead Data Modeler and Corporate Data Modeler Extract Extract Transition LDM 1 LDM2 LDM 3 LDM n SME, Lead Data Modeler and Data Modeler

  42. For DW Conformed Dimensions & Facts • Supports iterative/parallel build and aligns • with Kimball’s bus structure • BUT • Can get it wrong • Can loose control • Exponential complexity

  43. On the Horizon • Impact of XML • Impact of Taxonomies in Business • Corporate Reporting Strategies • Overarching Mobile Data Strategies • ODS for all ERP • Information Architecture • BAM - Business Activity Monitoring* • Network Appliances • Synergies with Application Architecture *Gartner April 2002

  44. CDA Together with Application Architecture Sample Principles DATA APPL’N • Business Processes should be implemented in the application not the database • Logical workflow and data flow must align • Applications must have owners just like data • We must be able to identify official source (aka system of record)

  45. Corporate Data Architecture That is Process Driven • Policies • Architecture & Standards • Addresses OLTP and OLAP • “Federated” world • Compliance • Vitality • Stewardship • Can be applied to all new • challenges on the horizon • With all the pieces….This could work!!

  46. ‘Enron Schema’ : Shows positive numbers where we practically expect negative values, and Anderson can prove that it is correct.

More Related