data integration to data governance l.
Skip this Video
Loading SlideShow in 5 Seconds..
Data Integration to Data Governance PowerPoint Presentation
Download Presentation
Data Integration to Data Governance

Loading in 2 Seconds...

play fullscreen
1 / 23

Data Integration to Data Governance - PowerPoint PPT Presentation

  • Uploaded on

data relationship management. Data Integration to Data Governance. Data In the News:.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Data Integration to Data Governance

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
data in the news
Data In the News:

Data slipupsRick Whiting , 10-May-2006Inaccurate business data lead to botched marketing campaigns, failed CRM projects--and angry customers. A home valued at US$121,900 somehow wound up recorded in Porter County's computer system as being worth a whopping US$400 million. Naturally, the figure ended up on documents used to calculate tax rates. By the time the blunder was uncovered in February, the damage was done.

market forces affecting the use of data
Market Forces Affecting the Use of Data

Privacy Regulations



Straight-through processing, customer service, Consumer confidence

Data Inaccuracies, Over-billing





SEC/NAD rule,

SARBOX, legal

liability, Mergers

Business Governance

credit card company where is the sensitive data
Credit Card Company: Where is the Sensitive Data?
  • Business Problem:
  • Risk of a security breach exposes potential regulatory fines, negative PR and customer backlash
  • Proposed Solution:
  • Identify sensitive data flows in structured databases so critical data can be consolidated and properly secured
  • Roadblock:
  • 50 data analysts over 5 years estimate makes project appear to be unbounded and infeasible
  • Status:
  • Project put on hold
health insurance company outsourcing development
Health Insurance Company: Outsourcing Development
  • Business Problem:
  • Data must be sent to India for offshore application development.
  • Sensitive data must be masked for HIPAA compliance
  • Proposed Solution:
  • Mask sensitive data before sending it outside the company
  • Roadblock:
  • Sensitive data, where is it?
  • Can two sets of data that individually contain no sensitive data be combined to make it sensitive?
  • Status:
  • Manual discovery of sensitive data slows outsourcing to a crawl
wall street firm data consistency will increase profitability
Wall Street Firm: Data Consistency will Increase Profitability
  • Business Problem:
  • Transaction errors are expensive and the risk of regulatory fines due to inconsistent reference data is unacceptable
  • Proposed Solution:
  • Deploy a master data management solution
  • Roadblock:
  • 5 years to determine the business rules that relate the master data system to legacy systems
  • Unable map two tables to each other after 6 weeks of work (70 tables total to map)
  • Status:
  • Project on hold
auto insurance migrating fragile legacy integration code to modern tools
Auto Insurance: Migrating Fragile Legacy Integration Code to Modern Tools
  • Business Problem:
  • Business changes force expensive and difficult to implement changes in hand written legacy integration code
  • Proposed Solution:
  • Migrate legacy code to a modern ETL (extract, transform, load) tool. Cost of maintenance of ETL is a fraction of legacy code
  • Roadblock:
  • No one knows the code. The cost of migration is unpredictable.
  • Status
  • Company continues to manually change hand written code ad hoc as the business demands
the common in the project schedule







T= 0

Project Timeline


The Common “?” in the Project Schedule
  • Data Relationship Discovery
  • You have to know where your data is, how it flows and relates across systems if you hope to secure it, move it, consolidate, integrate it ...
myth 1 we know our data
Myth #1: “We know our data”

I’m a professional. Of course I know my data!

  • Subject matter experts (SMEs) only know their own systems
  • But they can’t tell you how it changes and is transformed as it moves from system to system
  • Relationships between systems are complex:
  • SMEs sometimes change jobs!

But, once it leaves my hands, it is someone else’s problem!

Wow, that transformation is complex. Are you sure that is in my data?

I’m going to start my own consulting firm

myth 2 we know our data
Myth #2: “We know our data”

All of my data follows the business rules for this system!

  • Business rules are broken all the time as data crosses business and system boundaries:
    • 83 year old man in system A is a “youthful driver” in system B
    • Bond yield is listed as 5% in system X and 5.3% in system Y
  • Exceptions result in lost revenue, customer dissatisfaction, and regulatory fines
myth 3 we know our data
Myth #3: “We know our data”
  • Business rules change as organizations change
    • Mergers and Acquisitions
    • New products or services
    • Products/services are retired
    • Reorganizations
    • New IT systems are added

I can’t keep up with all the acquisitions and reorganizations. They mess up the way systems work together. It is very inconvenient.

the reality
The Reality
  • Companies lack a global view of their corporate data map
current trend data governance
Current Trend: Data Governance
  • What is it?
  • The latest over-hyped term
  • Data Integration is to Tactical asData Governance is to Strategic
  • Definition
  • Data Governance encompasses the people, processes and procedures to create a consistent, enterprise view of your data in order to:
    • Improve data security
    • Increase consistency & confidence in decision making
    • Decrease the risk of regulatory fines
the problem with data governance
The Problem with Data Governance
  • How do you do it?
    • Where is the sensitive data?
    • What are the business rules and data relationships
    • Where are the exceptions?
  • How do you ensure a consistent, repeatable process?
traditional proposed approach metadata

20th Century Data Relationship Discovery Tool

Traditional Proposed Approach: Metadata
  • What is it?
  • Another over-hyped term
  • Data about data: datatype (character, integer, number, date etc), column width, frequency, cardinality etc
  • The Problem
  • Single system metadata only:
    • Profiling
  • Traditional data integration tools do not discover metadata
    • Cleansing, ETL, EAI and EII
  • The Reality
  • Data analysts manually examine data values to figure out the data map
  • The most sophisticated tool generally used today is:

Traditional Data Relationship Discovery Tool

the solution data driven relationship discovery
The Solution: Data-Driven Relationship Discovery
  • New approach to a 40 year old problem
  • Sophisticated heuristics and algorithms analyze actual data values
  • Automates the discoveryand validation of:
    • Sensitive data flows
    • Business rules
    • Complex transformations
  • between structured data sets in a consistent and repeatable manner
solution data driven exception discrepancy discovery
Solution: Data-Driven Exception & Discrepancy Discovery
  • Identify exceptions to avoid:
    • Regulatory fines
    • Lost revenue
    • Customer dissatisfaction



data driven discovery results
Credit Card Company

Status:Project moving forward again

Reduced estimated effort from 250 engineering years to 25 eng. years

Eliminated project feasibility risk

Wall Street Firm

Status:Back on track

Over 5x (2 days vs 6 weeks manually) improvement in discovery of business rules made MDM project possible

Found bond yield discrepancies

Data-Driven Discovery Results
  • Health Insurance Company
  • Status:Outsourcing rollout accelerated
  • Now confident in sensitive data discovery accuracy and speed
  • Launching new data masking service companywide
  • Auto Insurance Company
  • Status:Predictable & affordable migration
  • 80% reduction in effort required to migrate hand-code to ETL tool
  • Mapping process discovered potentially costly business rule errors
summary data governance strategic data integration
Summary: Data Governance = Strategic Data Integration
  • Companies are implementing data governance projects to:
    • Improve Security
    • Increase Consistency
    • Decrease Regulatory Risk
  • First step of data governance… Discovery
  • Automated data-driven discovery is a consistent, repeatable and proven approach to identify:
    • Sensitive Data
    • Business Rules
    • Data Exceptions
key contacts
Key Contacts
  • Bob Shannon: U.S. East Coast Sales
    • Phone: (203) 878-8472
    • Email:
  • Brian Smogard: U.S. Central Sales
    • Phone: (612) 605-9236
    • Email:
  • Clive Harrison: U.S. West Coast and International Sales
    • Phone: 415-608-4632
    • Email:
  • If you have any other follow up questions, contact me:
  • Todd Goldman
    • Phone: (408) 919-0191 ext 1115
    • Email: