Building a reusable data integration framework
This presentation is the property of its rightful owner.
Sponsored Links
1 / 18

Building a Reusable Data Integration Framework PowerPoint PPT Presentation


  • 55 Views
  • Uploaded on
  • Presentation posted in: General

Building a Reusable Data Integration Framework. Strategic Agility. Matt Siano Principal Database Engineer HBS Information Technology June 2011. David Aznavoorian Director, Database Services HBS Information Technology June 2011. Agenda . Evolution of Enterprise Data Integration

Download Presentation

Building a Reusable Data Integration Framework

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Building a reusable data integration framework

Building a Reusable Data Integration Framework

Strategic Agility

Matt Siano

Principal Database Engineer

HBS Information Technology

June 2011

David Aznavoorian

Director, Database Services

HBS Information Technology

June 2011


Agenda

Agenda

  • Evolution of Enterprise Data Integration

  • The Need for a Data Integration Framework

    • Why Change?

    • Steps Taken by HBS

  • Enabling Change

    • Where to Start?

    • Key Considerations

  • The Result

    • A Look Inside

    • Success Cases

    • Lessons Learned

  • Questions


What s changing in enterprise data integration at hbs

What’s changing in Enterprise Data Integration at HBS?

  • An evolution away from point solutions coding. While traditional data interfaces are ‘hardcoded’ solutions, specific to one use. A data integration framework is intended to define the components of a reusable information architecture, whereby publishing and consuming updates across the Enterprise are based on common Business Entity Objects (Master data).


Why develop a data integration framework

Why develop a Data Integration Framework?

Information is an asset – as long as you can share it consistently


Traditional data interfaces are hardcoded solutions specific to one use

Traditional data interfaces are ‘hardcoded’ solutions, specific to one use…

Even experienced developers have trouble managing the data flow at this level


What are the warning signs for change

What are the warning signs for change?

  • Application upgrades and new releases cause significant data integration re-development cycles – Database to Database

  • Business process improvements difficult to implement without changing code – Hardcode Business Rules

  • Only “Matt” knows how that works – Single points of failure!

    What’s your favorite problem?

    Bottom Line:

  • We all pay the price in Time ($), Credibility and Quality


The steps taken at hbs towards change

The steps taken at HBS towards change

  • Being a Change Agent

    • Recognize you have a problem

    • Communicate and build a group of internal advocates

  • Build a Clear Case/DI Strategy

    • Market this as an Enterprise level strategy change

    • Remediating current problems is only a short-term goal. Plan and invest for the future, or you might never get that chance “next year”

  • Create a Governance model

    • Identify the HBS Groups that will influence, either directly or indirectly, the requirements upon which the HBS Enterprise Data Integration Strategy will be developed and deployed.


Enabling change

Enabling Change

Turning a Roadmap into Reality


Where did we start informatica is at the core of the di framework

Where did we start?…Informatica is at the core of the DI Framework

PowerCenter (Advanced Edition)

  • An enterprise data integration platform for accessing, discovering, and integrating data from virtually any business system, in any format, and delivering that data throughout the enterprise.

    Some major options procured:

  • Power Exchange for Web Servicesto act as a web services provider and consumer with 3rd party systems by using predefined transformations.

  • Real Time Optionto service requests on demand, as opposed to batch, for web services provider or other event based protocols.

  • High Availability Optionminimizes service interruptions during hardware and/or software outages and reduces costs associated with data downtime.

  • Data Cleanse and Match Optionfeatures powerful, integrated cleansing and matching capabilities to correct and remove duplicate customer data.


From building blocks to a di framework

From building blocks to a DI Framework

Thinking about data as a Business Object

  • Shared Business Entity Objects within the HBS Enterprise database are defined by a common object XSD, as an abstraction of the base tables that make up that entity. Thereby they relieve the Source applications from having to understand the direct database table interrelationships.

    Designing a reusable Transformation Layer

  • Mappings are the details of the transaction data handling (i.e. Transformations) that occur from Source to Target. Examples of typical transformations can include derived data capture, complex calculations, data validation and cleansing.

    Define an Interface Protocol

  • The primary role of the interface, or abstraction layer is to transform Source data inputs into a common format for transaction processing. The format we chose to represent these data inputs in a common form is XML.


Let s take a look inside

Let’s take a look inside

Informatica and HEDEx


Hbs enterprise data exchange hedex

HBS Enterprise Data Exchange (HEDEx)


When to use a di framework

When to use a DI framework

Guidelines for your team


When to use a di framework1

When to use a DI framework


Start with small success stories to build upon

Start with small success stories to build upon

External Relations – Alumni Data Management

  • An enhanced reporting solution for Donor Behavior which interfaces with our external Alumni clubs (Alumni Magnet) and data from the Harvard University Advance system. Type – Bulk Data Load Publisher

    • “I am so grateful and excited that we finally have these reports to guide and evaluate our fundraising strategies” – Charlotte Smith, Director, Development Marketing and Production

      Executive Education – Program Delivery

  • Transfers participant and program data, and enables the transition from enrollment (CRM) to program delivery tools that are Enterprise-based applications. Type – Change Data Publisher

    • Reduced maintenance hours - 40% drop from the same review period in the previous year and provides early notification of errors and self-service correction.


The real value in a di framework

The real value in a DI Framework!


Lessons we ve learned so far

Lessons we’ve learned so far

Form a Data Integration Special Interest Group to address

  • Define standards around Business Entity Objects for DI use across the enterprise.

  • Create a framework for documentation, review, development and consumption of DI components.

  • Draft a DI Strategy and Roadmap: A DI framework is a product like any other!

    Not everything works the way its advertised

  • Informatica is an enterprise level tool suite (e.g. not Toad for Oracle). The greater the effort put into building the Mappings and Workflows the first time, the better the opportunity for re-use. Caution: allow for the unexpected

    Training is not optional

  • While skill and determination will get you into a new technical tool suite, there is no substitute for formal training.


So what s next

So what’s next…

  • Share Point Integration…

  • Oracle PeopleSoft (Campus Solutions) Integration…

  • Restate our HEDEx Roadmap…


  • Login