Slide1 l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 15

GENERIC ETL DESIGN PowerPoint PPT Presentation


  • 188 Views
  • Uploaded on
  • Presentation posted in: General

GENERIC ETL DESIGN . VARADARAJAN VASU. [email protected] SENIOR PROJECT MGR/ARCHITECT. POLARIS SOFTWARE LAB . OBJECTIVE. Application area is bifurcated as ETL / Reporting . Major Operations Select/Insert/Update/Delete

Download Presentation

GENERIC ETL DESIGN

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Slide1 l.jpg

GENERIC ETL DESIGN

VARADARAJAN VASU

[email protected]

SENIOR PROJECT MGR/ARCHITECT

POLARIS SOFTWARE LAB


Slide2 l.jpg

OBJECTIVE

  • Application area is bifurcated as ETL / Reporting .

  • Major Operations Select/Insert/Update/Delete

  • To replace existing primitive methods used for ETL Design/ Automation

  • System should be highly intelligent to do all jobs on behalf of users

  • Build a comprehensive solution once and use it across verticals


Slide3 l.jpg

PERT PROCESS

  • PERT Stands for PROGRAM EXECUTION on REMOTE TERMINALS

  • Different from Program Evaluation Review Technique used by SEI

  • Technology used in Client/Server architecture


Slide4 l.jpg

PERT PROCESS FLOW

PERT START

FREE SPACE CHECK

ORACLE PROCESSES CHECK


Slide5 l.jpg

EXECUTABLE PRESENCE CHECK

PROCEDURE VALIDITY CHECK

CHECK FOR PARELLEL RUN

CHECK FOR RESTARTABILITY


Slide6 l.jpg

1. SYSTEM INTELLIGENT CHEKS - PARAMETERISED

2. DATE CHANGE - PARAMETERISED

3. DETERMINE STAGING RUN INFORMATION - PARAMETERISED

4. STAGE REFRESH LOADER

5. GATHER FINAL REFRESHINFORMATION - PARAMETERISED

6. FINAL REFRESH LOADER

7. DATA VALIDATION CHECKS - PARAMETERISED

8. MAKE SYSTEM READY FOR NEXT DAY RUN - PARAMETERISED

SUCCESS

PERT END


Slide7 l.jpg

SYSTEM INTELLIGENT CHECKS - Examples

  • SPACE CHECK

  • OBJECTS VALIDITY CHECK

  • EXECUTABLES VALIDITY CHECK

  • PROCESS RUNNING CHECK

  • PREVENT SUCCESS RUN

  • PREVENT PARELLL RUN

  • RESTARTABILITY

  • HANDLE UNAVOIDABLE INTERRUPTS FROM OS


Slide8 l.jpg

OPERATION READINESS- Examples

  • ARCHIVE

  • INDEXING

  • COMMUNICATING WITH EXTERNAL PARTIES

  • MAILING

  • COMPILING ETL EXECUTION STATISTICS

  • MOVING OBJECTS TO RESPECTIVE LOCATION

  • ANALYZING

  • CLEANUP EXERCISE


Slide9 l.jpg

SALIENT FEATURES OF PERT

  • SPACE CHECK

  • PROCEDURE OBJECTS VALIDITY CHECK

  • EXECUTABLES VALIDITY CHECK

  • PREVENT SUCCESS RUN

  • PREVENT PARELLL RUN

  • RESTARTABILITY

  • PROVISION TO SCHEDULE FOR UPCOMING RUN FREQUENCIES

  • BETTER ERROR LOGGING

  • HANDLE UNAVOIDABLE INTERRUPTS FROM OS

  • Load check for staging , Final

  • Provision for manual run


Slide10 l.jpg

ADVANTAGES

  • Design is dynamic in nature

  • Limited time availability to plug in new facility

  • Avoid redundancy in coding & testing efforts

  • Sleeping beauty is cost effectiveness

  • Restart facility to start from the aborted place during data extraction and population

  • ETL solution can be used for other similar ETL applications.


Slide11 l.jpg

CHALLENGES

  • Requirements Gathering

  • Database Design

  • Performance in Execution


Slide12 l.jpg

CASE STUDY


Slide13 l.jpg

NEAR REAL TIME EDW POPULATION


Slide14 l.jpg

CASE STUDY PROCESSING


Slide15 l.jpg

THANK YOU


  • Login