1 / 24

TotalETL: Advancing ETL Solutions for Multi-User Environments

TotalETL is a pioneering ETL (Extract, Transform, Load) company that specializes in creating robust solutions for large-scale data operations. The MQP project aims to prototype a client-server version of the existing infoSight desktop solution, transitioning towards a distributed, multi-user database-centric architecture. Key features include a comprehensive security model, job scheduling, version control, and event management. With aspirations to enhance software capabilities, TotalETL focuses on back-end design, facilitating seamless project management and robust database interactions.

darin
Download Presentation

TotalETL: Advancing ETL Solutions for Multi-User Environments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. TotalETL:infoServer Chris Fournier Nathan Clark Scott Longley Cyril Shilnikov MQP Project 2005 Sponsored by TotalETL inc.

  2. TotalETL • Small ETL Company • ETL (Extract Transform Load) • Used in large companies • Multimillion dollar business • Existing Product is infoSight-- desktop solution

  3. infoSight

  4. infoSight Current Features • GUI Project creation • Library of Transformers • Works with multiple input types • Single machine • Single user • One project at a time

  5. MQP Project goals • Prototype the client-server version of infoSight • Distributed • Multi-user • Database-centric • Extensible • Alpha-level code • Focus on back-end design

  6. Project Methodology • Met with TotalETL team on-site • Design requirements • Refine and discuss requirements as needed • Build core modules, demo end first term • Build additional modules, final demo.

  7. General design overview Thin Clients Distributed Server System Repository Thick Clients

  8. Actual design overview Client Security Manager Session Manager Event & Log Manager Project Manager Job Manager Version Manager Scheduling Manager Repository Manager DB

  9. Repository Manager • System core • Store all information about • System operation • Security • Projects • XML Parser to store Projects • JDBC to connect to DB’s

  10. Repository Table Design

  11. Project Manager & Version Control • Storage and Retrieval of Projects • In-memory Object -> XML File -> Repository • Version Control • Per user locking • Version tracking

  12. Job Manager • Combine Projects into Jobs • Set interdependencies • Running Jobs

  13. Schedule Manager • Schedule Jobs • On request • Per schedule • Multiple scheduling strategies

  14. Session Manager • Establish and maintain client connections • RMI • Simple, robust, built-in to Java • Front end for all functions in server • Security checking • Authentication of users • Authorization of commands

  15. Security Manager • Determine user’s privileges • Control access to Projects, Jobs, etc. • Custom Security Model • Role-based ACLs • Read, Write, Execute (Projects and Jobs) • Read, Create, Modify (System Configuration)

  16. Event Manager & Logger • Useful for future expansion • Complex Hierarchy of Events • All Events Logged • Log4J format

  17. Event Hierarchy InfoserverEvent UserEvent UserLoginEvent UserLoginFailedEvent ProjectEvent UserLogoutEvent (Other level-2 events) (Other level-3 events) Listeners

  18. Saving and Loading Projects Client Security Manager Session Manager Event & Log Manager Project Manager Version Manager Repository Manager DB

  19. Creating Jobs from Projects Client Security Manager Session Manager Event & Log Manager Project Manager Job Manager Repository Manager DB

  20. Scheduling Jobs to Run Client Security Manager Session Manager Event & Log Manager Job Manager Scheduling Manager Repository Manager DB

  21. Project Summary • Relational Database storage • Projects • Operational Information • Job Scheduling • Tailored Security Model • Version control • Logging

  22. Future work • Distributed servers • Clients, thick and thin • Support for more databases • More advanced scheduling algorithms

  23. Thanks • Professor E. A. Rundensteiner • Arun Shastry • Greg Goldberg • Rest of the TotalETL Team

  24. Questions?

More Related