1 / 39

BigDataStack Data Services Overview June 26 th 2019

BigDataStack Data Services Overview June 26 th 2019. Yosef Moatti – IBM Haifa Research Labs. Agenda. BigDataStack project Overview Danaos Real-time Shipping BigDataStack Data Services Overview. Real time shipping: business Goals. Recap of the Business Goals :

kristij
Download Presentation

BigDataStack Data Services Overview June 26 th 2019

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. BigDataStackData Services OverviewJune 26th 2019 Yosef Moatti – IBM Haifa Research Labs

  2. Agenda • BigDataStack project Overview • Danaos Real-time Shipping • BigDataStack Data Services Overview Periodic Review Meeting

  3. Real time shipping: business Goals Recap of the Business Goals: • Accurately predict specific main engine failures • Gets informed of “problems” ASAP • Broken sensors • Infringement of business rules (e.g., fuel consumption) • Build a solution which will also fit the coming Danaos data ingestion rates: Periodic Review Meeting

  4. Real time shipping: business Goals Recap of the Business Goals: • Accurately predict specific main engine failures • Gets informed of “problems” ASAP • Broken sensors • Infringement of business rules (e.g., fuel consumption) • Build a solution which will also fit the coming Danaos data ingestion rates: Periodic Review Meeting

  5. Real time shipping: business Goals Recap of the Business Goals: • Accurately predict specific main engine failures • Gets informed of “problems” ASAP • Broken sensors • Infringement of business rules (e.g., fuel consumption) • Build a solution which will also fit the coming Danaos data ingestion rates: Periodic Review Meeting

  6. Real time shipping: business Goals Recap of the Business Goals: • Accurately predict specific main engine failures • Gets informed of “problems” ASAP • Broken sensors • Infringement of business rules (e.g., fuel consumption) • Build a solution which will also fit the coming Danaos data ingestion rates: Periodic Review Meeting

  7. Real time shipping: business Goals Recap of the Business Goals: • Accurately predict specific main engine failures • Gets informed of “problems” ASAP • Broken sensors • Infringement of business rules (e.g., fuel consumption) • Build a solution which will also fit the coming Danaos data ingestion rates: Periodic Review Meeting

  8. Real time shipping: business Goals Recap of the Business Goals: • Accurately predict specific main engine failures • Gets informed of “problems” ASAP • Broken sensors • Infringement of business rules (e.g., fuel consumption) • Build a solution which will also fit the coming Danaos data ingestion rates: Periodic Review Meeting

  9. Real time shipping: business Goals Recap of the Business Goals: • Accurately predict specific main engine failures • Gets informed of “problems” ASAP • Broken sensors • Infringement of business rules (e.g., fuel consumption) • Build a solution which will also fit the coming Danaos data ingestion rates: Periodic Review Meeting

  10. Constraints on the Solution • Alerts must be triggered on streaming data • Access to recent data should be highly performant • Data scientists wish not to mind about data location Periodic Review Meeting

  11. High level view of the solution Vessels IoT Devices Gateway Edge Seamless component BigDataStack Data Center

  12. High level view of the solution Streaming data handled within data center Streaming data handled at Edge Vessels IoT Devices CEP CEP Gateway Streaming Data Management Edge Seamless component BigDataStack Data Center

  13. High level view of the solution Remote IBM COS Historic data Historic data Recent data Vessels IoT Devices LXS DB CEP CEP Gateway Edge Seamless component BigDataStack Data Center

  14. Recent data in LXS can be accessed directly Remote IBM COS `1 Data service Data scientist Vessels IoT Devices LXS DB CEP CEP Gateway Edge Seamless component BigDataStack Data Center

  15. Data Services which directly access LXS DB Remote IBM COS `1 Danaos Preventive Maintenance alg. Data Quality Assessment Vessels IoT Devices LXS DB CEP CEP Gateway Edge Seamless component Danaos Alarm detection daemon BigDataStack Data Center

  16. Historic data in Object Store can be accessed directly Remote IBM COS Data service Data scientist Vessels IoT Devices LXS DB CEP CEP Gateway Edge Seamless component BigDataStack Data Center

  17. Dataset can be accessed seamlessly Remote IBM COS Data service Data scientist Vessels IoT Devices LXS DB CEP CEP Gateway Edge Seamless component BigDataStack Data Center

  18. Recent data in LXS can be accessed directly Remote IBM COS `1 Data service Data scientist Vessels IoT Devices LXS DB with Data Skipping Data Mover CEP CEP Gateway Edge Seamless component BigDataStack Data Center

  19. Phase 1: processing at Edge Sensor data Periodic Review Meeting

  20. Phase 1: processing at Edge CEP at Edge At Edge Sensor data Periodic Review Meeting

  21. Phase 1: processing at Edge CEP at Edge At Edge CEP1 Sensor data Basic rules check Periodic Review Meeting

  22. Phase 1: processing at Edge CEP at Edge At Edge CEP1 2018-10-23 08:00:00,stw,15.062018-10-23 06:36:00,stw,15.162018-10-23 15:56:00,stw,14.92018-10-23 15:57:00,stw,14.922018-10-24 08:30:00,stw,16.182018-12-21 18:42:00,stw,17.112018-12-21 18:43:00,stw,16.922018-12-21 18:45:00,stw,16.772018-10-23 16:01:00,stw,14.74  Sensor data Sensor data 2018-10-23 08:00:00,stw,15.06, 02018-10-23 06:36:00,stw,15.16, 02018-10-23 15:56:00,stw,14.9, 02018-10-23 15:57:00,stw,14.92, 12018-10-24 08:30:00,stw,16.18. 02018-12-21 18:42:00,stw,17.11, 02018-12-21 18:43:00,stw,16.92, 02018-12-21 18:45:00,stw,16.77, 12018-10-23 16:01:00,stw,14.74, 0 Periodic Review Meeting

  23. Phase 1: processing at Edge CEP at Edge At Edge CEP1 2018-10-23 08:00:00,stw,15.062018-10-23 06:36:00,stw,15.162018-10-23 15:56:00,stw,14.92018-10-23 15:57:00,stw,14.922018-10-24 08:30:00,stw,16.182018-12-21 18:42:00,stw,17.112018-12-21 18:43:00,stw,16.922018-12-21 18:45:00,stw,16.772018-10-23 16:01:00,stw,14.74  Sensor data Sensor data 2018-10-23 08:00:00,stw,15.06, 02018-10-23 06:36:00,stw,15.16, 02018-10-23 15:56:00,stw,14.9, 02018-10-23 15:57:00,stw,14.92, 12018-10-24 08:30:00,stw,16.18. 02018-12-21 18:42:00,stw,17.11, 02018-12-21 18:43:00,stw,16.92, 02018-12-21 18:45:00,stw,16.77, 12018-10-23 16:01:00,stw,14.74, 0 Added columm: Alarm Info Periodic Review Meeting

  24. Phase 1: processing at Edge CEP at Edge At Edge CEP1 2018-10-23 08:00:00,stw,15.062018-10-23 06:36:00,stw,15.162018-10-23 15:56:00,stw,14.92018-10-23 15:57:00,stw,14.922018-10-24 08:30:00,stw,16.182018-12-21 18:42:00,stw,17.112018-12-21 18:43:00,stw,16.922018-12-21 18:45:00,stw,16.772018-10-23 16:01:00,stw,14.74  Sensor data Sensor data 2018-10-23 08:00:00,stw,15.06, 02018-10-23 06:36:00,stw,15.16, 02018-10-23 15:56:00,stw,14.9, 02018-10-23 15:57:00,stw,14.92, 12018-10-24 08:30:00,stw,16.18. 02018-12-21 18:42:00,stw,17.11, 02018-12-21 18:43:00,stw,16.92, 02018-12-21 18:45:00,stw,16.77, 12018-10-23 16:01:00,stw,14.74, 0 Added columm: Alarm Info Periodic Review Meeting

  25. Phase 2: data ingestion in Data Center Data streams Testbed Gateway Periodic Review Meeting

  26. Phase 2: data ingestion in Data Center 2018-10-23 08:00:00,stw,15.06, 02018-10-23 06:36:00,stw,15.16, 02018-10-23 15:56:00,stw,14.9, 02018-10-23 15:57:00,stw,14.92, 12018-10-24 08:30:00,stw,16.18. 02018-12-21 18:42:00,stw,17.11, 02018-12-21 18:43:00,stw,16.92, 02018-12-21 18:45:00,stw,16.77, 12018-10-23 16:01:00,stw,14.74, 0 Data streams Sensor data Testbed Gateway Periodic Review Meeting

  27. Phase 2: data ingestion in Data Center CEP at Data Center 2018-10-23 08:00:00,stw,15.06, 02018-10-23 06:36:00,stw,15.16, 02018-10-23 15:56:00,stw,14.9, 02018-10-23 15:57:00,stw,14.92, 12018-10-24 08:30:00,stw,16.18. 02018-12-21 18:42:00,stw,17.11, 02018-12-21 18:43:00,stw,16.92, 02018-12-21 18:45:00,stw,16.77, 12018-10-23 16:01:00,stw,14.74, 0 Data streams Sensor data Testbed Gateway Periodic Review Meeting

  28. Phase 2: data ingestion in Data Center CEP at Data Center 2018-10-23 08:00:00,stw,15.06, 02018-10-23 06:36:00,stw,15.16, 02018-10-23 15:56:00,stw,14.9, 02018-10-23 15:57:00,stw,14.92, 12018-10-24 08:30:00,stw,16.18. 02018-12-21 18:42:00,stw,17.11, 02018-12-21 18:43:00,stw,16.92, 02018-12-21 18:45:00,stw,16.77, 12018-10-23 16:01:00,stw,14.74, 0 Data streams Sensor data Testbed Gateway Complex rules check Periodic Review Meeting

  29. Phase 2: data ingestion in Data Center CEP at Data Center 2018-10-23 08:00:00,stw,15.06, 02018-10-23 06:36:00,stw,15.16, 02018-10-23 15:56:00,stw,14.9, 02018-10-23 15:57:00,stw,14.92, 12018-10-24 08:30:00,stw,16.18. 02018-12-21 18:42:00,stw,17.11, 02018-12-21 18:43:00,stw,16.92, 02018-12-21 18:45:00,stw,16.77, 12018-10-23 16:01:00,stw,14.74, 0 Data streams Sensor data Testbed Gateway Complex rules check LeanXcale DB Business Agreement Data KiVi storage system Periodic Review Meeting

  30. Phase 2: data ingestion in Data Center CEP at Data Center 2018-10-23 08:00:00,stw,15.06, 02018-10-23 06:36:00,stw,15.16, 02018-10-23 15:56:00,stw,14.9, 02018-10-23 15:57:00,stw,14.92, 12018-10-24 08:30:00,stw,16.18. 02018-12-21 18:42:00,stw,17.11, 02018-12-21 18:43:00,stw,16.92, 02018-12-21 18:45:00,stw,16.77, 12018-10-23 16:01:00,stw,14.74, 0 Data streams Sensor data Sensor data Testbed Gateway 2018-10-23 08:00:00,stw,15.06, 02018-10-23 06:36:00,stw,15.16, 02018-10-23 15:56:00,stw,14.9, 02018-10-23 15:57:00,stw,14.92, 12018-10-24 08:30:00,stw,16.18. 12018-12-21 18:42:00,stw,17.11, 02018-12-21 18:43:00,stw,16.92, 02018-12-21 18:45:00,stw,16.77, 12018-10-23 16:01:00,stw,14.74, 0 Complex rules check LeanXcale DB KiVi storage system Business Agreement Data Periodic Review Meeting

  31. Phase 2: data ingestion in Data Center CEP at Data Center 2018-10-23 08:00:00,stw,15.06, 02018-10-23 06:36:00,stw,15.16, 02018-10-23 15:56:00,stw,14.9, 02018-10-23 15:57:00,stw,14.92, 12018-10-24 08:30:00,stw,16.18. 02018-12-21 18:42:00,stw,17.11, 02018-12-21 18:43:00,stw,16.92, 02018-12-21 18:45:00,stw,16.77, 12018-10-23 16:01:00,stw,14.74, 0 Data streams Sensor data Sensor data Sensor data Testbed Gateway 2018-10-23 08:00:00,stw,15.06, 02018-10-23 06:36:00,stw,15.16, 02018-10-23 15:56:00,stw,14.9, 02018-10-23 15:57:00,stw,14.92, 12018-10-24 08:30:00,stw,16.18. 12018-12-21 18:42:00,stw,17.11, 02018-12-21 18:43:00,stw,16.92, 02018-12-21 18:45:00,stw,16.77, 12018-10-23 16:01:00,stw,14.74, 0 Complex rules check LeanXcale DB Business Agreement Data KiVi storage system Anomaly table Periodic Review Meeting

  32. Phase 3: Data Quality Assessment Added Data qualitycolumn Data Quality Assessment Sensor data 2018-10-23 08:00:00,stw,15.06, 0, 0.992018-10-23 06:36:00,stw,15.16, 0, 0.972018-10-23 15:56:00,stw,14.9, 0, 0.942018-10-23 15:57:00,stw,14.92, 1, 0.332018-10-24 08:30:00,stw,16.18. 1, 0.422018-12-21 18:42:00,stw,17.11, 0, 0.722018-12-21 18:43:00,stw,16.92, 0, 0.992018-12-21 18:45:00,stw,16.77, 1, 0.662018-10-23 16:01:00,stw,14.74, 0. 0.98 . 2018-10-23 08:00:00,stw,15.06, 02018-10-23 06:36:00,stw,15.16, 02018-10-23 15:56:00,stw,14.9, 02018-10-23 15:57:00,stw,14.92, 12018-10-24 08:30:00,stw,16.18. 12018-12-21 18:42:00,stw,17.11, 02018-12-21 18:43:00,stw,16.92, 02018-12-21 18:45:00,stw,16.77, 12018-10-23 16:01:00,stw,14.74, 0 KiVi storage system LeanXcale DB Periodic Review Meeting

  33. Phase 4: Preventive Maintenance Preventive Maintenance ML. 2018-10-23 08:00:00,stw,15.06, 0, 0.992018-10-23 06:36:00,stw,15.16, 0, 0.972018-10-23 15:56:00,stw,14.9, 0, 0.942018-10-23 15:57:00,stw,14.92, 1, 0.332018-10-24 08:30:00,stw,16.18. 1, 0.422018-12-21 18:42:00,stw,17.11, 0, 0.722018-12-21 18:43:00,stw,16.92, 0, 0.992018-12-21 18:45:00,stw,16.77, 1, 0.662018-10-23 16:01:00,stw,14.74, 0. 0.98 . Detected anomaly KiVi storage system LeanXcale DB Anomaly table: Ano1, …. Ano2, …. Periodic Review Meeting

  34. Phase 5: Danaos Alert Detection Daemon Danaos daemon KiVi storage system LeanXcale DB Anomaly table: Ano1, …. Ano2, …. Periodic Review Meeting

  35. Seamless Query path Seamless Component Periodic Review Meeting

  36. Seamless Query path SQL Query: SELECT id from emp WHERE emp = ‘007’ LXSFederator Seamless Component Periodic Review Meeting

  37. Seamless Query path SQL Query: SELECT id from emp WHERE emp = ‘007’ LXSFederator LXSDB Seamless Component Periodic Review Meeting

  38. Seamless Query path SQL Query: SELECT id from emp WHERE emp = ‘007’ LXSFederator Spark JDBC Thrift Connector LXSDB with Data Skipping Object Storage Seamless Component Periodic Review Meeting

  39. Thank you!

More Related