1 / 16

DiDaS Distributed Data Storage

Explore the distributed data storage project at Masaryk University in collaboration with CESNET. Learn about the motivation, infrastructure, applications, and future extensions of this innovative network storage solution.

effiej
Download Presentation

DiDaS Distributed Data Storage

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DiDaSDistributed Data Storage Ludek Matyska Masaryk University, Institute of Comp. Sci. And CESNET, z.s.p.o Ludek.Matyska@muni.cz APAN, Logistical Networking WS

  2. Outline • Motivation • Infrastructure • Applications • Future extensions APAN, Logistical Networking WS

  3. Motivation • Increased need for network storage • Computational Grids • Data Grids • Temporary Data Deposits • Transient Caches • Video deposits • National Library Requirements • Distribution of digitized content APAN, Logistical Networking WS

  4. Requirements • Transparent • Location independent • Good geographical distribution • Providing support for • Access quality (e.g. Streaming) • Reliability (no single point of failure) APAN, Logistical Networking WS

  5. Infrastructure • Data depots • Control: Personal computer • Storage: RAID of IDE disks • Capacity 1,5 TB each • Number: 7 (total capacity  10 TB) • Connectivity • Directly to the backbone • 100 Mb/s or 1 Gb/s APAN, Logistical Networking WS

  6. APAN, Logistical Networking WS

  7. APAN, Logistical Networking WS

  8. Data Layer • IBP (70% capacity) • General use • GridFTP servers (30% capacity) • Grid support • Computer independent temporary data storage • Comparison with IBP based solution APAN, Logistical Networking WS

  9. Traffic optimisation • Network traffic cost function • Inter-depots topology known • Instrumented clients • Measurement from depot to client • Simultaneous data transfer and measurements • Real-time transfer rate prediction • Choose depot • Decision between point and multipoint transfers APAN, Logistical Networking WS

  10. Applications • National Technical Library • Video Streaming • Nonspecific Users APAN, Logistical Networking WS

  11. National Technical Library • Requirements • Program of content digitalisation • Data stored on the central tape robot • Not optimised for distribution • Danger of overload • Model data: old cartographic maps APAN, Logistical Networking WS

  12. National Technical Library • DiDaS role • Cache like storage • Load balancing optimisation • Data transfer reliability (multistreaming) APAN, Logistical Networking WS

  13. Video Streaming • Permanent storage • Specific clients • QoS requirements (pre-caching) • Replica management • Not yet implemented APAN, Logistical Networking WS

  14. Nonspecific Users • Temporary data deposits • Provide data for load balancing • Transfer outside of DiDaS core • Access reliability • Automatic replica generation • Transparent multi-access • Ability to react on connectivity loss APAN, Logistical Networking WS

  15. Future work • New clients development • support for new application areas • Extended and transparent replica management • Full instrumentation • Data for • Load balancing • Replica creation/deletion • User access optimisation APAN, Logistical Networking WS

  16. Thank you for your interest APAN, Logistical Networking WS

More Related