1 / 17

Comparison of Data Storage Middleware for HPC, GRID, and Cloud

This paper compares various distributed data storage middleware solutions for high-performance computing, GRID, and cloud environments. It examines the European Middleware Initiative (EMI) and its storage solutions, as well as traditional approaches and modern technologies. The implementation details of dCache-based Tier 1 WLCG storage are also discussed, along with performance testing and future evaluation plans.

reyese
Download Presentation

Comparison of Data Storage Middleware for HPC, GRID, and Cloud

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A comparisonof distributed data storage middleware for HPC, GRID and Cloud Mikhail Goldshtein1, Andrey Sozykin1, GrigoryMasich2 and Valeria Gribova3 1Institute of Mathematics and Mechanics UrB RAS, Russia, Yekaterinburg 2Institute of Continuous Media Mechanics UrB RAS, Russia, Perm 3Institute of Automation and Control Processes FEB RAS, Russia, Vladivostok

  2. European Middleware Initiative EMI - Software platform for high performance distributed computing, http://www.eu-emi.eu Joint effort of the major European distributed computing middleware providers (ARC, dCache, gLite, UNICORE) Widely used in Europe, including Worldwide LHC Computing Grid (WLCG) Higgs boson: Alberto Di Meglio: Without the EMI middleware, such an important result could not have been achieved in such a short time

  3. Storage solutions in EMI dCache - http://www.dcache.org/ Disk Pool Manager (DPM) - https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm StoRM (STOrageResource Manager) - http://storm.forge.cnaf.infn.it/

  4. dCache

  5. Disk Pool Manager

  6. StoRM

  7. Usage statistics in WLCG

  8. Distributed storage systems Traditional approach: • Grid • Distributed file systems (IBM GPFS, Lustre File System, etc.) Modern technologies: • Standard Internet Protocols (Parallel NFS, WebDAV, etc.) • Cloud storage (Amazone S3, HDFS, etc.)

  9. Classic NFS

  10. Parallel NFS

  11. Comparison results

  12. Distributed dCache based Tire 1 WLCG storage

  13. Implementation

  14. Implementation details Hardware: 4 x Supermicro servers (3 in Yekaterinburg, 1 in Perm), 210 TB useful capacity (252 full capacity, RAID5 + Hotspare are used) ОС Scientific Linux 6.3 dCache 2.6 from EMI repository Protocol: NFS v4.1 (Parallel NFS) RHEL has a parallel NFS client, no need to install additional software to clusters

  15. Performance testing IOR test (http://www.nersc.gov/systems/trinity-nersc-8-rfp/nersc-8-trinity-benchmarks/ior/)

  16. Future works Evaluation of NFS performance over 10GE and WAN Evaluation of dCache in the experiments (Particle Image Velocimetry and so on) Participation in GRID projects: • Grid of Russian National Nanotechnology Network • WLCG (through Joint Institute for Nuclear Research, Dubna, Russia) Connection to Hadoop Cluster (when dCache will support HDFS)

  17. Andrey Sozykin Institute of Mathematics and Mechanics UrB RAS, Russia, Yekaterinburg avs@imm.uran.ru Thank you!

More Related