A comparison of distributed data storage middleware for hpc grid and cloud
Download
1 / 17

A comparison of distributed data storage middleware for HPC, GRID and Cloud - PowerPoint PPT Presentation


  • 114 Views
  • Uploaded on

A comparison of distributed data storage middleware for HPC, GRID and Cloud. Mikhail Goldshtein 1 , Andrey Sozykin 1 , Grigory Masich 2 and Valeria Gribova 3 1 Institute of Mathematics and Mechanics UrB RAS, Russia, Yekaterinburg

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' A comparison of distributed data storage middleware for HPC, GRID and Cloud' - domani


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
A comparison of distributed data storage middleware for hpc grid and cloud

A comparisonof distributed data storage middleware for HPC, GRID and Cloud

Mikhail Goldshtein1, Andrey Sozykin1, GrigoryMasich2 and Valeria Gribova3

1Institute of Mathematics and Mechanics UrB RAS, Russia, Yekaterinburg

2Institute of Continuous Media Mechanics UrB RAS, Russia, Perm

3Institute of Automation and Control Processes FEB RAS, Russia, Vladivostok


European middleware initiative
European Middleware Initiative

EMI - Software platform for high performance distributed computing, http://www.eu-emi.eu

Joint effort of the major European distributed computing middleware providers (ARC, dCache, gLite, UNICORE)

Widely used in Europe, including Worldwide LHC Computing Grid (WLCG)

Higgs boson:

Alberto Di Meglio: Without the EMI middleware, such an important result could not have been achieved in such a short time


Storage solutions in emi
Storage solutions in EMI

dCache - http://www.dcache.org/

Disk Pool Manager (DPM) - https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm

StoRM (STOrageResource Manager) - http://storm.forge.cnaf.infn.it/






Distributed storage systems
Distributed storage systems

Traditional approach:

  • Grid

  • Distributed file systems (IBM GPFS, Lustre File System, etc.)

    Modern technologies:

  • Standard Internet Protocols (Parallel NFS, WebDAV, etc.)

  • Cloud storage (Amazone S3, HDFS, etc.)







Implementation details
Implementation details

Hardware: 4 x Supermicro servers (3 in Yekaterinburg, 1 in Perm), 210 TB useful capacity (252 full capacity, RAID5 + Hotspare are used)

ОС Scientific Linux 6.3

dCache 2.6 from EMI repository

Protocol: NFS v4.1 (Parallel NFS)

RHEL has a parallel NFS client, no need to install additional software to clusters


Performance testing
Performance testing

IOR test (http://www.nersc.gov/systems/trinity-nersc-8-rfp/nersc-8-trinity-benchmarks/ior/)


Future works
Future works

Evaluation of NFS performance over 10GE and WAN

Evaluation of dCache in the experiments (Particle Image Velocimetry and so on)

Participation in GRID projects:

  • Grid of Russian National Nanotechnology Network

  • WLCG (through Joint Institute for Nuclear Research, Dubna, Russia)

    Connection to Hadoop Cluster (when dCache will support HDFS)


Thank you

Andrey Sozykin

Institute of Mathematics and Mechanics UrB RAS, Russia, Yekaterinburg

[email protected]

Thank you!


ad