A comparison of distributed data storage middleware for hpc grid and cloud
This presentation is the property of its rightful owner.
Sponsored Links
1 / 17

A comparison of distributed data storage middleware for HPC, GRID and Cloud PowerPoint PPT Presentation


  • 75 Views
  • Uploaded on
  • Presentation posted in: General

A comparison of distributed data storage middleware for HPC, GRID and Cloud. Mikhail Goldshtein 1 , Andrey Sozykin 1 , Grigory Masich 2 and Valeria Gribova 3 1 Institute of Mathematics and Mechanics UrB RAS, Russia, Yekaterinburg

Download Presentation

A comparison of distributed data storage middleware for HPC, GRID and Cloud

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


A comparison of distributed data storage middleware for hpc grid and cloud

A comparisonof distributed data storage middleware for HPC, GRID and Cloud

Mikhail Goldshtein1, Andrey Sozykin1, GrigoryMasich2 and Valeria Gribova3

1Institute of Mathematics and Mechanics UrB RAS, Russia, Yekaterinburg

2Institute of Continuous Media Mechanics UrB RAS, Russia, Perm

3Institute of Automation and Control Processes FEB RAS, Russia, Vladivostok


European middleware initiative

European Middleware Initiative

EMI - Software platform for high performance distributed computing, http://www.eu-emi.eu

Joint effort of the major European distributed computing middleware providers (ARC, dCache, gLite, UNICORE)

Widely used in Europe, including Worldwide LHC Computing Grid (WLCG)

Higgs boson:

Alberto Di Meglio: Without the EMI middleware, such an important result could not have been achieved in such a short time


Storage solutions in emi

Storage solutions in EMI

dCache - http://www.dcache.org/

Disk Pool Manager (DPM) - https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm

StoRM (STOrageResource Manager) - http://storm.forge.cnaf.infn.it/


Dcache

dCache


Disk pool manager

Disk Pool Manager


Storm

StoRM


Usage statistics in wlcg

Usage statistics in WLCG


Distributed storage systems

Distributed storage systems

Traditional approach:

  • Grid

  • Distributed file systems (IBM GPFS, Lustre File System, etc.)

    Modern technologies:

  • Standard Internet Protocols (Parallel NFS, WebDAV, etc.)

  • Cloud storage (Amazone S3, HDFS, etc.)


Classic nfs

Classic NFS


Parallel nfs

Parallel NFS


Comparison results

Comparison results


Distributed dcache based tire 1 wlcg storage

Distributed dCache based Tire 1 WLCG storage


Implementation

Implementation


Implementation details

Implementation details

Hardware: 4 x Supermicro servers (3 in Yekaterinburg, 1 in Perm), 210 TB useful capacity (252 full capacity, RAID5 + Hotspare are used)

ОС Scientific Linux 6.3

dCache 2.6 from EMI repository

Protocol: NFS v4.1 (Parallel NFS)

RHEL has a parallel NFS client, no need to install additional software to clusters


Performance testing

Performance testing

IOR test (http://www.nersc.gov/systems/trinity-nersc-8-rfp/nersc-8-trinity-benchmarks/ior/)


Future works

Future works

Evaluation of NFS performance over 10GE and WAN

Evaluation of dCache in the experiments (Particle Image Velocimetry and so on)

Participation in GRID projects:

  • Grid of Russian National Nanotechnology Network

  • WLCG (through Joint Institute for Nuclear Research, Dubna, Russia)

    Connection to Hadoop Cluster (when dCache will support HDFS)


Thank you

Andrey Sozykin

Institute of Mathematics and Mechanics UrB RAS, Russia, Yekaterinburg

[email protected]

Thank you!


  • Login