integrating hdf5 with srb n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Integrating HDF5 with SRB PowerPoint Presentation
Download Presentation
Integrating HDF5 with SRB

Loading in 2 Seconds...

play fullscreen
1 / 22

Integrating HDF5 with SRB - PowerPoint PPT Presentation


  • 131 Views
  • Uploaded on

Integrating HDF5 with SRB. Object-level Access to Remote Files. P eter Cao, NCSA December 1, 2005. Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration. Outline. Introduction to the HDF-SRB project The HDF-SRB model SRB Support in HDFView.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Integrating HDF5 with SRB' - angelo


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
integrating hdf5 with srb

Integrating HDF5 with SRB

Object-level Access to Remote Files

Peter Cao, NCSA

December 1, 2005

Sponsored by NLADR, NFS PACI Project

in Support of NCSA-SDSC Collaboration

HDF & HDF-EOS Workshop IX

outline
Outline
  • Introduction to the HDF-SRB project
  • The HDF-SRB model
  • SRB Support in HDFView

HDF & HDF-EOS Workshop IX

overview of srb what is srb
Overview of SRBWhat is SRB
  • The SDSC Storage Resource Broker (SRB) is client-server middleware that provides a uniform interface for connecting to distributed data storage in multiple types of storage resources.

HDF & HDF-EOS Workshop IX

overview of srb architecture
Overview of SRBArchitecture

SRB Client

MCAT

SRB Server

HPSS

Unitree

DB2

ObjStore

HDF5

FTP

Distributed Storage Resources: database system, archival storage system, file system, ftp

HDF & HDF-EOS Workshop IX

project description motivation
Project DescriptionMotivation

High performance distributed data system

HDF & HDF-EOS Workshop IX

remote data access on srb methods
Remote Data Access on SRBMethods
  • Normal ways to access SRB:
    • Get the whole file: large files (100TB SCEC)
    • Use POSIX low level calls: low performance
  • New way:
    • Implement proxy operations to access objects or parts of objects in one request

HDF & HDF-EOS Workshop IX

project description goals
Project DescriptionGoals
  • Use SRB as middleware to transfer data between the server and client
  • Use Object-level access for interactive and efficient access to part of the file

Working prototype of client/server system for

object-level access to HDF5 stored in the SRB

HDF & HDF-EOS Workshop IX

normal srb file access architecture

HDF5

Normal SRB File AccessArchitecture

client

HDF5 File

(whole file or a sequence of bytes)

SRB Server

MCAT

HDF & HDF-EOS Workshop IX

object level file access architecture

HDF5

Object-level File AccessArchitecture

client

server

HDF5 Library

HDF5 Object

(File, Group, Dataset, Subset, Attribute)

HDF5 Object

(File, Group, Dataset, Subset, Attribute)

SRB Server

HDF5-SRB Module

(pack/unpack messages)

HDF5-SRB Module

(pack/unpack messages)

MCAT

HDF & HDF-EOS Workshop IX

examples of file access

HDF5

Examples of File Access

I need to see the eye of Hurricane Bob!

HDF & HDF-EOS Workshop IX

examples of file access whole file transfer

HDF5

Examples of File AccessWhole file transfer

Get the file

HDF & HDF-EOS Workshop IX

examples of file access whole file transfer1

HDF5

HDF5

HDF5

HDF5

Examples of File AccessWhole file transfer

Transfer large image – slow!

HDF & HDF-EOS Workshop IX

examples of file access srb posix api
Examples of File AccessSRB POSIX API

I need to see the eye of Hurricane Bob!

HDF5

HDF & HDF-EOS Workshop IX

examples of file access srb posix api1
Examples of File AccessSRB POSIX API

Open file

file’s open

find image

image found

open image

image open

HDF5

Many small messages – slow and complex!

HDF & HDF-EOS Workshop IX

examples of file access object level

HDF5

Examples of File AccessObject level

Get me the eye of hurricane Bob

HDF & HDF-EOS Workshop IX

examples of file access object level1

HDF5

Examples of File AccessObject level

Get me the eye of hurricane Bob

One request & small image – fast & simple!

HDF & HDF-EOS Workshop IX

hdf5 srb model new objects apis
HDF5-SRB Model New objects/APIs
  • A new set data objects
    • H5File, H5Group, H5Dataset, H5Datatype, etc
    • Encapsulated client requests and server results
  • Enhanced SRB APIs
    • Pack/Unpack routines (exchange data between byte stream and structure) to handle complicated struct – string, pointers, pointers to arrays, arrays of pointers, etc
    • New srbGenProxyFunct (general Proxy Function) handles other types of request besides HDF5

HDF & HDF-EOS Workshop IX

hdf5 srb model data flow
HDF5-SRB Model Data Flow

Client API

srbObjRequest(void *obj, int objID)

Server API

srbObjProcess(void *obj, int objID)

5. H5Object

3. H5Obj::op()

7. unpackMsg()

6. packMsg()

1. packMsg()

2. unpackMsg()

HDF5 Library

4. Access file

srbGenProxyFunct

HDF5 file

SRB Server

HDF & HDF-EOS Workshop IX

runing server client
Runing Server/Client
  • A SRB server that supports HDF5
    • HDF5 library and other external libraries (SZIP, ZLIB)
    • A SRB version 3.4 or later from http://www.sdsc.edu/srb/
    • Follow instruction on how to run SRB server from UG packed with SRB source release or online at http://hdf.ncsa.uiuc.edu/hdf-srb-html/HDF-SRB-UG.html
  • Any client application that implements HDF5-SRB Objects
    • No HDF5 library is required on the client
    • Example client application: HDFView 2.3 or above

HDF & HDF-EOS Workshop IX

short demo hdfview
Short DemoHDFView
  • Support Windows and Linux
  • May support SGI, AIX if there is new funding

HDF & HDF-EOS Workshop IX

future work
Future Work?
  • Writing capabilities: the current HDF-SRB implementation only supports read-only operations. We propose to add write functionality so that users will be able to create new files and data objects, and to modify data content and attributes.
  • Better support for complex datatypes such as compound datatype and variable length datatype
  • Support for HDF5 indexing and ingesting HDF5 metadata, which will enable users to access HDF5 objects directly through MCAT
  • Support for files across different servers (or SRB federation)
  • More features for HDFView: compared to its local file-handling capabilities, the current version of HDFView has very limited features for remote file access on an SRB server.

HDF & HDF-EOS Workshop IX

question comments
Question / Comments?

HDF & HDF-EOS Workshop IX