Wp19 desy development plan
This presentation is the property of its rightful owner.
Sponsored Links
1 / 6

WP19 DESY Development Plan PowerPoint PPT Presentation


  • 55 Views
  • Uploaded on
  • Presentation posted in: General

WP19 DESY Development Plan. Frank Schlünzen Jürgen Starek. Object stores - Motivation. Data catalogues (DC) and digital objects (DO) are loosely coupled ACL changes in DC are not propagated to DO ( and vice versa ) No messaging between DC and DO

Download Presentation

WP19 DESY Development Plan

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Wp19 desy development plan

WP19 DESY Development Plan

Frank Schlünzen

Jürgen Starek


Object stores motivation

Objectstores - Motivation

  • Data catalogues (DC) and digital objects (DO) arelooselycoupled

  • ACL changes in DC are not propagatedto DO (andviceversa)

  • Nomessagingbetween DC and DO

    • Removal, corruption, update of a DO doesn‘tresult in DC notification

  • Norights-management on filesystemlayer

    • All filesownedby super-users

    • Needs totrustapplicationsforrightsmanagement

  • Synch

    • Status of DC and DO caneasilyget out ofsync

    • Metadata in DC and DO caneasilyget out ofsync

    • ACLs in DC and DO areprobablynever in synch

    • UIDs insideand outside of DC needsynch


  • Object stores motivation1

    Objectstores - Motivation

    • Objectstoreshavethe potential tosolvea numberofthatissues

    • Particularlyinterestingsystems (forus)

      • dCache

        • isclosetobeinga fullobjectstoreimplementation

        • Extendableforindexed, efficientlysearchable user-defined meta-dataforanyobject

        • Full, scalabletape backend

      • CEPH

        • veryinterestingsystemwithpotentially high efficiencyforsmallfiles

        • mightoffergoodperformance in combinationwithenhanced meta-dataandaclhandling

        • naturallychoiceforclouds (open nebula)

        • Mature: in linuxkernel


    Object stores proposal

    Objectstores - Proposal

    In order to supplement the existing in-house knowledge about the properties of different file systems and storage hardware solutions, we propose to survey existing and planned object store solutions, evaluating possible performance or maintenance advantages over classical POSIX file systems.

    • Survey Topics

      • Which Object Store systems are available as commercial products or free software, and which similar systems exist or are planned?

      • What methods to access data in these object stores exist?

      • Are there systems that go beyond POSIX I/O and POSIX rights management? How do they present these improved semantics to the user?

      • What are common designs and best practices in designing and operating object stores?

      • What is the I/O performance of these systems when storing differently-sized objects?

      • What is the best performing hardware setup for these systems?

      • Is there a quality-of-service-mechanism for data streams? Is there an I/O scheduler?

      • What metadata are supported by the systems?


    Virtual analysis plattform

    Virtual Analysis Plattform

    Remote access to compute resources for remote data analysis in a fully interactive and graphical environment is a common requirement at EuroFEL RIs.

    • Currently, this is realized through rather inconvenient and poorly scalable mechanisms.

    • Data protection and encapsulation of the environment is an issue.

    • We aim to evaluate some in-house available virtual/cloud resources and to set-up a prototype system for photon science applications:

      • provide a virtual host/environment tailored to a specific experiment and application, with a pre-defined software stack and data.

      • The user should have seamless access to his and only his data.

    • We will investigate possibilities to provide data access through different path-ways (e.g. ICAT, dCache, object store).

    • Possibilities to connect IdPs will be investigated as well.


    P hdf5 and fhgfs

    (p)HDF5 andFhGFS

    Visualization of individual slices of images within a HDF-container archived in ICAT is a common user requirement. Virtual appliances are one way to visualize data remotely, alternatively could be realized as web services:

    • Investigate h5ws, a globus based web service

    • Investigate paraview which offers webGL, vrml, js representations

    • if feasible – implement a prototype solution.

      Many photon-science applications are i/o rather than cpu-limited. pHDF5 is one way to accelerate parallel read/write access to HDF5-container. Currently a number of photon-science labs are supporting development of HDF5-accellerators by HDF.org.

    • test upcoming (p)HDF5 implementations on available parallel filesystems, in particular FhGFS.

    • realize a non-persistent FhGFS in distributed memory and test performance and stability of such a system.

    • Test FhGFS capabilities to create filesystems on demand and entirely in user space


  • Login