1 / 35

Federating Archives in the DELAMAN Network

Federating Archives in the DELAMAN Network. Storage Resource Broker. Reagan W. Moore San Diego Supercomputer Center moore@sdsc.edu http://www.npaci.edu/DICE/SRB. Distributed Data Management Using Data Grids. Build a shared collection Authenticate users independently of the storage systems

cortez
Download Presentation

Federating Archives in the DELAMAN Network

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Federating Archives in the DELAMAN Network Storage Resource Broker Reagan W. Moore San Diego Supercomputer Center moore@sdsc.edu http://www.npaci.edu/DICE/SRB

  2. Distributed Data Management Using Data Grids • Build a shared collection • Authenticate users independently of the storage systems • Control access independently of the storage systems • Organize the file name space independently of the storage systems • Manage context (metadata) independently of content (files) • Maintain consistency between context and operations on content

  3. Storage Resource Broker • Generic distributed data management technology • Data grids - sharing • Digital libraries - publication • Persistent archives - preservation • Federated server architecture / thin client • 250,000 lines of “C” code • Supports all major compute and storage platforms • All requirements listed on following Scenario slides are supported

  4. Scenario 1- Data Migration • Provide URIDs (logical file names) that are independent of storage system • Provide metadata for each file • Support browse and discovery on collection hierarchy • Support access interfaces to the data • Support registration of existing files into a shared collection • Single sign-on environment • GSI / challenge response / tickets

  5. Managing Distributed Data Data Access Methods (Web Browser, DSpace, OAI-PMH) • Storage Repository • Storage location • User name • File name • File context (creation date,…) • Access constraints Naming conventions provided by storage systems

  6. Data Grids Provide a Level of Indirection for Each Naming Convention Data Access Methods (C library, Unix, Web Browser) Data Collection • Storage Repository • Storage location • User name • File name • File context (creation date,…) • Access constraints • Data Grid • Logical resource name space • Logical user name space • Logical file name space (URID) • Logical context (metadata) • Control/consistency constraints Data is organized as a shared collection

  7. Provide Context for Data • Properties of files • Provenance - source • Descriptive attributes • State information resulting from operations on files • Organize properties as metadata in a collection hierarchy • Define operations on file properties • Manage state information - location, replicas, containers, checksums • Separate context management from content management • Maintain consistency of context as operations are done on content • Support context management • Schema extension, automated SQL generation, bulk metadata load • Metadata extraction through a remote procedure parsing the file

  8. Federated Server Architecture Peer-to-peer Brokering Read Application Parallel Data Access Logical Name Or Attribute Condition 1 6 5/6 SRB server SRB server 3 4 5 SRB agent SRB agent 2 Server(s) Spawning R1 MCAT 1.Logical-to-Physical mapping 2.Identification of Replicas 3.Access & Audit Control R2 Data Access

  9. C, C++, Java Libraries Unix Shell Databases DB2, Oracle, Sybase, SQLserver, Postgres, mySQL, Informix File Systems Unix, NT, Mac OSX Archives - Tape, Sam-QFS, DMF, HPSS, ADSM, UniTree, ADS Storage Resource Broker - Data Grid Application HTTP DSpace OpenDAP OAI, WSDL, WSRF DLL / Python, Perl Linux I/O Java, NT Browser Kepler Actors Federation Management Consistency & Metadata Management / Authorization,Authentication,Audit Latency Management Metadata Transport Logical Name Space Data Transport Catalog Abstraction Storage Repository Virtualization Databases DB2, Oracle, Sybase, Postgres, mySQL, Informix ORB

  10. Scenario 2 - Data Exchange • Support access controls on the URIDs • Java administration GUI to support owner control of access controls • Can delegate permission to set access controls • Access controls apply on all replicas independent of storage system • Support latency management for moving files across wide area networks • Parallel I/O, replication, staging, aggregation of data / metadata / I/O commands • Support integrity validation • Manage checksums for each file

  11. Latency Management -Bulk Operations • Bulk register • Create a logical name for a file • Bulk load • Create a copy of the file on a data grid storage repository • Bulk unload • Provide containers to hold small files and pointers to each file location • Bulk delete • Mark as deleted in metadata catalog • After specified interval, delete file • Bulk metadata load • Support parsing of metadata from a remote file at remote storage • Requests for bulk operations for access control setting, …

  12. Scenario 3 - Community Access • Within the shared collection, the digital entities are owned and managed by the data grid • Files, URLs, SQL commands, database binary large objects can be registered into the shared collection • Access controls for • Files / metadata / storage systems • Access controls are defined for multiple roles • Schema extension, create new metadata • Modify metadata • Add annotations • Turn on audit trails • Write data • Read data

  13. Scenario 4 - Explorative Studies • Uniform access mechanisms to data across all storage systems • Support for queries on databases • Support for formatting results (XML, HTML) • Support audit trails, encryption • Support user-defined collection hierarchy • Soft links (build a logical collection of pointers to data within the data grid) • Support for multiple types of discovery • By URID (Logical File Name) • By query on metadata (may be unique to a single file) • By GUID (handle system)

  14. Scenario 5 - Education • SRB is used to build digital libraries • Assemble class material • Manage student reports • Display material through web browsers • Federation of digital libraries • Controlled sharing across independent data grids or digital libraries • Support for cross-registration of logical name spaces • Authentication done by “home” data grid • Access controls managed by both data grids

  15. Federation Data Access Methods (Web Browser, DSpace, OAI-PMH) Data Collection A Data Collection B • Data Grid • Logical resource name space • Logical user name space • Logical file name space • Logical context (metadata) • Control/consistency constraints • Data Grid • Logical resource name space • Logical user name space • Logical file name space • Logical context (metadata) • Control/consistency constraints Access controls and consistency constraints on cross registration of digital entities

  16. Scenario 6 - Updating Resources • Maintain system level metadata • Owner of registered file • Creation time, modification time, size, audit trails • Replica locations • Support for synchronization of replicas • Can modify a replica, subsequent reads are to the modified copy • Can synchronize copies to the modified version • Support for physical file containers • Aggregate small files before storage

  17. Scenario 7 - Web-based Editions • Support for digital library interfaces on top of the data grid • Transana - technology to manipulate, edit, and manage classroom video (University of Wisconsin) • DSpace - digital library system to manage ingestion of material into a collection • OAI-PMH - Open Archives Initiative protocol for metadata harvesting • OpenDAP - Data Access Protocol that supports both semantic and structural manipulation of registered files • Windows browser, Web browser, Java, WSDL interfaces • Collaborating on development of portlet interface

  18. C, C++, Java Libraries Unix Shell Databases DB2, Oracle, Sybase, SQLserver,Postgres, mySQL, Informix File Systems Unix, NT, Mac OSX Archives - Tape, Sam-QFS, DMF, HPSS, ADSM, UniTree, ADS Storage Resource Broker - Data Grid Application HTTP DSpace OpenDAP OAI, WSDL, WSRF DLL / Python, Perl Linux I/O Java, NT Browser Kepler Actors Federation Management Consistency & Metadata Management / Authorization,Authentication,Audit Latency Management Metadata Transport Logical Name Space Data Transport Catalog Abstraction Storage Repository Virtualization Databases DB2, Oracle, Sybase, Postgres, mySQL, Informix ORB

  19. Scenario 8 - Unconnected Editions • Ability to download data from shared collection to local resource • Support for PCs, workstations, supercomputers • Generalization of anonymous FTP • Can issue a ticket permitting • Limited number of read accesses valid for specified time interval • Can set public access to a sub-collection • Can restrict access by user name/domain/zone

  20. Local Archives • Maintain files in local file system • Register existence of the files into the data grid • Issue synchronization command to replicate into the archive • Maintain a data grid on the local system • Entire environment can be installed on a Mac in 15 minutes (Perl install script) • Use data grid federation to synchronize name spaces, files, metadata from local data grid to archives data grid

  21. Scenario 9 - Collaborative Commmentary • Comments can be added by owner • Annotations can be added by authorized persons • Annotations marked by person name, date • Can restrict annotation right by group • Can choose to create explicit metadata attributes to manage comments • Can store multiple comments per object • Can search across metadata • Or can use digital library interfaces to manage comments

  22. Sites Using the SRB

  23. Generic Infrastructure • SDSC developed the Storage Resource Broker (SRB) to support access to distributed data • Effort started in 1996 as a DARPA funded project • Now support over 30 national/international projects • Development team of 12 staff is led by • Michael Wan, data management systems • Arcot Rajasekar , information management systems

  24. SDSC SRB Team (left to right) • Arun Jagatheesan • George Kremenek • Sheau-Yen Chen • Arcot Rajasekar (SRB development lead) • Reagan Moore (SRB PI) • Michael Wan (SRB architect) • Roman Olschanowsky (BIRN) • Bing Zhu • Charlie Cowart • Lucas Gilbert • Tim Warnock • Wayne Schroeder (SRB product) • Adam Birnbaum (SRB production) • Antoine De Torcy • Vicky Rowley (BIRN) • Marcio Faerman (SCEC) • Students & emeritus • Erik Vandekieft • Reena Mathew • Xi (Cynthia) Sheng • Allen Ding • Grace Lin • Qiao Xin • Daniel Moore • Ethan Chen • Jon Weinburg • Supported by overt 20 projects (NSF, DOE, NASA, NARA, NIH, LOC, NHPRC)

  25. Data Grid Capabilities • Data manipulation • Containers • Parallel I/O • Firewall interactions • Resource interactions • Fault tolerance • Load leveling • Replication • HIPAA security requirements • Authentication of all users • Access controls on data and metadata • Audit trails • Data encryption • Centralized control • Application interfaces • C library, Shell commands, Java, Perl, Python, WSDL, workflow

  26. Data Management System Features • Data grid for managing distributed data • Latency management for bulk analyses of collections • Infrastructure independent name spaces for describing data, resources, users, and state information • Digital library for managing data context • Curation services for managing collections • Descriptive metadata for discovery • Persistent archive to manage technology evolution • Interoperability mechanisms between heterogeneous storage systems and user access mechanisms

  27. BIRN - Biomedical Informatics Research Network Data Grid Wash U. Duke NIH/NCRR Centers for Imaging and Computing Cal Tech Harvard NPACI/ SDSC Cal-(IT)2 “Surface Web” “Deep Web” Duke UCLA Integrating Cyber Infrastructure to Link: •Advanced Imaging Instruments •Data Intensive Computing •Multi-Scale Brain Databases Wireless “Pad” Web Interface

  28. Digital Library • Collection hierarchy for organizing data • User-defined metadata • Collection level metadata • Metadata manipulation • Schema extension • Bulk metadata processing • Queries on metadata • Access controls on metadata • Views on collections • Digital library APIs • DSpace, Fedora, OAI-PMH, web browsers • METS metadata XML schema

  29. Select Receiver (Lat/Lon) Output Time History Seismograms Select Scenario Fault Model Source Model SCEC Community Library Southern California Earthquake Center Store seismic data • Managing over 90 TBs, over 1.7 million files • Store community models for seismic velocity • Data distributed between USC, SDSC SCEC community digital library • Storage Resource Broker data grid technology • NMI portal interface • Digital library services to display seismograms • Visualizations of seismic waves at the surface • Visualization of seismic wave propagation through the volume

  30. National Virtual Observatory Virtual Observatory Architecture Discover Compute Publish Collaborate Portals, User Interfaces, Tools • Provide access to large star catalogs and large image sky surveys • 2MASS • SDSS • DPOSS • USNO-B • Macho Topcat VOPlot SkyQuery OASIS Mirage DIS Aladin conVOT interfaces to data Registry Layer Data Services Compute Services HTTP Services SOAP Services Grid Servicesstateless, registered self-describing persistent, authenticated Semantics (UCD) crossmatch visualization Bulk Access ADS image data mining OAI source detection OpenSkyQuery SIAP, SSAP VOTable FITS, GIF,… Virtual Data Digital LibraryOther registriesXML, DC, METS Workflow (pipelines) Authentication & Authorization Existing Data Centers My Space storage services Grid MiddlewareSRB, Globus, OGSASOAP, GridFTP Databases, Persistency, Replication Disks, Tapes, CPUs, Fiber

  31. National Science Digital Library Web Interface to Persistent Archive • Preserve educational material that has been registered into a central repository at Cornell through URLs • Crawl web and retrieve material, 10 levels of indirection • Convert internal URLs into data grid handles • Aggregate files into containers for storage • Preserve using SRB data grid technology • Currently housing over 26 million files

  32. NARA U Md SDSC MCAT MCAT MCAT Principle copy stored at NARA with complete metadata catalog Replicated copy at U Md for improved access, load balancing and disaster recovery Deep Archive at SDSC, no user access, but complete copy National Archives and Records Administration - Research Prototype Persistent Archive • Demonstrate preservation environment • Authenticity • Integrity • Management of • technology evolution • Mitigation of risk of data loss • Replication of data • Federation of catalogs • Management of preservation • metadata • Scalability • EAP collection • 350,000 files • 1.2 TBs in size Federation of Three Independent Data Grids

  33. For More Information Reagan W. Moore San Diego Supercomputer Center moore@sdsc.edu http://www.npaci.edu/DICE http://www.npaci.edu/DICE/SRB http://www.npaci.edu/dice/srb/mySRB/mySRB.html

More Related