1 / 14

Managing Metadata In SRB Arcot (Raja) Rajasekar

Managing Metadata In SRB Arcot (Raja) Rajasekar. Storage Resource Broker. SRB Space. Metadata in Data Grids. Infrastructure independence Enable distribution of data across multiple storage systems Provide essential capabilities needed for a scalable data management system

chione
Download Presentation

Managing Metadata In SRB Arcot (Raja) Rajasekar

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Managing MetadataIn SRBArcot (Raja) Rajasekar Storage Resource Broker SRB Space

  2. Metadata in Data Grids • Infrastructure independence • Enable distribution of data across multiple storage systems • Provide essential capabilities needed for a scalable data management system • Administrative metadata • Logical Name Spaces • Provides management needed for resources, users, files • Collection-based data management • Discovery through Attributes/Schema • User-defined metadata based discovery • Beyond textual search

  3. Metadata & SRB DB • SRB-aware Metadata • Metadata in external databases • Database access independent of SRB • SRB-aware DB Schema • Has columns containing SRB Object URL/PathNames • Values can be used to create SRB Object URL/PathNames • Application queries DB then accesses data in SRB • SRB-integrated Metadata • Non-MCAT database • Accessible through SRB’s DAI interface • SRB-aware DB Schema • SRB Object URL/PathNames • SRB access-controlled DB • No native DB interface needed by Application • MCAT-integrated Metadata • SRB Object access integrated with Metadata query APP SRB FS SRB DB APP MCAT FS

  4. MCAT-integrated Metadata • Queriable System Metadata • Owner, FileSize, DataType, Comments, CreationTime, ResourceType, Location, … • Free-form Metadata (User-defined) • Attribute-Value-Unit Triplets • No limits on number of Triplets • Metadata for Data, Collection, User, Resource • Supports Attribute-list Schemas • Useful for Interactive Data Ingestion & Querying • Dublin Core, EcoCore, Email, … • Extensible Schema Metadata • User Tables integrated into MCAT Core Schema • Logical Schema (aka view) exposed to user • No limit on number of logical schemas • Can query on subset of logical schemas

  5. Extensible MCAT Schema ExtDB-1 Schema MCAT Schema ExtDB-5 Schema ExtDB-3 Schema ExtDB-2 Schema ExtDB-4 Schema SRB Client MCAT Server External DB External DB MCAT DB External DB External DB External DB SRB Server An extensible Universal Schema is exposed to the MCAT and User SRB Server SRB Servers ExtDB-6 Schema SRB Servers Local Disk HPSS Others Object Ring Buffer

  6. How to Query & Use System Metadata? • Query Commands - each command has multiple options • Data: SgetD, Sls, Slscont, SmodD • Container: SgetColl, SmodColl • Resource: SgetR, SmodR • User: SgetU, SmodifyUser • Others: SgetT, Schksum, Stoken • Integrated Query and Data Access • Scat, Sget • Scat “S*.c&CTIME > ‘2005-07-26-15.00’ “ • Sget –A “CTIME > ‘2005-07-26-15.00’ “ • Scat “S*.c&CTIME > ‘2005-07-26-15.00’&VNUM>=‘2’”

  7. How to Query & Use Freeform Metadata? • Query Commands • Smeta – depregated • Sufmeta – user-friendly • List Files: Sufmeta -Q alpha = 200 beta = bar • List Collections: Sufmeta -Q –c title like ‘*srb*’ • Insert/Modify Commands • File Metadata:Sufmeta alpha 200 foo.dat • Coll Metadata:Sufmeta -c beta 200 myColl • Bulk Metadata:Sufmeta –D -f mdFile '|' (multiple srbObj) Sufmeta -f mdFile2 '|' foo.dat

  8. How to Query & Use Freeform Metadata? • Integrated Query & Data Access • SgetD -A "ATTRCONDD color = red" '*' • this queries for data with userdefined attribute-value pairs the form is <AttrName> <Operation> <Value> with spaces between them • SgetD -A "ATTRCONDD color = red &ATTRCONDD shape like box*“ '*‘ • can use conditions on ATTRCONDD upto 5 times • SgetD -A "ATTRCONDC author = fred & ATTRCONDD color = red” '*‘ • Querying for multiple conditions on same attribute • SgetD -A "SIZE <15000 && > 10000 " '*' • SgetD -A "RESOURCE like '*raja*' || like '*du*‘ " '*'

  9. Query Attributes • DATA name of the object • COPY replica number of the object • GUID GUID value of the object. This is a string • DTYPE data type of the object • COLLECTION collection where the object can be found • RESOURCE resource name where the object resides • RTYPE resource type where the object resides • NETADDR net address of the resource • PATH physical path name of the object • DCOMMENTS comments on the object • DTIME time of last modification of the object • DATA_CHECKSUM checksum of the object • USER user name • OWNER owner of the object • TICKET ticket issued on the object • SIZE size of the object • CTIME creation timestamp of the object • DIRTY dirty-flag of the object • VNUM version string (even though called number) of the object • ATTRCONDD user-defined attribute-condition on the object (5) • ATTRCONDC user-defined attribute-condition on the collection (5) • ATTRCONDU user-defined attribute-condition on users (4) • ATTRCONDR user-defined attribute-condition on the resource (4) • And any other of the defined attributes…..

  10. How to Use Extensible Metadata? • Query Commands • Query for Schema Attribute Names • Squery -E -L 420:mets.creator -> sekar.me_core.author 421:mets.data_id -> sekar.me_core.data_number 422:mets.object_name -> sekar.me_core.data_name 423:roadnet.data_id -> srb.sensor.data_id 424:roadnet.location_lat -> srb.sensor.location_lat 425:roadnet.location_lon -> srb.sensor.location_lon E: show extensible attributes; L: show their physical counterparts Internal ID External Schema Name Database Schema/DBLink Name Database Attribute Name External Attribute Name Database Table Name

  11. How to Query Extensible Metadata? • Query Commands • Query for Metadata • Find all values of latitudes in the sensor schema? Squery -S roadnet.location_lat • Find all locations where wind sensors are located? Squery -S roadnet.location_lat -S roadnet.location_lon roadnet.sensor_type like '*wind*' • Query for Discovery of Data • Find all sensor-streams that are related to wind sensors? Squery -S DATA_NAME roadnet.sensor_type like '*wind*’ DATA_TYPE = ‘sensor stream’ • Find all files related to a particular location? Squery -S DATA_NAME roadnet.location_lat = ’70N’ roadnet.location_lon = ’35E’

  12. How to Modify Extensible Metadata? Object Name Collection Name • Insertion into a Table • SmodE -i Spwd.c /home/sekar.sdsc/mytest srb.validity 'sensor_name|start_date|end_date|comments' "'SCA'|'2004-02-10-10.30'|'2004-02-22-10.20‘ |'internal software error'“ • Deletion from a Table • SmodE -d Spwd.c /home/sekar.sdsc/mytest srb.validity sensor_name "= 'LAB'" Attribute Values Within “” Attribute Name List Table Name Deletion Condition

  13. For More Information • Man Pages: • Shelp • Sattrs • Sufmeta (Smeta being deprecated) • Squery • Stoken • Sget, Sput, SgetD, SgetColl, SgetR, SgetU, SgetT • SmodD, SmodColl, SmodR, SmodifyUser, Sticket • Plus other Man Pages • ReadMe Files: • README.MCAT.ATTRIBUTES • README.extensibleschema

  14. Attribution SRB Team & Collaborators srb@sdsc.edu srb-chat@sdsc.edu http://www.sdsc.edu/srb/ Zone A Zone B Trust Relation SRB SRB Duke MGH UCLA DB DB MCAT Trust Relation Trust Relation Storage Resource Broker SRB Zone C DB SDSC BWH NCMIR

More Related