1 / 18

InterLib(-Related) Activities at SDSC/DICE

InterLib(-Related) Activities at SDSC/DICE. IBM HPSS (Storage/Archival, e.g. ADL) SDSC SRB/(E)MCAT (Data Handling/Information Discovery) AMICO Image Collection (CDL Testbed) Excelon as XML Data Server

lan
Download Presentation

InterLib(-Related) Activities at SDSC/DICE

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. InterLib(-Related) Activities at SDSC/DICE • IBM HPSS (Storage/Archival, e.g. ADL) • SDSC SRB/(E)MCAT (Data Handling/Information Discovery) • AMICO Image Collection (CDL Testbed) • Excelon as XML Data Server • MIX: Mediation of Information using XML (with DB-Lab UCSD) Bertram Ludaescher ludaesch@sdsc.edu

  2. HPSS, SRB, MCAT • HPSS: Storage/Archival of large datasets • (UCB, UCSB, Stanford) • SRB/(E)MCAT: Data Handling/Information Discovery • transparent access to remote storage • replication • containers for large number of small items • caching • authorization • proxy operation support (filtering, data subsetting) • usage of security infrastructure (GSI)

  3. SRB Interface Application Application MCAT Core SRB Master SRB Agent SRB Server MCAT Dublin Core Eco Core SRB Server SRB Server

  4. Managing Metadata: EMCAT • Extensible Meta Data Catalog - EMCAT • Exploits dependencies & relationships (m:n, tc, <=>, …) • T-Language - Markup, Filter & Presentation • Meta Data Repository (Object-, System-, Collection-level) • Based on Kernel Meta Meta Data • Extensible • Uniform Access and Federation interface • Metadata exchange Interface Protocol • MAPS- Meta data Attribute Presentation Structure • query, update and result structures • Close to Z39.50

  5. SRB/MCAT Future • Performance Improvements and Consolidation • Delayed Action Manager - mirror, cronjobs • Support for Methods • Handling Very Large Data sets - partitions • More Drivers - Sybase, NTFS, LDAP • Extensible MCAT • Language Support - Perl, Fortran http://www.npaci.edu/DICE/SRB

  6. The AMICO Digital Library Projecthttp://www.amico.orghttp://www.npaci.edu/DICE/AMICOArt Museum Image Consortium Richard Marciano et. al. 55,146 objects750 MB 53,763 thumbnail images319 MB 57,609 full tiff images180 GB

  7. AMICO Consortium of 26 (now 31) museums AGO_ Art Gallery of Ontario AIC_ Art Institute of Chicago AKAG Albright-Knox Art Gallery, Buffalo, NY ASIA Asia Society BMFA Boston Museum of Fine Arts CCP_ Center for Creative Photography, U. Arizona CMA_ The Cleveland Museum of Art DMCC Davis Museum and Cultural Center, Wellesley College, MA FASF Fine Arts Museums of San Francisco GEH_ George Eastman House, Rochester, NY JPGM J. Paul Getty Museum, Los Angeles, CA LACM Los Angeles County Museum of Art LOC_ Library of Congress MACM Musée d'art contemporain de Montréal MBAM Musée des beaux-arts de Montréal MCAS Museum of Contemporary Art, San Diego MIA_ The Minneapolis Institute of Arts MMA_ The Metropolitan Museum of Art NGC_ National Gallery of Canada, Ottawa/Ontario NMAA National Museum of American Art, Smithsonian Institution PMA_ Philadelphia Museum of Art SFMO San Francisco Museum of Modern Art SJMA San Jose Museum of Art TFC_ The Frick Collection, NY WAC_ Walker Art Center, Minneapolis, MN WMAA Whitney Museum of American Art, NY

  8. Raw Metadata Structure - catdata:8 files 16,604 year1.d990429 14,430 year1.d990512 22,938 year1.d990520 54,303 year1.d990627 15 year1.d990708 54,298 year1.d990731 93 year1.d990806 657 year1.d990813 - tiffmetadata:23 files 2963 AGO_.tiffmetadata.txt 1016 AIC_.tiffmetadata.txt 894 AKAG.tiffmetadata.txt 187 ASIA.tiffmetadata.txt 7591 BMFA.tiffmetadata.txt 401 CCP_.tiffmetadata.txt 1455 CMA_.tiffmetadata.txt 56 DCMC.tiffmetadata.txt 470 DMCC.tiffmetadata.txt 10141 FASF.tiffmetadata.txt 2137 GEH_.tiffmetadata.txt 1459 JPGM.tiffmetadata.txt 1013 LACM.tiffmetadata.txt 20654 LOC_.tiffmetadata.txt 86 MACM.tiffmetadata.txt 50 MBAM.tiffmetadata.txt 31 MCAS.tiffmetadata.txt 1440 MIA_.tiffmetadata.txt 550 MMA_.tiffmetadata.txt 1507 NGC_.tiffmetadata.txt 1416 NMAA.tiffmetadata.txt 154 PMA_.tiffmetadata.txt 158 SFMO.tiffmetadata.txt 86 SJMA.tiffmetadata.txt 68 Such.tiffmetadata.txt 396 WAC_.tiffmetadata.txt 37069 replacements.txt 57499 replacements2.txt - thumbmeta: 52,689 files AGO_.1016.25_thum.met* AGO_.1016.32_thum.met* AGO_.1016.39_thum.met* …... WAC_.994C_thum.met WAC_.996C_thum.met WAC_.998C_thum.met WAC_.99C_thum.met* WMAA.1557_56_thum.met WMAA.31_426_thum.met

  9. AMICO Metadata Conversion Steps Tape Read “Raw” Metadata files: - catdata (8 files), - tiffmetada (23 files), - thumbmeta (52,689 files) Consolidated Metadata files: - 1 catdata - 1 tiffmetadata - 1 thumbmeta Merge Convert to XML Multiple XML files per museum 3 XML files: - 1 catdata - 1 tiffmetadata - 1 thumbmeta Split-by- museums 1 XML file per museum 1 XML file per museum Split-by- file size Multiple museum XML files per machine eXcelon Data Server eXcelon Data Server eXcelon Data Server Split-by- machines eXcelon Dump&Load Utility

  10. Data Server Fileserver 180GB RAID 180GB RAID 100Mbit Ethernet DB2 SRB HPSS HPSS Data Server Alternative System Architectures AMICO metadata server * eXcelon * Oracle 8i * DB2

  11. Current catalog metadata count(per museum)

  12. Average tiff size in MB(per museum)

  13. XQL Query Binder doc .xml Machine2 Machine1 Excelon Metadata Layout XMLStore Museum directories Museum1 Museum2 Museum-n File1.xml File2.xml

  14. Request for image (X.509) View based on AMICO DTD tif file SRB/MCAT HPSS MIX: Mediation of Information Using XML ...… for the AMICO CDL Prototype BBQ Interface (slide carousel interface) XMAS query XML doc MIXm engine Wrapper AMICO XML Database AMICO XML Database MARC Database XMAS: XML Matching and Structuring query language

  15. SDSC/DICE Discussion Topics • ADL: caching of HPSS data • ADEPT access to ADL for CDL testbed: SRB? • “Union Catalog”: • AMICO DTD <=XMAS=> MARC • SDLIP access to SRB/MCAT and MIX • Use of GINF (Stanford) • ...

More Related