1 / 24

Introduction to Apache OODT

Introduction to Apache OODT. Yang Li Mar 9, 2012. What is OODT. Object Oriented Data Technology Science data management Archiving Systems that span scientific disciplines

bendek
Download Presentation

Introduction to Apache OODT

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Introduction to Apache OODT Yang Li Mar 9, 2012

  2. What is OODT • Object Oriented Data Technology • Science data management • Archiving Systems that span scientific disciplines • Enable interoperability among data agnostic systems (astrophysics, planetary, space science data systems, open source web analytics)

  3. History • 2001 • deployed to make virtual specimen bank for Early Detection Research Network (oncology) • 2004 • Core architectural software of Planetary Data System Data Distribution deployed by NASA (planetary science) • 2007 • deployed for the Orbiting Carbon Observatory and Seawindsmissions (earth science) • 2008 • deployed in for National Polar-Orbiting Environmental Satellite System (atmospheric science)

  4. Framework • Catalog & Archive • Utilities • Grid • Agility

  5. Catalog & Archive • Deal with large-scale ingest of data, metadata extraction of data, post-processing of data into derived and higher-order products, cataloging of data, searching of catalogs, versioning, and retrieval • Components: • Catalog, Crawling framework, Curation, File manager, Metadata, PCS, Push/Pull framework, Resource management, Workflow, CAS install, Web apps

  6. Catalog • Virtualize underlying catalogs for use in the CAS system • Heterogeneous catalog models are mapped to a common dictionary, and then integrated locally so that they may be queried across and ingested into

  7. CAS Crawler • Standardize the common ingestion activities • identification of files and directories to crawl • satisfaction of ingestion pre-conditions • metadata extraction • Ingestion

  8. CAS Crawler

  9. Curation • A web application for managing policy for products and files and metadata that have been ingested via the CAS component • Use a servlet container to deploy the web app • Staging area • Directories on local machine holding data products • Metadata generation area • Create metadata files to associate with data products

  10. File Manager • Provide everything to catalog, archive and manage files, and directories, and their associated metadata • Separate data stores and metadata stores as standard interfaces

  11. Workflow • Provides everything to execute workflows, and science processing pipelines. • Separate workflow repositories and workflow engines as standard interfaces

  12. Resource Management • Job management • Execution, monitoring, traking • Underlying software system and hardware resources • e.g. disk space, computational resources, and shared identity

  13. Resource Management (Cont) • Critical objects • Job, Job Input, Job Spec, Job Instance, Resource Node

  14. Metadata • A Multi-valued, generic Metadata container class • Internal map of string keys pointing to vectors of strings • [std:string key]⇒std:vector of std:strings

  15. Framework • Catalog & Archive • Common Utilities • Grid • Agility

  16. Common Utilities • Provide needed support for catalogs, archives, and grids • Query Expression • Platform neutral and extensible way of posing questions • Single Sign On • Commons • Lots of miscellaneous utilities, including I/O streams, logging, XML, and more

  17. Query Expression • Provide a way to express queries in a generic manner • Use boolean postfix expressions to capture the domain, range, and constraint of a query, regardless of the source of the query • Encapsulate the results of a query • standard way to pass a query and its results between servers, clients, nodes, and other components.

  18. Framework • Catalog & Archive • Utilities • Grid • Agility

  19. Grid • Profile (metadata) and Product (data) services • Product • Retrieves resources (products) in platform-neutral formats • Profile • Describes and discovers resources using extensible metadata called "profiles" • Web Grid • provides profile and product services over a REST-ful interface. • XML Product/Profile handlers • provides XML-configurable, Database profile and product handlers.

  20. Product • Provide access to data products • datasets, images, documents, or anything with an electronic representation • Accept standard query expressions and return zero or more matching products • Transform products from proprietary formats and into Internet standard formats without impacting local stores or operations.

  21. Profile • Describes and Locates resources using metadata descriptions • resource's inception, composition, and location • Catalogs metadata descriptions and provides creating, updating, and querying capabilities.

  22. Framework • Catalog & Archive • Utilities • Grid • Agility

  23. Agility • Re-implementation of Grid in Python with a focus on high performance in the face of gargantuan data sets as well as accelerated development and integration into existing systems.

  24. Questions

More Related