Npaci summer institute
Download
1 / 58

NPACI Summer Institute - PowerPoint PPT Presentation


  • 323 Views
  • Updated On :

NPACI Summer Institute. Data Management Introduction Real-life Experiences with Data Grids Reagan Moore Arcot Rajasekar (moore, sekar)@sdsc.edu. Summer Institute agenda Prepared presentations Preferred presentations on data access Your input Introduction to Data Intensive Computing

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'NPACI Summer Institute' - Patman


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Npaci summer institute l.jpg

NPACI Summer Institute

Data Management Introduction

Real-life Experiences with Data Grids

Reagan Moore

Arcot Rajasekar

(moore, sekar)@sdsc.edu


Topics l.jpg

Summer Institute agenda

Prepared presentations

Preferred presentations on data access

Your input

Introduction to Data Intensive Computing

Basic concepts

Topics


Presentations l.jpg

Monday

Introduction to data-intensive computing

SRB introduction and overview

mySRB web interface

inQ browser interface

Shell commands

Presentations


Presentations4 l.jpg

Tuesday

Building collections using the SRB

GridPort interface to the SRB - automated collection building

Databases - creating a catalog

Teragrid data model

Database federation

Presentations


Presentations5 l.jpg

Wednesday

Introduction to grid computing

MGRID - case study

Grid in 1 hour

Telescience portal

MCELL

Introduction to parallel I/O

NVO

APST tutorial

Presentations


Presentations6 l.jpg

Thursday

Data mining

Data visualization

Friday

Introduction to ROCKS

Presentations


Introduction to data intensive computing l.jpg

Survey of data management requirements

Data intensive computing

Common approach

Basic concepts

What’s real?

Production systems

Introduction to Data Intensive Computing


Basic tenet l.jpg

Data without a context is only useful to the owner

Context is used to automate discovery, access, and use

Data context can include a wide variety of types of information

Provenance

Descriptive

Data model

Structural

Authenticity

Basic Tenet


2nd basic tenet l.jpg

Fundamental concepts (computer science abstractions) underlie all of the data management requirements

The fundamental concepts can be implemented as generic software

2nd Basic Tenet


Data management requirements which do you need l.jpg

Data collecting underlie all of the data management requirements

Sensor systems, object ring buffers and portals

Data organization

Collections, manage data context

Data sharing

Data grids, manage heterogeneity

Data publication

Digital libraries, support discovery

Data preservation

Persistent archives, manage technology evolution

Data analysis

Processing pipelines, knowledge generation

Data Management RequirementsWhich do You Need?


Common infrastructure l.jpg

Demonstrate how real-time data management, digital libraries, persistent archives, and analysis pipelines can be built on data grids

Examine capabilities needed for each environment

Present real-world examples from multiple scientific disciplines

Common Infrastructure


Nsf infrastructure programs l.jpg

Partnership for Advanced Computational Infrastructure - PACI libraries, persistent archives, and analysis pipelines can be built on data grids

Data grid - Storage Resource Broker

Distributed Terascale Facility - DTF/ETF

Compute, storage, network resources

Digital Library Initiative, Phase II - DLI2

Publication, discovery, access

Information Technology Research projects - ITR

SCEC Southern California Earthquake Center

GEON GeoSciences Network

SEEK Science Environment for Ecological Knowledge

GriPhyN Grid Physics Network

NVO National Virtual Observatory

National Middleware Initiative - NMI

Hardening of grid technology (security, job execution, grid services)

National Science Digital Library - NSDL

Support for education curricula modules

NSF Infrastructure Programs


Federal infrastructure programs l.jpg

NASA libraries, persistent archives, and analysis pipelines can be built on data grids

Information Power Grid - IPG

Advanced Data Grid - ADG

Data Management System - Data Assimilation Office

Integration of DODS with Storage Resource Broker data grid

Earth Observing Satellite EOS data pools

Consortium of Earth Observing Satellites CEOS data grid

Library of Congress

National Digital Information Infrastructure and Preservation Program - NDIIPP

National Archives and Records Administration and National Historical Public Records Commission

Prototype persistent archives

NIH

Biomedical Informatics Research Network data grid

DOE

Particle Physics Data Grid

Federal Infrastructure Programs


Sdsc collaborations l.jpg

Hayden Planetarium Simulation & Visualization libraries, persistent archives, and analysis pipelines can be built on data grids

NVO -Digital Sky Project (NSF)

ASCI - Data Visualization Corridor (DOE)

Particle Physics Data Grid (DOE) {GrPhyN (NSF)}

Information Power Grid (NASA)

Biomedical Informatics Research Network (NIH)

Knowledge Network for BioComplexity (NSF)

Mol Science – JCSG, AfCS

Visual Embryo Project (NLM)

RoadNet (NSF)

Earth System Sciences – CEED, Bionome, SIO Explorer

Advanced Data Grid (NASA)

Hyper LTER

Grid Portal (NPACI)

Tera Scale Computing (NSF)

Long Term Archiving Project (NARA)

Education – Transana (NPACI)

NSDL – National Science Digital Library (NSF)

Digital Libraries – ADL, Stanford, UMichigan, UBerkeley, CDL

… 31 additional collaborations

SDSC Collaborations


Data grid concepts l.jpg

Logical name space libraries, persistent archives, and analysis pipelines can be built on data grids

Global persistent identifier

Storage repository abstraction

Standard operations supported on storage systems

Information repository abstraction

Standard operations to manage collections in databases

Access abstraction

Standard interface to support alternate APIs

Latency management mechanisms

Aggregation, parallel I/O, replication, caching

Security interoperability

GSSAPI, inter-realm authentication, collection-based authorization

Data Grid Concepts


Logical name space example hayden planetarium l.jpg

Generate “fly-through” of the evolution of the solar system

Access data distributed across multiple administration domains

Gigabyte files, total data size was 7 TBytes

Very tight production schedule - 3 months

Logical Name SpaceExample - Hayden Planetarium


Hayden data flow l.jpg
Hayden Data Flow system

NCSA

AMNH

NYC

SGI

NY

Production parameters, movies, images

2.5 TB

UniTree

data simulation

SDSC

CalTech

GPFS

7.5 TB

IBM SP2

BIRN

HPSS 7.5 TB

UVa

visualization


Logical name space l.jpg

Global, location-independent identifiers for digital entities

Organized as collection hierarchy

Attributes mapped to logical name space

Attributed managed in a database

Types of system metadata

Physical location of file

Owner, size, creation time, update time

Access controls

Logical Name Space


Identifiers l.jpg

Logical name entities

Global identifier for virtual organization

Unique identifier

Handle or OID unique across virtual organizations

Descriptive name

Descriptive attributes for discovery

Physical name

Physical entity name which varies between locations

Identifiers


Federated srb server model l.jpg
Federated SRB server model entities

Peer-to-peer Brokering

Read Application

Parallel Data Access

Logical Name

Or

Attribute Condition

1

6

5/6

SRB

server

SRB

server

3

4

5

SRB agent

SRB agent

2

Server(s) Spawning

R1

MCAT

1.Logical-to-Physical mapping

2.Identification of Replicas

3.Access & Audit Control

R2

Data Access


Storage repository abstraction l.jpg

Set of operations used to manipulate data entities

Manage data collections stored in

Archives (HPSS, UniTree, ADSM, DMF)

Hierarchical Resource Managers

Tapes, tape robots

File systems (Unix, Linux, Mac OS X, Windows)

FTP sites

Databases (Oracle, DB2, Postgres, SQLserver, Sybase, Informix)

Virtual Object Ring Buffers

Storage Repository Abstraction


Storage repository abstraction23 l.jpg

Byte level access entities

Unix semantics

Latency management

Bulk operations

Object oriented storage

Movement of application to the data

Access to heterogeneous systems

Protocol conversion

Storage Repository Abstraction


Byte level operations l.jpg

Unix File System operations entities

creat(), open(), close(), unlink()

read(), write(), seek(), sync()

stat(), fstat(), chmod()

mkdir(), rmdir(), opendir(), closedir(), readdir()

Application drivers

Management of file structure at remote site

Paging systems for visualization

Pre-fetch (partial file read)

Byte Level Operations


Hayden conclusions l.jpg

The SRB was used as a central repository for all original, processed or rendered data.

Location transparency crucial for data storage, data sharing and easy collaborations.

SRB successfully used for a commercial project in “impossible” production deadline situation dictated by marketing department.

Collaboration across sites made feasible with SRB

Hayden Conclusions


Latency management example asci doe l.jpg

Demonstrate the ability to load collections at terascale rates

Large number of digital entities

Terabyte sized data

Optimize interactions with the HPSS High Performance Storage System

Server-initiated I/O

Parallel I/O

Latency ManagementExample - ASCI - DOE


Latency management l.jpg

Bulk data load rates

Bulk data access

Bulk registration of files

Aggregation into a container

Extraction from a container

Staging, required by the Hierarchical Resource Manager

Status, required by the Hierarchical Resource Manager

Latency Management


Srb latency management l.jpg
SRB Latency Management rates

Remote Proxies,

Staging

Data Aggregation

Containers

Prefetch

Network

Destination

Network

Destination

Source

Caching

Client-initiated I/O

Streaming

Parallel I/O

Replication

Server-initiated I/O


Asci data flow l.jpg

Data movement across 3 hosts rates

ASCI Data Flow

applications

SRB server

SRB clients

data cache

local FS

SRB server

MCAT

Oracle

HPSS


Asci small files l.jpg

Ingest a very large number of small files into SRB rates

time consuming if the files are ingested one at a time

Bulk ingestion to improve performance

Ingestion broken down into two parts

the registration of files with MCAT

the I/O operations (file I/O and network data transfer)

Multi-threading was used for both the registration and I/O operations.

Sbload was created for this purpose.

reduced the ASCI benchmark time of ingesting ~2,100 files from ~2.5 hours to ~7 seconds.

ASCI Small Files


Latency management example digital sky project l.jpg

2MASS (2 Micron All Sky Survey): rates

Bruce Berriman, IPAC, Caltech; John Good, IPAC, Caltech, Wen-Piao Lee, IPAC, Caltech

NVO (National Virtual Observatory):

Tom Prince, Caltech, Roy Williams CACR, Caltech, John Good, IPAC, Caltech

SDSC – SRB :

Arcot Rajasekar, Mike Wan, George Kremenek, Reagan Moore

Latency ManagementExample - Digital Sky Project


Digital sky data ingestion l.jpg
Digital Sky Data Ingestion rates

SRB

SUN E10K

Data Cache

star catalog

Informix

SUN

HPSS

800 GB

….

input tapes from telescopes

10 TB

SDSC

IPAC CALTECH


Digital sky 2mass l.jpg

http://www.ipac.caltech.edu/2mass rates

The input data was originally written to DLT tapes in the order seen by the telescope

10 TBytes of data, 5 million files

Ingestion took nearly 1.5 years - almost continuous reading of tapes retrieved from a closet, one at a time

Images aggregated into 147,000 containers by SRB

Digital Sky - 2MASS


Containers l.jpg

Images sorted by spatial location rates

Retrieving one container accesses related images

Minimizes impact on archive name space

HPSS stores 680 Tbytes in 17 million files

Minimizes distribution of images across tapes

Bulk unload by transport of containers

Containers


Digital sky web based data retrieval l.jpg

average 3000 images a day rates

Digital Sky Web-based Data Retrieval

SRB

SUN E10K

Informix

SUNs

WEB

IPAC CALTECH

HPSS

800 GB

WEB

SUNs

SGIs

….

10 TB

JPL

SDSC


Moving application to the data l.jpg

Execution of defined operations directly at the storage system

Metadata extraction from files

Extraction of a file from a container

Validation of a digital signature

Data subsetting

Data filtering

Server initiated parallel I/O streams

Encryption as a property of the data file

Compression as a property of the data file

Moving Application to the Data


Remote proxies l.jpg

Extract image cutout from Digital Palomar Sky Survey system

Image size 1 Gbyte

Shipped image to server for extracting cutout took 2-4 minutes (5-10 Mbytes/sec)

Remote proxy performed cutout directly on storage repository

Extracted cutout by partial file reads

Image cutouts returned in 1-2 seconds

Remote proxies are a mechanism to aggregate I/O commands

Remote Proxies


Real time data example roadnet project l.jpg

Manage interactions with a virtual object ring buffer system

Demonstrate federation of ORBs

Demonstrate integration of archives, VORBs and file systems

Support queries on objects in VORBs

Real-Time DataExample - RoadNet Project


Heterogeneous systems l.jpg

Database blob access system

Database metadata access

Object ring buffer access

Archive access

Hierarchical Resource Manager access

http access

Preferred API - Python, Java, C library, Shell command, OAI, WSDL, OGSA, http, DLL

Heterogeneous Systems


Federated vorb operation l.jpg
Federated VORB Operation system

Logical Name of the Sensor

wiith Stream Characteristics

Automatically Contact

ORB2

Through VORB server

At Nome

Get Sensor Data

( from Boston)

VORB

server

1

VORB

agent

VORB

server

4

San Diego

Check ORB1

ORB1 is down

VORB

agent

3

2

6

ORB1

Nome

VCAT

5

Format Data and Transfer

Contact VORB Catalog:

1.Logical-to-Physical mapping

Physical Sensors Identified

2. Identification of Replicas

ORB1 and ORB2 are identified

as sources of reqd. data

3.Access & Audit Control

ORB2

R2

Check ORB2

ORB2 is up. Get Data


Slide42 l.jpg

Information Abstraction system

Example - Data Assimilation Office

HSI has implemented metadata schema in SRB/MCAT

Origin: host, path, owner, uid, gid, perm_mask, [times]

Ingestion: date, user, user_email, comment

Generation: creator (name, uid, user, gid), host (name, arch, OS name & flags), compiler (name, version, flags), library, code (name, version), accounting data

Data description: title, version, discipline, project, language, measurements, keywords, sensor, source, prod. status, temporal/spatial coverage, location, resolution, quality

Fully compatible with GCMD


Slide43 l.jpg

Data Management System: system

Software Architecture


Slide44 l.jpg

DODS Access system

Environment Integration


Peer to peer federated systems l.jpg

Consistency constraints in federations system

Cross-register a digital entity from one collection into another

Who controls access control lists?

Who updates the metadata?

Grid bricks versus tape archives

Persistent collections

Peer-to-Peer Federated Systems


Grid services l.jpg

Execution of a service creates state information system

Map state information onto logical name space

Associate state information with a digital entity

Manage state information in a registry

Consistency constraints on management of state information when changes are made

Grid Services


Slide47 l.jpg

C, C++, system

Libraries

Unix

Shell

Databases

DB2, Oracle,

Postgres

Archives

HPSS, ADSM,

UniTree, DMF

File Systems

Unix, NT,

Mac OSX

SDSC Storage Resource Broker

& Meta-data Catalog - Access Abstraction

Application

Linux

I/O

OAI

WSDL

Access

APIs

DLL /

Python

Java, NT

Browsers

GridFTP

Consistency Management / Authorization-Authentication

Prime

Server

Logical Name

Space

Latency

Management

Data

Transport

Metadata

Transport

Catalog Abstraction

Storage Abstraction

Databases

DB2, Oracle, Sybase,

SQLServer, Informix

Servers

HRM


Logical name spaces l.jpg

User distinguished name system

Certificate authority

Resource logical name - resources themselves

MDS

Grid services registry

File logical name

Replica catalog

Application abstraction - functions on resources

Virtual data grid

Grid services registry

Logical Name Spaces


Mappings on name space l.jpg

Define logical resource name system

List of physical resources

Replication

Write to logical resource completes when all physical resources have a copy

Load balancing

Write to a logical resource completes when copy exist on next physical resource in the list

Fault tolerance

Write to a logical resource completes when copies exist on “k” of “n” physical resources

Mappings on Name Space


Consistency constraints l.jpg

Service state information system

Partial completion of service

State information update in registry

Synchronous, asynchronous, deferred

Service composition

Order of execution

Virtual organization - need definition

Specification of set of registries

Federation of virtual organizations

Relationships between registries

Consistency Constraints


Internal service state information l.jpg

Digital entity manipulation system

Write locks on files

Write locks on containers

Synchronization flags on replicas

Access controls

Encryption / compression

Mappings on state information

Encryption / compression

Access controls on state information

Replica location of state information

Internal Service State Information


State information update l.jpg

Operation completion criteria system

Posix read/write semantics

Logical resource name access (list of physical resource names)

Replication (write to all physical resources)

Fault tolerance (write to “k” of “n”)

Load leveling (write to next physical resource in list)

Information update criteria

Update state information in registry on completion

Return state information, but do not update registry

Set partial completion flags in registry, implies the need for completion mechanisms

State Information Update


Service composition l.jpg

Implied semantics system

Write to replicated container

Do access controls apply to the container or to the file in the container?

Are access controls applied to the physical name or to the logical name?

Execution order when update multiple registries

When replicate a container, update:

File location, container location, access control on file

Service Composition


Virtual organization l.jpg

When execute a service, the input parameters may come from: system

Service invocation parameters

Indirection, call to a related service

Implied, assumed name space

Specification of the registries within a virtual organization

Certificate authority / CAS / Replica catalog / Container catalog / Metadata catalog / MDS

Interactions between logical name spaces

Foreign key for data based on logical name or unique ID

Virtual Organization


Federation of virtual organizations l.jpg

Specification of the registry that manages system

State information updates

Access controls

Federation based upon

Snapshots of information - time stamped

Publication links shadow links

Destination of link source of link

Push of updates pull of updates

Federation of Virtual Organizations


Data grid brick l.jpg

Data grid to authenticate users, manage file names, manage latency, federate systems

Intel Celeron 1.7 GHz CPU

SuperMicro P4SGA PCI Local bus ATX mainboard

1 GB memory (266 MHz DDR DRAM)

3Ware Escalade 7500-12 port PCI bus IDE RAID

10 Western Digital Caviar 200-GB IDE disk drives

3Com Etherlink 3C996B-T PCI bus 1000Base-T

Redstone RMC-4F2-7 4U ten bay ATX chassis

Linux operating system

Cost is $2,200 per Tbyte plus tax

Gig-E network switch costs $500 per brick

Effective cost is about $2,700 per TByte

Data Grid Brick


Slide57 l.jpg

Knowledge Based Data Grid Roadmap latency, federate systems

Ingest

Services

Management

Access

Services

Relationships

Between

Concepts

Knowledge

Repository for

Rules

Knowledge or

Topic-Based

Query / Browse

Knowledge

XTM DTD

  • Rules - KQL

(Model-based Access)

Information

Repository

Attribute- based

Query

Attributes

Semantics

SDLIP

Information

XML DTD

(Data Handling System)

Data

Fields

Containers

Folders

Storage

(Replicas,

Persistent IDs)

Grids

Feature-based

Query

MCAT/HDF


For more information l.jpg
For More Information latency, federate systems

Reagan W. Moore

San Diego Supercomputer Center

[email protected]

http://www.npaci.edu/DICE/

http://www.npaci.edu/DICE/SRB/

http://www.npaci.edu/dice/srb/mySRB/mySRB.html


ad