Chinese Virtual Observatory
Download
1 / 45

China-VO Data Access Service Based on OGSA - PowerPoint PPT Presentation


  • 80 Views
  • Uploaded on

Chinese Virtual Observatory. China-VO Data Access Service Based on OGSA. Jian Sang National Astronomical Observatory of China. Outline. VO,Grid and OGSA Build the catalog data service Build the image mosaic service Faced technical difficulties. The total area of astro telescopes in m**2.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' China-VO Data Access Service Based on OGSA' - saxton


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Chinese Virtual Observatory

China-VO Data Access Service

Based on OGSA

Jian Sang

National Astronomical Observatory of China

IVOA Small Projects Meeting 2003


Outline
Outline

  • VO,Grid and OGSA

  • Build the catalog data service

  • Build the image mosaic service

  • Faced technical difficulties

IVOA Small Projects Meeting 2003


The increase of astronomical data

The total area of astro telescopes in m**2

The total Gigapixels of CCDs.

The number of pixels and the data double every year!

The Increase Of Astronomical Data

IVOA Small Projects Meeting 2003


Challenges
Challenges

  • The quantity of data nearly amounts to PB.

  • The data is distributed and stored in heterogeneous DBMSs in heterogeneous

    host environments.

IVOA Small Projects Meeting 2003


The vo s goal
The VO’s Goal

  • The VO’s initial goal is to federate existing astronomical data archives and provide standard services for manipulating these data.

HOW TO REACH THIS GOAL?

The Grid technology can solve the problem!

IVOA Small Projects Meeting 2003


What is grid
What is Grid

  • Grid technology has been driven by genesis from metacomputing, but…

  • In practice, the Grid is about resource sharing and coordinated problem solving in dynamic, multi-institutional virtual organizations

  • Focus on how to enable, maintain and control the sharing of resources to achieve a common goal

IVOA Small Projects Meeting 2003


What grid offers
What “Grid“ offers:

  • Resource management protocols and services that support secure remote access to shared data resources and computing and the co-allocation of multiple resources.

  • Security solutions that support management of credentials and policies.

  • Information query protocols and services that provide configuration and status information about resources,organizations and services.

  • Data Management services that locate and transport datasets between storage systems and applications.

IVOA Small Projects Meeting 2003


What is ogsa
What is OGSA

  • The Open Grid Services Architecture (OGSA) represents an evolution towards a Grid system architecture based on Web services concepts and technologies. 

  • The OGSA integrates key Grid technologies (including the Globus Toolkit with Web services mechanisms to create a distributed system framework based around the Open Grid Services Infrastructure (OGSI).

In Grids ,Everything is Service

IVOA Small Projects Meeting 2003


The open grid services architecture
The Open Grid Services Architecture

  • Service orientation to virtualize resources

  • From Web services:( everything is service)

    -Standard interface definition mechanisms:multiple protocol bindings,multiple implementations,local/remote transparency

  • Building on Globus Toolkit:

    -Grid service :semantics for service interactions

    -Management of transient instances

    -Factory,Registry,Discovery,other services

    -Reliable and secure transport

  • Multiple host environments:J2EE,.NET,C,…

IVOA Small Projects Meeting 2003


The structure of grid service
The Structure of Grid Service

IVOA Small Projects Meeting 2003


Grid service interfaces
Grid service interfaces

IVOA Small Projects Meeting 2003


Construct the astronomical data grid
Construct The Astronomical Data Grid

The astronomical data service is the most fundamental and important component in Virtual Observatory.

In the aspect of data share, the VO can be think as a astronomical Data Grid

VO=Astronomical Data Grid

IVOA Small Projects Meeting 2003


Outline1
Outline

  • VO,Grid and OGSA

  • Build the catalog data access service

  • Build the image mosaic service

  • Faced difficulties

IVOA Small Projects Meeting 2003


The classification of astronomical data service
The Classification of Astronomical Data Service

  • Astronomical Catalog Service

  • Image Mosaic Service

  • Spectrum Data Service

  • Simulation Data Service

IVOA Small Projects Meeting 2003


Existing Astronomical Datasets we have

IVOA Small Projects Meeting 2003


Build catalog data service
Build Catalog Data Service

How to federate the catalog data into VO,that is, how to build Data Service using the existing databases and programs?

IVOA Small Projects Meeting 2003


Define catalog service interface
Define Catalog Service Interface

Some standards we used:

  • Input Query Language : SQL(now),ADQL (plan)

  • Output Data Format: VOTable 1.0

  • Catalog resource metadata registry protocol:

    VOResource 0.9

input: ADQL query sentence

output: VOTable format result

it makes service interface/API simple.

IVOA Small Projects Meeting 2003


How to use existing databases and programs to create catalog data service
How to use existing databases and programs to create catalog data service

How to create a catalog data service that can understand ADQL and generate VOTable format result??

we adopt two ways!

  • Reconstruct the existing catalog DBMS

  • Encapsulate search program,like pmm

    The CDS has offered search program for big catalog like USNO A2,0…..

IVOA Small Projects Meeting 2003


Catalog data service based on db
Catalog data service based on DB data service

GT3 Interface

VOTable

ADQL

ADQL/SQL

Translator

VOTable

Wrapper

SQL

ResultSet

JDBC

Catalog/metadata

DBMS

IVOA Small Projects Meeting 2003


Advantage and disadvantage
Advantage and disadvantage data service

  • Can sufficiently use the functions of SQL language and implement complex query.

  • DBMSs offer the most powerful functions for data management and maintenance.

  • Need many works to reconstruct the DBs.

  • To big catalogs, like USNOB1.0,2MASS PSC, query efficiency is low

IVOA Small Projects Meeting 2003


Data service based on search program
Data service based on search program data service

GT3 Interface

ADQL

VOTable

ADQL

Translator

VOTable

Wrapper

parameters

stream

JNI/

program

Data Files

IVOA Small Projects Meeting 2003


Advantage and disadvantage1
Advantage and disadvantage data service

  • Positional search is quicker than DB

  • Only offer search functions that programs could offer. Many programs only offer position search functions,no statistical functions.

IVOA Small Projects Meeting 2003


Catalog access service provided by us
Catalog Access Service Provided by us data service

IVOA Small Projects Meeting 2003


How to call a catalog data service
How to call a Catalog data service data service

Resource

Registry

1.<Find Factory>

<registry>

2.<Factory GSH>

3.<create data service>

Data

Service

Factory

Grid

Client

4.<Data service GSH>

Create

Data

service

5.<data request(ADQL)>

Data

Service

Instance

6.<result (VOTable)>

Database

IVOA Small Projects Meeting 2003


Use data service to build www service for end user

End Users data service

Data service

Data service

Data service

Use Data Service to build www service for end user

Web Client

End user don’t know where the data services are

http

Data Mining

Service

Data Visualization

Service

Data processing

Service

Web server

Grid Client

Resources

Register

Services

Register

MySQL

Oracle 9i

Files

IVOA Small Projects Meeting 2003


Use data service to create other service
Use data service to create other service data service

Our next work is to build a

multi-wavelength cross-identification service (MWCI)based on the catalog data service.

What is multi-wavelength cross-identification ?

To cross-identify datasets by positional consistency, we can understand objects from different wavelength properties.

IVOA Small Projects Meeting 2003


The steps of multi wavelength cross identification
The steps of multi-wavelength data servicecross-identification

  • Cross-identify datasets from different wavelengths within error radius.

  • Divide the result of cross-identification into three situations: one-to-one, one-to-two, one-to-many.

  • Choose the one-to-one entry for data mining

  • The other two situations need statistical analysis to determine which source are the true counterpoint.

IVOA Small Projects Meeting 2003


Requirements
Requirements data service

  • Locate the datasets that users want to use.

    (dataset discovery)

  • How to cross-match the datasets in heterogeneous DBMSs at different locations effectively and efficiently.

  • Find storage resource to store the results

IVOA Small Projects Meeting 2003


Registry data service

MWCI

Factory

Data

Service

2

4

2MASS

1

MWCI

5

MWCI Service Provider

User

Application

.

.

.

3

6

.

.

.

5

Data

Service

storage

Factory

7

6

NVSS

4

storage

Storage Service Provider

IVOA Small Projects Meeting 2003


Outline2
Outline data service

  • VO,Grid and OGSA

  • Build the catalog data access service

  • Build the image mosaic service

  • Faced technical difficulties

IVOA Small Projects Meeting 2003


Build the image mosaic service
Build The Image Mosaic Service data service

  • Use DSS-I sky image build our first image mosaic service.

IVOA Small Projects Meeting 2003


The definition of interface of service
the definition of interface of service data service

  • Input parameters:

    1.RA,2.Dec,3.image height,4.image width

  • transport protocols :gridFTP

  • Output Data format :fits

IVOA Small Projects Meeting 2003


Realization of dss i image mosaic service
Realization of DSS-I image data servicemosaic service

GridFTP

GT3 Interface

JNI/

Fits file

parameters

GetImage

DSS-I Image

Files

IVOA Small Projects Meeting 2003


Outline3
Outline data service

  • VO,Grid and OGSA

  • Build the catalog data access service

  • Build the image mosaic service

  • Faced technical difficulties

IVOA Small Projects Meeting 2003


Technical difficulties
Technical Difficulties data service

  • service/resource registry and discovery!

  • ADQL2SQL translator

  • protocol shortcoming

IVOA Small Projects Meeting 2003


Protocol shortcomings
protocol shortcomings data service

  • The shortcomings of VOTable 1.0 protocol

  • 1.How to encapsulate result of join query!!

  • 2.The standard to encapsulating spectrum data

  • 3.the definition of FIELD element is not strict and uncompleted

  • The shortcoming of UCD

  • 1.Can’t express concrete meaning,such as “ERROR” ,Error for what??

  • 2. incomplete, example:HTMID has no UCD

  • Lack of standard for Unit

IVOA Small Projects Meeting 2003


Thank You data service

Q & A

?

www. .org

IVOA Small Projects Meeting 2003


Our provided catalogs in catalog service
Our provided catalogs in Catalog Service data service

IVOA Small Projects Meeting 2003


The step of calling a data service
The Step Of Calling A Data Service data service

IVOA Small Projects Meeting 2003


Transparencies for astro data access
Transparencies for Astro Data Access data service

  • Heterogeneity Transparency

  • Name Transparency

  • Distribution Transparency

IVOA Small Projects Meeting 2003


What is grid service
What is Grid Service? data service

IVOA Small Projects Meeting 2003


What is the data grid
What Is The Data Grid data service

  • DataGrid :A dynamic logical namespace that enables coordinated sharing of heterogeneous distributed storage resources and digital entities based on local and global policies across administrative domains in a virtual enterprise.

  • DataGrid

    • Logical name space for location independent identifiers

    • Abstractionsfor storage repositories, information repositories, and access APIs

    • Latency management

IVOA Small Projects Meeting 2003


Using a data grid in abstract

Data delivered data service

Ask for data

  • The data is found and returned

  • Where & how details are managed by data grid

Using a Data Grid –in Abstract

Data Grid

  • User asks for data from the data grid

IVOA Small Projects Meeting 2003



ad