Japanese virtual observatory
Download
1 / 55

天文データベース・計算資源 連携機構 Japanese Virtual Observatory - PowerPoint PPT Presentation


  • 79 Views
  • Uploaded on

天文データベース・計算資源 連携機構 Japanese Virtual Observatory. 大石雅寿 ( 国立天文台 ). Nobeyama 45m. Data Resources in NAOJ. Subaru. Subaru 8.2m Optical-Infrared Telescope Kiso 105cm Schmidt Camera Okayama 188cm Optical Telescope Nobeyama 45m Radio Telescope Nobeyama Millimeter Array

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' 天文データベース・計算資源 連携機構 Japanese Virtual Observatory' - eliot


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Japanese virtual observatory

天文データベース・計算資源連携機構 Japanese Virtual Observatory

大石雅寿 (国立天文台)

宇宙地球系情報科学研究会


Data resources in naoj

Nobeyama 45m

Data Resources in NAOJ

Subaru

Subaru 8.2m Optical-Infrared Telescope

Kiso 105cm Schmidt Camera

Okayama 188cm Optical Telescope

Nobeyama 45m Radio Telescope

Nobeyama Millimeter Array

Nobeyama Radioheliograph

VSOP

VERA

ALMA

宇宙地球系情報科学研究会


Catalogues, published tables

Whole sky view shows heterogeneity

宇宙地球系情報科学研究会


Simple Spectrum Access

Image / Spectrum / Catalog interoperability

宇宙地球系情報科学研究会


Smoka archive in naoj

  • http://smoka.nao.ac.jp/

  • Public science archive of the

  • Subaru Telescope,

  • 188cm telescope at Okayama Astrophysical Observatory,

  • 105cm Schmidt telescope at Kiso Observatory / University of Tokyo.

  • Reduced data of Subaru Suprime-Cam is now available.

SMOKA Archive in NAOJ

宇宙地球系情報科学研究会


Data resources in jaxa isas

Data Resources in JAXA/ISAS

ASCA

ASCA X-ray astronomy satellite

YOHKOsolar physics satellite

Ginga X-ray astronomy satellite

HALCA VLBI satellite

Geotail geomagnetosphere satellite

Akebonoaurora observation satellite

ASTRO-F Infrared satellite

ASTRO-E2 X-ray satellite

SOLAR-B

YOHKO

HALCA

宇宙地球系情報科学研究会


Darts

DARTS

http://www.darts.isas.jaxa.jp

宇宙地球系情報科学研究会


Data productivity
Data Productivity

  • Nobeyama Radio :~1TB/yr

  • [email protected] :~20TB/yr

  • ASTRO-F(will be launched)

    :several 100 GB in total

  • ALMA(planned)  :~PB/yr

    Flood of excellent data (survey data)

    Digitized & Archived

宇宙地球系情報科学研究会


Archive

Already more retrieval than ingest!

Archiveを用いた天文学研究

  • large fraction of astro-papers based on archives

  • HST archive use growing faster than archive

graphics from US NVO project

宇宙地球系情報科学研究会



Virtual observations
Virtual Observations

宇宙地球系情報科学研究会


Vo projects in the world
VO Projects in the world

  • 15の国と地域で進行中 (参加希望国増加中)

  • International Virtual Observatory Alliance (IVOA)VO間の連携のための標準を策定

  • 日本は分散連携DBへの統一検索言語のとりまとめ役

宇宙地球系情報科学研究会


Standardization in ivoa
Standardization in IVOA

  • 分散DBへの連携検索言語(VOQL)

  • メタデータ:内容,交換プロトコルbased on OAI-PMH

  • 画像,スペクトル等の取得プロトコルSkyNode, SIAP, SSAP, STC, etc.

  • DBの各属性値の呼称の統一UCD (Unified Contents Descriptions)

  • 出力形式:VOTable (XML)  天文標準のFITSフォーマットを包含

宇宙地球系情報科学研究会



国際天文学連合(IAU)決議

IAU XXVth Gen. Assem. 決議 (2003 Jul.)

  • 望遠鏡データのアーカイブへの蓄積とインターネット上での公開

  • 知的所有権の放棄&著作権保護

  • アーカイブ公開のための予算確保

宇宙地球系情報科学研究会


Oecd 04 aug
OECD勧告(‘04 Aug)

Recommendations:

New projects and facilities must take the data management, storage, maintenance, and dissemination into account at the earliest planning stages, consulting potential users in the process. Agencies and governments should consider adopting the IAU resolutions as the basis for progress in this field. Agencies should recognise that this is an important long-term issue and should co-ordinate plans, provide adequate funding on a long-term basis, and support development and maintenance of the needed infrastructure. Agencies should encourage broadening of existing VO collaboration into a fully representative global activity.

宇宙地球系情報科学研究会


Jvo japanese virtual observatory
JVO : Japanese Virtual Observatory

  • Purpose:

    • Easy access to federated Astronomical databases

    • Interoperability through IVOA

Interoperability through IVOA

JVO

Researcher

Satellite-borne telescopes

Ground-base telescopes

Nobeyama

Subaru

ASTRO-EII

宇宙地球系情報科学研究会


JVOの概念図

宇宙地球系情報科学研究会


Design toward prototype 3
Design toward Prototype 3

  • Support IVOA Standard Protocols

    • SIAP, SSAP, SkyNode

    • implement ADQL

  • improve registry, employing OAI-PMH architecture

  • flexible workflow architecture

  • introduce User management

    • LDAP

    • User Storage Area (support VOStore?)

  • API to control JVO with SOAP

宇宙地球系情報科学研究会


Science targets in prototype 3
Science targets in prototype 3

  • Cosmic String Search through the gravitational lens effect.

  • Environmental dependence of QSO evolution.

  • Automatic classification of late type star.

  • Search for very metal-poor stars.

  • etc.

宇宙地球系情報科学研究会


  • シンプル解析機能

    • 天体検出(Sextractor)

    • HyperZ

    • 画像演算

    • Photometry

  • 特殊解析機能

    • 重力レンズ

    • 銀河ダスト

    • 化学組成の進化

解析 interface

Grid サービス

OAI-PMH

Grid サービス

Grid サービス

OAI-PMH

OAI-PMH

メタDB

メタDB

メタDB

天文DB

天文DB

ユーザDB

ユーザーインターフェイス

  • ユーザ登録

  • ユーザ認証

  • JVOQLエディタ

  • リソースファインダによるDB検索

  • UCDファインダー

  • 天体名レゾルバー

  • 実行状況表示

  • VOTable Viewer

  • Plotter

  • 画像データ Viewer

  • スペクトルデータ Viewer

  • 複数カタログの重ね合わせプロット

  • 異なる波長画像の重ね合わせ表示

ユーザ認証

データ表示

検索 interface

JSP

JSP

JSP

  • リソースファインダーによるDB検索 (白)

JSP

JSP

DBナビゲータ

Servlet

JVOシステム部

解析実行状況

検索指示

(JVOQL)

検索実行状況

解析指示

DBメタデータ検索

状態確認

状態確認

検索文 (XPath)

検索結果 (XML)

コントローラー JB

コントローラー JB

Grid サービス

JVOQL

Parsed JVOQL

スケジューラ JB

JVOQLパーサ JB

GSI-SFSによるデータ転送

Registry

(XMLDB)

観測指示書

GSI-SFSによるデータ転送

Parsed JVOQL, 実績

エクゼキュータ JB

スケジューラ JB

実績

観測指示書

OAI-PMH

検索命令

SOAPによる

結果渡し

エクゼキュータ JB

検索命令

SOAPによる

結果渡し

検索命令

SOAPによる

結果渡し

DB・解析部

OAI-PMH

Grid サービス

GridFTP

GridFTP

宇宙地球系情報科学研究会


Jvo query language jvoql

  • JVOQL is designed as a prototype of VO Query Language.

  • Characteristics of the JVOQL:

    • SQL based Query Language

    • Query Language for the distributed astronomical DB.

    • Can search and retrieve observational data as well as catalog data

    • Upward compatible with the ADQL and SIAP syntax.

    • Scalable syntax structure. Very simple core syntax and extension syntax packages.

JVO Query Language (JVOQL)

宇宙地球系情報科学研究会



Image search

FITS file management table

Parameter based Image Query

Image Search Engine

SQL

File name, Metadata

Search Parameters

Image Search

Image FITS file

XML

Image cutout

SQL based Image Query

Data Search on a virtual table.

SQL

File name, Metadata

SQL

XML,FITS,

Jpeg

Column metadata

request

image

cutout

宇宙地球系情報科学研究会


Jvoql syntax requirement 1

  • Unified query language for both the catalog and observation data such as image data, spectrum, 3D-cube, photon list …

    • Parameter query (SIAP) can be replaced by SQL thinking that the parameters are columns of a relational table.

    • Observational data or pointer (URL) to retrieve the data is also a column of the relation table.

JVOQL Syntax Requirement 1

http://jvo.nao.ac.jp/imageData?Pos=24,5&Size=0.2&format=VOTable

Select imageURL, …

From naoj:imageData

Where pos=Point(24,5) and size=0.2 and format=‘VOTable’

“pos” , “size”, “imaegURL” are virtual columns.

宇宙地球系情報科学研究会


Jvoql syntax requirement 2

Ext. 1

Ext. X

Ext. n

Ext. 2

Ext. 6

Basic

Basic

Ext. 3

Ext. 3

Ext. 5

Ext. 4

  • VOQL should have scalable syntax:

    • Small size DBs  very simple syntax for easy implementation.

    • Large size DBs  sophisticated syntax for efficient data search.

    •  hybrid syntax structure : a basic syntax to be implemented by all the VO data service and extension syntax packages.

JVOQL Syntax Requirement 2

Information which extensions are implemented will be registered in the registry service or data service itself returns the information through “voqlSpec” interface.

宇宙地球系情報科学研究会


Jvoql adql x

JVOQL2Parser parses JVOQL and constructs SelectSQL object. SelectSQL object has a method to return a SelectType object which is a data model defined in ADQL schema.

This is an example of translation from JVOQL to ADQL-x.

Select ra, dec, mag

From spcam

Where (ra, dec) within

Point((20, 30), 1.0)

JVOQL  ADQL/x

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<Select xmlns:ns1="urn:vo-coord" xmlns:region="urn:vo-region" xmlns="http://www.ivoa.net/xml/ADQL/v0.8">

<SelectionList>

<Item Table="spcam" Name="ra" xsi:type="columnReferenceType"/>

<Item Table="spcam" Name="dec" xsi:type="columnReferenceType"/>

<Item Table="spcam" Name="mag" xsi:type="columnReferenceType"/>

</SelectionList>

<From>

<Table Name="spcam" Alias="" xsi:type="tableType"/>

</From>

<Where>

<Condition xsi:type="regionSearchType">

<Region> <region:Circle unit="deg" coord_system_id="IRCS">

<region:Center>20.0 30.0</region:Center> <region:Radius>1.0</region:Radius>

</region:Circle>

</Region>

</Condition>

</Where>

</Select>

宇宙地球系情報科学研究会


Region search using htm index

Region search is a common search criterion for an astronomical database. For efficient search data should be properly indexed on the object coordinate.

Catalog table

HTM Index table

Region Search using HTM index

Select ra, dec, mag

From Catalog

Where Point(ra,dec) within Box((20,+15), 1.0)

Select c.ra, c.dec, c.mag

From Catalog as c

Natural Left Join htmIndex as i

Where i.htm between 16522500 and 16522512

OR

i.htm between 16522500 and 16522512

宇宙地球系情報科学研究会

http://www.sdss.jhu.edu/htm/


Parsing jvoql and generating workflow
Parsing JVOQL and Generating Workflow astronomical database. For efficient search data should be properly indexed on the object coordinate.

  • “JVOQL Parser” generates query for each host

  • “Scheduler” generates:

    • count query job for host1

    • count query job for host2

  • “Executer” executes jobs on remote hosts

  • “Scheduler” generates based on the result of execution

    • query job for host1

    • xmatch job for host2

    • image query for host1 and host2

  • “Executer” executes jobs on remote hosts

宇宙地球系情報科学研究会


Remote execution
Remote execution astronomical database. For efficient search data should be properly indexed on the object coordinate.

  • Proto1:

    • Globus toolkit ver 2

    • using globus-job-runcommand

    • 1 call = 30 sec

      • 1 query ~10 min!

  • Proto2:

    • Globus toolkit ver 3

    • using Grid Service

    • 1 call = 2-3 sec

      • overhead time is only ~ 30 ms

宇宙地球系情報科学研究会


Data transfer
Data Transfer astronomical database. For efficient search data should be properly indexed on the object coordinate.

  • Web/Grid Service

    • Query result is always returned to Portal server

  • GridFTP

    • Query result can be directly transferred to XMatch server

Portal Server

Xmatch

Query

Query

VOTable1

VOTable2

Data Server

Data Server

Portal Server

Xmatch

Query

Query

VOTable2

Data Server

Data Server

VOTable1

宇宙地球系情報科学研究会


Data transfer1
Data Transfer astronomical database. For efficient search data should be properly indexed on the object coordinate.

  • We tried:

    • GridFTP, RFT

    • GSI-SFS

  • Learned:

    • SFS needs to be modified

      • A server cannot be a client.

    • RFT is promising in Globus environment

    • need support for HTTP and Web Services

宇宙地球系情報科学研究会


メタデータ astronomical database. For efficient search data should be properly indexed on the object coordinate.

  • 2種類のメタデータ

    • どのサーバに天文データがあるか?

      • 分散アーカイブ連携検索のためにシステムが必要な情報

    • どのような内容の天文データか?

      • FITSヘッダに書かれるような情報

      • 観測装置、波長帯、観測日時、天域、…

  • 実装方法の改良

    • プロトタイプ1: UDDI

      • サーバ 情報のみ検索。データ内容の検索は不可

    • プロトタイプ2: XMLDB

      • IVOAで検討されたメタデータ仕様 (XML形式で定義) に準拠

      • XMLDB(カレアレア)を用いて検索

宇宙地球系情報科学研究会


Oai pmh
メタデータ交換 astronomical database. For efficient search data should be properly indexed on the object coordinate. :OAI-PMH

宇宙地球系情報科学研究会


Resource metadata

service metadata astronomical database. For efficient search data should be properly indexed on the object coordinate.

column metadata

Identity metadata

curation metadata

content metadata

Resource Metadata

宇宙地球系情報科学研究会

Catalog、Table Resourceはcolumn metadata 以外を持つ。

Column Resource はすべての metadata をもてる。(省略可)


利用できるアーカイブの検索 astronomical database. For efficient search data should be properly indexed on the object coordinate.

検索画面

検索結果

宇宙地球系情報科学研究会


検索文の自動生成 astronomical database. For efficient search data should be properly indexed on the object coordinate.

宇宙地球系情報科学研究会


Jvo skynode architecture

ResultSet astronomical database. For efficient search data should be properly indexed on the object coordinate.

getMetadata()

getString()

getFits()

getJpeg()

  • Apache Axis and Tomcat are used for implementing the VO standard interface, and Globus Toolkit is used for an optional Grid interface.

  • Four kinds of query languages are converted to java class SelectSQL and query is executed through JDBC-like interface.

  • Query results is obtained as ResultSet, and table data is formatted to VOTable or CSV file.

JVO Portal

JVO SkyNode Architecture

Internet

JVO SkyNode

JVOQL

Grid

GTK 3 (optional)

SelectSQL

Translator

Query Executer

Formatter

JDBC for

SkyNode

JVO SkyNode DBMS

HTTP/SOAP

AXIS

JVOQL

ADQL-x

HTTP/Parameter

Tomcat

SIAP

SSAP

宇宙地球系情報科学研究会


Free software used in jvo skynode

  • Java (J2SE) astronomical database. For efficient search data should be properly indexed on the object coordinate. : Generally used in the development.

  • JAXB 1.0: used for generating Java class files from VO standard schema, ADQL, VOTable, VOResource etc…

  • JavaCC : used for parsing JVOQL and constructing SelectSQL java object.

  • PostgreSQL : Backend DBMS.

  • HTM library : developed by JHU, used for region search.

  • Apache AXIS, Tomcat : Web service and servlet.

  • Globus Toolkit : Grid service.

  • etc…

Free software used in JVO Skynode

宇宙地球系情報科学研究会


Access time to the us vo
Access time to the US VO astronomical database. For efficient search data should be properly indexed on the object coordinate.

宇宙地球系情報科学研究会


Jvo is seen from the uk vo
JVO is seen from the UK VO astronomical database. For efficient search data should be properly indexed on the object coordinate.

宇宙地球系情報科学研究会


宇宙地球系情報科学研究会 astronomical database. For efficient search data should be properly indexed on the object coordinate.


宇宙地球系情報科学研究会 astronomical database. For efficient search data should be properly indexed on the object coordinate.


宇宙地球系情報科学研究会 astronomical database. For efficient search data should be properly indexed on the object coordinate.


宇宙地球系情報科学研究会 astronomical database. For efficient search data should be properly indexed on the object coordinate.


宇宙地球系情報科学研究会 astronomical database. For efficient search data should be properly indexed on the object coordinate.


宇宙地球系情報科学研究会 astronomical database. For efficient search data should be properly indexed on the object coordinate.


Cosmic strings
Cosmic Strings? astronomical database. For efficient search data should be properly indexed on the object coordinate.

  • Theoretical origin of the elementary particles which existed (?) at t ~ 10-35 s,T ~ 1015 GeV

  • Not a POINT, has its SIZE

  • Width < 10-22m, Length ~ size of the Universe.

  • Mass of 10km string ~ the Earth

宇宙地球系情報科学研究会


Gravitational lens
Gravitational Lens astronomical database. For efficient search data should be properly indexed on the object coordinate.

  • A phenomenon that the space-time is distorted by a huge mass object, and multiple images of another object behind the massive object are observed around it.

宇宙地球系情報科学研究会


Gravitational lens by cosmic strings
Gravitational Lens by Cosmic Strings astronomical database. For efficient search data should be properly indexed on the object coordinate.

宇宙地球系情報科学研究会


Cosmic String Search Result astronomical database. For efficient search data should be properly indexed on the object coordinate.

宇宙地球系情報科学研究会


Search for gravitational lenses produced by cosmic strings
Search for Gravitational Lenses astronomical database. For efficient search data should be properly indexed on the object coordinate. produced (?) by Cosmic Strings

  • SXDS data observed by Subaru

  • Query results were obtained less than 5 min, displaying SEDs

  • It has been proven

    that VO can

    accelerate

    researches.

宇宙地球系情報科学研究会


JVO astronomical database. For efficient search data should be properly indexed on the object coordinate. による仮想データ検索システム

ObservationSystem

ObservationSystem

ASTRO-E2

Remote

Observation DATA

かに星雲の時刻と波長での違い

Slide courtesy of Robert Brunner @ CalTech.

Real Time

Observation DATA

「超」仮想DB

ObservationSystem

すばる望遠鏡

Archived DATA

インターネット

Mass storage archiver

Query

Archived DATA

DATA

解析/加工処理

ユーザ/研究者

ObservationSystem

SimulatedDATA

公開天文台

On-Demand DATA

宇宙地球系情報科学研究会

解析データ

Simulation System

国立天文台4次元デジタル

宇宙プロジェクト ご提供


Alma in chile sub mm wave
ALMA in Chile (sub-mm wave) astronomical database. For efficient search data should be properly indexed on the object coordinate.

宇宙地球系情報科学研究会


END astronomical database. For efficient search data should be properly indexed on the object coordinate.

宇宙地球系情報科学研究会


ad