The World-Wide Telescope: Astronomy with Terabytes. Alex Szalay The Johns Hopkins University Jim Gray Microsoft Research. Outline. Challenges: A New Kind of Science Publishing the Data Data Federation The Virtual Observatory Analyzing Terabytes of Data. Living in an Exponential World.
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
The World-Wide Telescope: Astronomy with Terabytes
Alex SzalayThe Johns Hopkins University
Jim Gray Microsoft Research
Exponential data growth: Distributed collections Soon Petabytes
New analysis paradigm: Data federations, Move analysis to data
New publishing paradigm: Scientists are publishers and Curators
describing natural phenomena
using models, generalizations
simulating complex phenomena
synthesizing theory, experiment and computation with advanced data management and statistics
Project www site
You can GREP 1 MB in a second
You can GREP 1 GB in a minute
You can GREP 1 TB in 2 days
You can GREP 1 PB in 3 years
Oh!, and 1PB ~4,000 disks
At some point you need indices to limit searchparallel data search and analysis
This is where databases can help
You can FTP 1 MB in 1 sec
You can FTP 1 GB / min (= 1 $/GB)
… 2 days and 1K$
… 3 years and 1M$
FTP and GREP are not adequate
take the analysis to the data!
In your address space
NVO WESIX service: build your object catalog in 5 mins
SELECT o.objId, o.r, o.type, t.objId
FROM SDSS:PhotoPrimary o,
AND o.type=3 and (o.I - t.m_j)>2
Each SkyNode publishes
Schema Web Service
Database Web Service
Plans Query (2 phase)
Is itself a web service
International Virtual Observatory Alliance (IVOA)
Standards driven by evolving new technologies
Application to astronomy domain
Dealing with the astronomy legacy
Cross-match your data with numerous catalogs
OpenSkyQuery allows you to cross-match astronomical catalogs and select subsets of catalogs with a general and powerful query language. You can also import a personal catalog of objects and cross-match it against selected databases.
Search, plot, and retrieve SDSS, 2dF, and other spectra
The Spectrum Services web site is dedicated to spectrum related VO services. On this site you will find tools and tutorials on how to access close to 500,000 spectra from the Sloan Digital Sky Survey (SDSS DR1) and the 2 degree Field redshift survey (2dFGRS). The services are open to everyone to publish their own spectra in the same framework. Reading the tutorials on XML Web Services, you can learn how to integrate the 45 GB spectrum and passband database with your programs with few lines of code.
Upload images to SExtractor and cross-correlate the objects found with selected survey catalogs.
This NVO service does source extraction and cross-matching for any astrometric FITS image. The user uploads a FITS image, and the remote service runs the SExtractor software for source extraction. The resulting catalog can be cross-matched with any of several major surveys, and the results returned as a VOTable. The web page also allows use of Aladin or VOPlot to visualize results.
Ra ± x
Zones allow 60,000 Parallel Jobs
Partition Parallelism: 3.7 hours
Pipeline Parallelism: 2.5 hours
Or… as fast as we can read USNOB + .5 hours
Angular Galaxy Surveys
Galaxy Redshift Surveys
Petabytes/year by the end of the decade…