Four talks
Download
1 / 35

four talks - PowerPoint PPT Presentation


  • 317 Views
  • Uploaded on

Four Talks . Jim Gray, Microsoft Alex Szalay, Ani Thakar, Jan Vandenberg, JHU Chris Stoughton, Fermilab. The article we actually wrote Online Scientific Publication, Curation, Archiving The advertised talk Computer Science Challenges in the VO A Web Server (SkyServer) tour

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'four talks ' - Gabriel


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Four talks l.jpg
Four Talks

Jim Gray, Microsoft

Alex Szalay, Ani Thakar, Jan Vandenberg, JHU

Chris Stoughton, Fermilab

  • The article we actually wroteOnline Scientific Publication, Curation, Archiving

  • The advertised talk Computer Science Challenges in the VO

  • A Web Server (SkyServer) tour

  • A Web Service (SdssCutout) tour

  • These lead up to Alex Szalay’s talk on Web Services


The paper we wrote online scientific publication curation archiving l.jpg

The Paper We WroteOnline Scientific Publication, Curation, Archiving

Jim Gray, Microsoft

Alex Szalay, Ani Thakar, Jan Vandenberg, JHU

Chris Stoughton, Fermilab


Premise l.jpg
Premise

  • Once published, scientific data needs to be available forever,so that the science can be reproduced/extended.

  • What does that mean?

    • Data

      • Ephemeral Data: could not be reproduced

      • Stable data: could be drived from emphemeral data.

    • Meta-data: how the data was collected/derivedis ephemeral

      • Must be preserved

      • Includes design docs, software, email, pubs, personal notes


Publishing data l.jpg
Publishing Data

Roles

Authors

Publishers

Curators

Archives

Consumers

Traditional

Scientists

Journals

Libraries

Archives

Scientists

Emerging

Collaborations

Project web site

Data+Doc Archives

Digital Archives

Scientists


The core problem no economic model l.jpg
The Core Problem: No Economic Model

  • The archive user has not yet been born. How can he pay you to curate the data?

  • The Scientist gathered data for his own purposeWhy should he pay (invest time) for your needs?

  • Answer to both:that’s the scientific method

  • Curating data (documenting the design, the acquisition and the processing)Is very hard and there is no reward for doing it.The results are rewarded, not the process of getting them.

  • Storage/archive NOT the problem (it’s almost free)

  • Curating/Publishing is expensive.


What sdss is doing capture the bits l.jpg
What SDSS is Doing: Capture the Bits

  • Best-effort documenting data and process.

  • Publishing data: often by UPS(~ 5TB today (dr1) and so ~15k$ for a copy)

  • Replicating data on 3 continents.

  • EVERYTHING online (tape data is dead data)

  • Archiving all email, discussions, ….

  • Keeping all web-logs.

  • Now we need to figure out how to organize/search all this metadata.


Sdss data inflation data pyramid l.jpg

Level 1AGrows 5TB pixels/year growing to 25TB~ 2 TB/y compressed growing to 13TB~ 4 TB today (level 1A in NASA terms)

Level 2Derived data products ~10x smaller But there are many catalogs.

Publish new edition each year

Fixes bugs in data.

Must preserve old editions

Creates data pyramid

Store each edition

1, 2, 3, 4… N ~ N2 bytes

Net: Data Inflation: L2 ≥ L1

Level 1A

4 editions of Level 2 products

E4

E3

E2

time

E1

4 editions oflevel 1A data(source data)

4 editions of level 2 derived data products. Note that each derived product is small, but they are numerous. This proliferation combined with the data pyramid implies that level2 data more than doubles the total storage volume.

SDSS Data Inflation – Data Pyramid


Summary l.jpg
Summary

  • Virtual Observatory will be an ecosystem ofauthors, curators, publishers, archivers, readerscontributing & using shared data.

  • The process and roles are changingauthor + project = publisher + curator

  • Ephemeral & stable data Capture ephemeral information. All design & metadata info is ephemeralCan tradeoff recomputing derived data

  • EconomicsAuthor/Curate cost dominates

  • SDSSData Inflation, Data Pyramid


Four talks9 l.jpg
Four Talks

  • The article we actually wroteOnline Scientific Publication, Curation, Archiving

  • The advertised talk Computer Science Challenges in the VO

  • A Web Server (SkyServer) tour

  • A Web Service (SdssCutout) tour

  • These lead up to Alex Szalay’s talk on Web Services


The advertised talk computer science challenges in the vo l.jpg

The Advertised TalkComputer Science Challenges in the VO

Jim Gray, Microsoft


Virtual observatory l.jpg
Virtual Observatory

  • Premise: Most data is (or could be online)

  • So, the Internet is the world’s best telescope:

    • It has data on every part of the sky

    • In every measured spectral band: optical, x-ray, radio..

    • As deep as the best instruments (2 years ago).

    • It is up when you are up.The “seeing” is always great (no working at night, no clouds no moons no..).

    • It’s a smart telescope: links objects and data to literature on them.


Virtual observatory data federation of web services l.jpg
Virtual ObservatoryData Federation of Web Services

  • Massive datasets live near their owners:

    • Near the instrument’s software pipeline

    • Near the applications

    • Near data knowledge and curation

    • Computer centers become Data Centers

  • Archives are replicated for

    • Performance

    • Availability/Reliability

  • Each Archive publishes a web service

    • Schema: documents the data

    • Methods on objects (queries)

  • Scientists get “personalized” extracts

  • Uniform access to multiple Archives

    • A common global schema


Some unique things about astro data l.jpg
Some Unique Things About Astro Data

  • There is a desire to compare data from different instruments

    • Most astronomers publish their data (especially surveys)

    • Combining data from different instruments gives more info

    • Szalay observes Metcalf’s law: utility grows as N2

    • This is less true in some other fields

  • It’s tractable

    • sizes fit in current regimes (10s of terabytes today)

    • tasks fit Beowulfs

  • Astro data is great sandbox for CS research.

    • High-dimensional data

    • Temporal, spatial, image datatypes

    • Few privacy/commercial concerns

    • There is lots of it


My 1 challenge going beyond files a file is an array of bytes science vs commerce l.jpg

Data in files FTP a local copy /subset.ASCII or Binary.

Each scientist builds own analysis toolkit

Analysis is tcl script of toolkit on local data.

Some simple visualization tools: x vs y

Data in a database

Standard reports for standard things.

Report writers for non-standard things

GUI tools to explore data.

Decision trees

Clustering

Anomaly finders

My #1 Challenge: going beyond files(a file is an array of bytes)Science vs Commerce


But some science is hitting a wall ftp and grep are not adequate l.jpg

You can GREP 1 MB in a second

You can GREP 1 GB in a minute

You can GREP 1 TB in 2 days

You can GREP 1 PB in 3 years.

Oh!, and 1PB ~10,000 disks

At some point you need indices to limit searchparallel data search and analysis search and analysis tools

This is where databases can help

You can FTP 1 MB in 1 sec

You can FTP 1 GB / min (= 1 $/GB)

… 2 days and 1K$

… 3 years and 1M$

But…some science is hitting a wallFTP and GREP are not adequate


What s needed not drawn to scale l.jpg

Data Mining

Algorithms

Miners

Scientists

Science Data & Questions

Database

To store data

Execute

Queries

Plumbers

Question & AnswerVisualization

Tools

What’s needed?(not drawn to scale)


Cs challenges for astronomers l.jpg

Scientists

Science Data & Questions

CS Challenges For Astronomers

  • Objectify your field:

    • Precisely define what you are talking about.

    • Objects and Methods / Attributes

    • This is REALLY difficult.

    • UCDs are a great start but, there is a long way to go

  • “Software is like entropy, it always increases.” -- Norman Augustine, Augustine’s Laws

    • Beware of legacy software – cost can eat you alive

    • Share software where possible.

    • Use standard software where possible.

    • Expect it will cost you 25% to 40% of project. 

  • Explain what you want to do with the VO

    • 20 queries or something like that.


Challenge to data miners linear and sub linear algorithms l.jpg

Data Mining

Algorithms

Miners

Challenge to Data Miners: Linear and Sub-Linear Algorithms

Techniques

  • Today most correlation / clustering algorithmsare polynomial N2 or N3 or…

  • N2 is VERY big when N is big (1018 is big)

  • Need sub-linear algorithms

  • Current approaches are near optimal given current assumptions.

  • So, need new assumptionsprobably heuristic and approximate


Challenge to data miners rediscover astronomy l.jpg

Data Mining

Algorithms

Miners

Challenge to Data Miners: Rediscover Astronomy

  • Astronomy needs deep understanding of physics.

  • But, some was discovered as variable correlations then “explained” with physics.

  • Famous example: Hertzsprung-Russell Diagramstar luminosity vs color (=temperature)

  • Challenge 1 (the student test): How much of astronomy can data mining discover?

  • Challenge 2 (the Turing test):Can data mining discover NEW correlations?


Plumbers organize and search petabytes l.jpg

Plumbers

Database

To store data

Execute

Queries

Plumbers: Organize and Search Petabytes

  • Automate

    • instrument-to-archive pipelinesIt is is a messy business – very labor intensiveMost current designs do not scale (too many manual steps)BaBar (1TB/day) and ESO pipeline seem promising.A job-scheduling or workflow system

    • Physical Database design & access

      • Data access patterns are difficult to anticipate

      • Aggressively and automatically use indexing, sub-setting.

      • Search in parallel

  • Goals

    • Answer easy queries in 10 seconds.

    • Answer hard queries (correlations) in 10 minutes.


Slide21 l.jpg
Q: How can a computer scientist help, without learning a LOT of Astronomy?A: Scenario Design: 20 questions.

  • Astronomers proposed 20 questions Typical of things they want to do Each would require a week (or month) of programming in tcl / C++/ FTP

  • Goal, make it easy to answer questions

  • DB and tools design motivated by this goal

    • Implemented DB & utility procedures

    • JHU Built GUI for Linux clients


The 20 queries l.jpg

Q11: Find all elliptical galaxies with spectra that have an anomalous emission line.

Q12: Create a grided count of galaxies with u-g>1 and r<21.5 over 60<declination<70, and 200<right ascension<210, on a grid of 2’, and create a map of masks over the same grid.

Q13: Create a count of galaxies for each of the HTM triangles which satisfy a certain color cut, like 0.7u-0.5g-0.2i<1.25 && r<21.75, output it in a form adequate for visualization.

Q14: Find stars with multiple measurements and have magnitude variations >0.1. Scan for stars that have a secondary object (observed at a different time) and compare their magnitudes.

Q15: Provide a list of moving objects consistent with an asteroid.

Q16: Find all objects similar to the colors of a quasar at 5.5<redshift<6.5.

Q17: Find binary stars where at least one of them has the colors of a white dwarf.

Q18: Find all objects within 30 arcseconds of one another that have very similar colors: that is where the color ratios u-g, g-r, r-I are less than 0.05m.

Q19: Find quasars with a broad absorption line in their spectra and at least one galaxy within 10 arcseconds. Return both the quasars and the galaxies.

Q20: For each galaxy in the BCG data set (brightest color galaxy), in 160<right ascension<170, -25<declination<35 count of galaxies within 30"of it that have a photoz within 0.05 of that galaxy.

Q1: Find all galaxies without unsaturated pixels within 1' of a given point of ra=75.327, dec=21.023

Q2: Find all galaxies with blue surface brightness between and 23 and 25 mag per square arcseconds, and -10<super galactic latitude (sgb) <10, and declination less than zero.

Q3: Find all galaxies brighter than magnitude 22, where the local extinction is >0.75.

Q4: Find galaxies with an isophotal surface brightness (SB) larger than 24 in the red band, with an ellipticity>0.5, and with the major axis of the ellipse having a declination of between 30” and 60”arc seconds.

Q5: Find all galaxies with a deVaucouleours profile (r¼ falloff of intensity on disk) and the photometric colors consistent with an elliptical galaxy. The deVaucouleours profile

Q6: Find galaxies that are blended with a star, output the deblended galaxy magnitudes.

Q7: Provide a list of star-like objects that are 1% rare.

Q8: Find all objects with unclassified spectra.

Q9: Find quasars with a line width >2000 km/s and 2.5<redshift<2.7.

Q10: Find galaxies with spectra that have an equivalent width in Ha >40Å (Ha is the main hydrogen spectral line.)

The 20 Queries

Also some good queries at: http://www.sdss.jhu.edu/ScienceArchive/sxqt/sxQT/Example_Queries.html


Two kinds of sdss data in an sql db objects and images all in db l.jpg
Two kinds of SDSS data in an SQL DB anomalous emission line. (objects and images all in DB)

  • 15M Photo Objects ~ 400 attributes

50K Spectra with ~30 lines/

spectrum


Q15 fast moving objects l.jpg
Q15: Fast Moving Objects anomalous emission line.

  • SELECT r.objID as rId, g.objId as gId,

  • dbo.fGetUrlEq(g.ra, g.dec) as url

  • FROM PhotoObj r, PhotoObj g

  • WHERE r.run = g.run and r.camcol=g.camcol and abs(g.field-r.field)<2 -- nearby

  • -- the red selection criteria

  • and ((power(r.q_r,2) + power(r.u_r,2)) > 0.111111 )

  • and r.fiberMag_r between 6 and 22 and r.fiberMag_r < r.fiberMag_g and r.fiberMag_r < r.fiberMag_i

  • and r.parentID=0 and r.fiberMag_r < r.fiberMag_u and r.fiberMag_r < r.fiberMag_z

  • and r.isoA_r/r.isoB_r > 1.5 and r.isoA_r>2.0

  • -- the green selection criteria

  • and ((power(g.q_g,2) + power(g.u_g,2)) > 0.111111 )

  • and g.fiberMag_g between 6 and 22 and g.fiberMag_g < g.fiberMag_r and g.fiberMag_g < g.fiberMag_i

  • and g.fiberMag_g < g.fiberMag_u and g.fiberMag_g < g.fiberMag_z

  • and g.parentID=0 and g.isoA_g/g.isoB_g > 1.5 and g.isoA_g > 2.0

  • -- the matchup of the pair

  • and sqrt(power(r.cx -g.cx,2)+ power(r.cy-g.cy,2)+power(r.cz-g.cz,2))*(10800/PI())< 4.0

  • and abs(r.fiberMag_r-g.fiberMag_g)< 2.0

  • Find near earth asteroids:

  • Finds 3 objects in 11 minutes

    • (or 52 seconds with an index)

  • Ugly, but consider the alternatives (c programs an files and…)


Data visualization and human computer interface l.jpg

Question & Answer anomalous emission line. Visualization

Tools

Data Visualization(and human-computer interface)

  • Make it easy to ask questions

  • Make it easy to understand the answers.

  • Bad news: we have had no takers on the “visualization 20 questions”

  • This is still a VERY retro area.

  • But. The following demos show some progress.


Four talks27 l.jpg
Four Talks anomalous emission line.

  • The article we actually wroteOnline Scientific Publication, Curation, Archiving

  • The advertised talk Computer Science Challenges in the VO

  • A Web Server (SkyServer) tour

  • A Web Service (SdssCutout) tour

  • These lead up to Alex Szalay’s talk on Web Services


Skyserver tour http skyserver sdss org l.jpg
SkyServer Tour anomalous emission line. http://skyserver.sdss.org/

  • Shows benefit of a database

    • everything online

    • Easy to find things – index helps

    • Automatic parallel search is essential

  • Beware:

    • I’m a lunatic re using databases for everything

    • Most people do not put images in DB

    • I do, because it is

      • Simpler

      • Easier to manage

      • The right thing to do.


Four talks29 l.jpg
Four Talks anomalous emission line.

  • The article we actually wroteOnline Scientific Publication, Curation, Archiving

  • The advertised talk Computer Science Challenges in the VO

  • A Web Server (SkyServer) tour

  • A Web Service (SdssCutout) tour

  • Leads up to Alex Szalay’s talk on Web Services


What s a web service l.jpg
What’s a Web Service anomalous emission line.

You

Web

Server

http url

  • Web SERVER:

    • Given a url + parameters

    • Returns a web page (often dynamic)

  • Web SERVICE:

    • Given a XML document (soap msg)

    • Returns an XML document

    • Tools make this look like an RPC.

      • F(x,y,z) returns (u, v, w)

    • Distributed objects for the web.

    • + naming, discovery, security,..

  • Internet-scale distributed computing

Web page

Your

program

Web

Service

soap

Data

In your address space

objectin xml


Data federations of web services l.jpg
Data Federations of Web Services anomalous emission line.

  • Massive datasets live near their owners:

    • Near the instrument’s software pipeline

    • Near the applications

    • Near data knowledge and curation

    • Super Computer centers become Super Data Centers

  • Each Archive publishes web services

    • Schema: documents the data

    • Methods on objects (queries)

  • Scientists get “personalized” extracts

  • Uniform access to multiple Archives

    • A common global schema

Federation


Grid and web services synergy l.jpg
Grid and Web Services Synergy anomalous emission line.

  • I believe the Grid will be many web services

  • IETF standards Provide

    • Naming

    • Authorization / Security / Privacy

    • Distributed Objects

      Discovery, Definition, Invocation, Object Model

    • Higher level services: workflow, transactions, DB,..

  • Synergy: commercial Internet & Grid tools


Sdss cutout http skyservice pha jhu edu sdsscutout l.jpg
SDSS Cutout anomalous emission line. http://SkyService.pha.jhu.edu/SdssCutout/

  • A simple web service

  • You can have a copy of the code

  • Needs an online database backend


Four talks34 l.jpg
Four Talks anomalous emission line.

  • The article we actually wroteOnline Scientific Publication, Curation, Archiving

  • The advertised talk Computer Science Challenges in the VO

  • A Web Server (SkyServer) tour

  • A Web Service (SdssCutout) tour

  • Leads up to Alex Szalay’s talk on Web Services


References and links l.jpg
References and Links anomalous emission line.

  • SkyServer

    • http://skyserver.sdss.org/

    • http://SkyService.pha.jhu.edu/SdssCutout/

  • Virtual Observatory

    • http://www.us-vo.org/

    • http://www.voforum.org/

  • World-Wide Telescope

    • paper in ScienceV.293 pp. 2037-2038. 14 Sept 2001. (MS-TR-2001-77 word or pdf.)

  • SDSS DB:

    • Get your personal copy athttp://research.microsoft.com/~gray/sdss


ad