1 / 25

VSO Status Update

VSO Status Update. HDMC Meeting 9 June 2008. Vision. To allow solar physicists to identify and search for data even if they don't know it exists. Make it easier for them to get to the data whether they knew where it was before or not. Why VSO?. Why not CDAWeb?

lola
Download Presentation

VSO Status Update

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VSO Status Update HDMC Meeting 9 June 2008

  2. Vision • To allow solar physicists to identify and search for data even if they don't know it exists. • Make it easier for them to get to the data whether they knew where it was before or not.

  3. Why VSO? • Why not CDAWeb? • Will they take > 2TB/day of data? • Why not FTP and VSPO: • Do you want to scan > 70k files per day to determine which ones meet your needs? • Every time you perform a search? (these numbers are just for SDO)

  4. How to accomplish: • Standardized API for querying & requesting data using a consistent model at each provider • System to allow UIs to query a single API and have the queries distributed & responses aggregated.

  5. Current Status • Data Providers : 14 • Observatories : 26 • Instruments : 55 • Data Sets : 180? ... dating back to 1915

  6. Current Status (SPASE terms) • Repositories : 19 • Observatories : 26 • Observatory Groups : 3 • Instruments : 66+ • Instrument Groups : ~10 • Numerical Data : ~180 products • Display Data : (not tracked directly) • Granules : ? millions

  7. Data Sets Planned: • In progress: • Hinode/SOT • Huairou • NRL EIT & LASCO • TIMED/SEE • Planned: • Hinode/EIS • SMART • SDO ... and catalogs

  8. SPASE use: • um ... no?

  9. Current Interfaces / APIs • VSO SOAP API (finding & ordering): • Web-based interface • 3 custom VSO-derived interfaces (NSO, SAO, SSC) • IDL-based interface • Dominic Zarro's IDL commands in SSW • Perl command line interface • AstroGrid • Registry API • Catalog 'Display' API for HelioViewer

  10. Interfaces / APIs • In progress: • IDL client for Registry API • More complex Catalogs API(s) • Planned: • Cart API • and UI in IDL • Catalog APIs for HELM • access via IDL • SPASE compliant registry API • or at least SPASE descriptions of holdings

  11. Data Coverage: • Incomplete. • Measurement method: • People are still asking us to add data sets.

  12. Goals: • Prepare for the flood of SDO data • (and other recent missions: STEREO, Hinode )

  13. Plans for the Future: • Catalogs • Summary Rows • Looking to the work of others

  14. Empowering new searches! We’re doing some work under the hood!

  15. Moving beyond time • An event or feature driven approach • Science catalogs • LASCO CME, GOES X-ray Flare, NOAA Active Region Number, Coronal Holes, Bright points etc. • Science queries One of the drivers for this • SDO • So much data need to move to feature / event driven approach, • But useful now …

  16. Give me all of these! Actually I really want all XRT jets observed on the limb with coincident SOT and EIS data. Would have made things much easier if I could have performed this query!

  17. Science Queries How do we know what queries to support? • We get asked! • “It would be nice if … / I’d really like to be able to …” Coincident observations • Observations of same features in multiple instruments within specified time window. Cadence • Find an hours worth of observations where there are images every minute. (Helioseismology) DEM Datasets • Find XRT datasets suitable for DEM analysis. (Closely spaced observations, number of different filters/wavelengths). EIS data would be a bonus!

  18. Science Queries Investigating EIT dimming regions as proxies for CMEs • Combine GOES Flare List + LASCO CME list to generate list of on disk flares without associated CMEs. Compare list to catalog of EIT dimming regions. Do spatial comparison. Generate list of EIT dimming regions whose properties are observed to be associated with CME list. • Currently waiting on EIT dimming region catalog! Can do the first part to at least generate list of EIT images that should be analyzed. • Support driven by science questions!

  19. Conclusions • We are moving from just data discovery in a certain time range, to finding datasets of interest that fit a number of scientific criteria. • Searches for features and events and in the process blurring the distinction between data and catalogs. • SDO will require these new approaches. • But it’s going to return a LOT of data.

  20. Plans : Summary Rows • VSO data providers can return a record summarizing multiple data granules • Data Providers can define their own groupings, UIs can decide to expand them. • Summary products still work like regular products • Data providers serve a tarball or data cube in response to a data transfer request • UIs that don’t understand summary products can still retrieve the collection • Changing over to a new JavaScript toolkit to provide AJAX summary row expansion

  21. Plans : Look to Others • Catalogs • Bobby Candey's HELM project (VxO funded) • Peter Fox's SESDI project (NASA funded) • Related IVOA & astronomy standards • Presentation / UI • Use existing JavaScript toolkits • ExtJS and GWT ... still requires customization to serve our needs, but gives head start

  22. Plans : Look to Others (2) • Massive amounts of data: • How libraries handle interlibrary loan • How libraries handle 'de-duplicating' of records • Functional Requirements of Bibliographic Records … is going to require significant changes at the data providers.

  23. The Future of VSO • We are moving from just data discovery in a certain time range, to finding datasets of interest that fit a number of scientific criteria. • Searches for features and events and in the process blurring the distinction between data and catalogs. • SDO will require these new approaches.

  24. http://virtualsolar.org/

More Related