SDO Data Access and Distribution in Europe and the WisSDOm Data Centre in ROB, Brussels

1. Authors David Boyes, Benjamin Mampaey, Cis Verbeeck, Veronique Delouille, Jean-François Hochedez STCE + ROB SDO Data Access and Distribution in Europe and the WisSDOm Data Centre in ROB, Brussels Cospar10 Bremen

2. Cospar10 Bremen What will be covered • Where is the data and the access architecture for the users • Some basic terms • User access methods • modules • basic web access • virtual observatories • simplified web access • pseudo files and other developments • Interesting issues • Retention • Saved searches • Evolving calibration • Neat stuff to come • Cutouts • Helioviewer • Grid integration

3. Where is the data for the users • Data is available from one or more data centre(s) - all are networked • Some users are "close", some are "far" - distance matters • All data is available somewhere • Users can get data (an "export") • from the nearest centre directly • via the nearest centre from a remote centre • directly from another centre • Most of this is automatic • you will see differences in e.g. delays Cospar10 Bremen

4. How the data is accessed(a bit technical) • the system is the netDRMS • created by the JSOC at Stanford • files are generated by content • system holds data files + metadata • SUMS + DRMS • mediator is an "export" module • makes your very own file • FITS, tar of FITS etc. • SQL etc. is hidden from user Cospar10 Bremen

5. Cospar10 Bremen Access summary ... • No files until you ask for them • Data is referenced by content - provided as a file(s) with whatever name you want • The exported files are built using stored elements, so e.g. FITS with Rice compression quite direct as AIA data is stored internally in this format • Can get anything but... • you may as well ask for all metadata • the files can be large - best not to ask for 100's

6. Cospar10 Bremen Some basic terms • series • basic collection of data items with shared properties • by convention named <project>.<data> • all series records share a metadata format (i.e. keywords) • keywords • FITS style keywords plus added metadata only keywords • correspond to columns in the metadata (DRMS) database • online means • available from a disk at the site • so offline means : not yet arrived/available, deleted but can be fetched • data format • whatever is stored is native (FITS, JP2000), conversion is post-processing • characterised by resolution, cadence (e.g. 4K x 4K at 10s, 1K x 1K at 90s) • naturally can't do better, but can reduce by "cutouts" in time or space • data records • can be several items as a group (e.g. image + bad pixel map + alternative format) • data is SUMS plus metadata, referenced by metadata tables (DRMS) - usually one to one • each is self contained, for example cadence is not part of data

7. Cospar10 Bremen Example series • aia_test.lev1 AIA images 4Kx4K full disk full cadence • aia_test.synoptic2 AIA images reduced to 1Kx1K full disk and 90s cadence • hmi_test.M_45s magnetograms, 45s cadence • hmi_test.v_45s dopplergrams, 45s cadence • jpeg2K to come, browsing and forecasting

8. Cospar10 Bremen User access methods • Direct via “modules” • on site of data centre • Query based • precursor to full data access • checks a part of the data (metadata) without having to retrieve the very large part • Indirect via network • web/http based • delivers data somewhere - maybe to fetch immediately or later • Direct via wrapper • on site e.g. IDL (Matlab on way)

9. Cospar10 Bremen A practical pause - limitations • Sheer size of request - even if you have a 2TB USB stick, that's only 2 days • Network speed - at about 200Mb/s it takes a day to get a day's worth • Search/database speed - millions of records • Raw data access/retrieval speed - the basic image data takes time to get from disk • Retention time - you can get anything, but you probably have to wait for a full day from 2 years ago that nobody else has ever used

10. Cospar10 Bremen Access by : modules - the basic bricks • At the data centres, for example • show_series • show_info • jsoc_export_as_fits [jdb@db1 ~]$ show_series aia_test.lev1 aia_test.synoptic2 drms.sites hmi.doptest hmi_test.m_45s hmi_test.s_720s lm_jps.lev1_test4k10s [jdb@db1 ~]$ show_info -s ds=aia_test.synoptic2 First Record: aia_test.synoptic2[2010-05-21T15:00:00.57Z][171] is first of 6 records matching first keyword, Recnum = 1 Last Record: aia_test.synoptic2[2010-07-14T11:58:41.07Z][335] is first of 2 records matching first keyword, Recnum = 445376 Last Recnum: 445377 [jdb@db1 ~]$ jsoc_export_as_fits reqid=REQ_FTP expversion=0.5 rsquery=aia_test.lev1[:#209866] path=tmp method=url protocol=FITS '10552320' bytes exported.

11. Cospar10 Bremen Access by : basic web access • System developed by JSOC : lookdata.html • Online via JSOC web site, but heavily loaded • Being tested at ROB • Provides an easy access to an overview of all the available data • Formulating a selection query does require knowledge of query syntax • Provides for a wide variety of data packaging • normal user FITS or internal format (FITS with no keywords) • via web for immediate or later access, as one or more individual files or as tar • ROB working on fewer packaging options

12. Access by : basic web access Cospar10 Bremen

13. Cospar10 Bremen Access by : Virtual Observatories • VSO • development of existing VSO • prototype for SDO running and definitive version in preparation • http://sdac.virtualsolar.org/cgi/search • Soteria • demo provider made for ROB/USET, SDO provider being coded now • http://soteria-space.eu/ • Uniform search paradigm • Infrastructure hides efficient searches with complex syntax e.g. SQL in various flavours

14. Access by : Soteria Virtual Observatory • One part of an EU project • Based on current web access technology • The example is for the ROB USET telescope as a data provider, each SDO site will able be able to act as a provider Cospar10 Bremen

15. Cospar10 Bremen Access by : simplified web access • Work in progress • Limited offer to direct request of tar files or individual FITS format files, front end for PFS • Simplified enquiry based such as : • aia.lev1 + time + period + cadence + wavelengths • Preparation is actually more complex than basic access - for example it requires decisions as to what keys are useful for what series

16. Access by : pseudo files (PFS) • Systematically named files in a directory tree with no real files until you access them • Typically based on query covering a much wider range than you really need (or could use) • Real files kept in cache so further access very cheap Cospar10 Bremen

17. Cospar10 Bremen Access by : pseudo files (PFS) • Example with 160 file names, all AIA wavelengths, 15min cadence • In prototype at ROB, source downloadable mnt `-- aia_test.lev1 `-- 2010 `-- 06 `-- 17 |-- H0000 | |-- AIA20100617_000000570000_0171.fits | |-- AIA20100617_000003570000_0304.fits | |-- AIA20100617_000009580000_94.fits | |-- AIA20100617_000018570000_1600.fits | |-- AIA20100617_000050070000_211.fits | |-- AIA20100617_000053050000_335.fits | |-- AIA20100617_000056100000_193.fits ...... | |-- AIA20100617_004505070000_335.fits | |-- AIA20100617_004506570000_1600.fits | |-- AIA20100617_004508070000_193.fits | |-- AIA20100617_004509580000_94.fits | `-- AIA20100617_004511070000_131.fits |-- H0100 | |-- AIA20100617_010000580000_0171.fits | |-- AIA20100617_010002080000_211.f ....... |-- AIA20100617_043008060000_193.fits |-- AIA20100617_043009550000_94.fits |-- AIA20100617_043011090000_131.fits |-- AIA20100617_043018580000_1600.fits |-- AIA20100617_044500560000_0171.fits |-- AIA20100617_044502050000_211.fits |-- AIA20100617_044503570000_0304.fits |-- AIA20100617_044505070000_335.fits |-- AIA20100617_044506570000_1600.fits |-- AIA20100617_044508070000_193.fits |-- AIA20100617_044509580000_94.fits `-- AIA20100617_044511070000_131.fits 9 directories, 160 files

18. Cospar10 Bremen Access by : useful methods in development • Order and notify via e-mail for manual fetch • Order and automatic delivery (e.g. sftp)

19. Cospar10 Bremen Interesting issue - Retention • All netDRMS sites have full information for selected series - their “subscribed” series • But is it on line? • sites keep the latest, but must selectively discard • Enquiry modules can tell if online, but implications (delay...) if not? • You can request it, but it can take some time to obtain • for now quick, but after a year or so a record nobody has looked at will be from tape

20. Cospar10 Bremen Interesting issue - Saved searches • How to describe a selection of data • Can save result as a record list for a reasonable number of records but this does not save the query • save both query and result? • For both your own use and publication • Saved query might give different results (e.g. online only) • Relates to the issue of calibration

21. Cospar10 Bremen Interesting issue - Evolving calibration and which data did I use? • More accurate calibration will be available as time goes on and more calibration points are acquired • So the newest and best data can change • This done for most by applying a calibration series e.g. via Solarsoft • But there can also be metadata changes • The raw data is unlikely to change

22. Cospar10 Bremen Neat stuff to come - cutouts • This is well on the way again being developed by JSOC and LMSAL - for those who don't need the full 4Kx4K • Very much reduced data storage requirements • Closely related to event tracking and the HEK

23. Neat stuff to come - Helioviewer • www.helioviewer.org • Existing project now being directed towards use with SDO data • JPEG2000 based viewer with event marker overlay • integration with JPEG2000 series • rapid browsing with links to full data • ROB is CoI in requested next stage Cospar10 Bremen

24. Cospar10 Bremen Neat stuff to come - grid integration • The data element size (10's of MB) is natural for use in a high performance grid • The data already geographically distributed - variety of access routes • Distributed variety of resources - large clusters, pipelines, GPU's • Sites are on high performance research networks

25. Cospar10 Bremen Thanks to • JSOC at Stanford • LMSAL • Belnet and Geant2 for networking • The enthusiastic cooperation from the partner data centres • Our sister institutes at the ROB site for hosting the data centre and infrastructure

26. Cospar10 Bremen Web addresses • The main source : JSOC at jsoc.stanford.edu • HEK : www.lmsal.com/hek • ROB : wissdom.oma.be • SAO : www.cfa.harvard.edu/sao • GDS : www.mps.mpg.de/projects/seismo/GDC-SDO • UCLan : www.star.uclan.ac.uk • IAS : idc-medoc.ias.u-psud.fr