1 / 46

CIAO Software and Standard Processing Pipeline for Chandra Data

Learn about CIAO (user software) and the standard processing pipeline software for Chandra data. Includes information on downloads, documentation, helpdesk, and updates.

andrealancaster
Download Presentation

CIAO Software and Standard Processing Pipeline for Chandra Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Jonathan McDowell CUC Sep 2017 CXC

  2. Chandra data and software I will report on CIAO (user software) and the standard processing pipeline software, which come into being thanks to: CXC Data Systems team: software design, development. operations/archive, etc. CXC Science Data Systems team: requirements, documentation, testing, helpdesk, interface with science community CUC Sep 2017

  3. Your SDS Contacts Current team: SAO Scientists Jonathan McDowell SDS lead, data model, coords Antonella Fruscione (½ time) SDS-SAO dep.lead, Docs and Release lead Aneta Siemiginowska Catalog, Sherpa, Astrostatistics Doug Burke Catalog, Scripts, Infrastructure, Sherpa/ChIPS, Releases Frank Primini Catalog, Photometry, Source Detection, HRC SAO IT Specialists Kenny Glotfelty Helpdesk, scripts, docs, legacy expertise Nick Lee Helpdesk, scripts, docs Bill Joye DS9 MIT Scientists (~3.5FTE) Mike Nowak SDS-MIT lead, Catalog, timing, responses Dave Huenemorder Gratings, responses [Glenn Allen: departed CXC Aug 2017] Moritz Guenther MARX CUC Sep 2017 CXC

  4. Community Support: Downloads, Documentation, Helpdesk

  5. Total CIAO downloads by month. Trickle of CIAO 4.7 32-bit Linux users.

  6. Fraction of total lifetime downloads by OS (includes P2+P3)

  7. Download stats (lifetime)

  8. Docs CIAO 4.9 - routine release rollout Replicated existing site, version updates, reset <new/> and <updated/> tags, etc. Write user-friendly release notes Updated help files etc. Rewrote point source spectrum extract Created inventory of which tools and which datasets are used in threads. Some threads could benefit from using more recent data. Some threads can be combined to reduce long term maintenance. Miscellaneous updates Use mask file for HRC exposure maps Replace unicode equations with LaTeX/MathJax Corrected steps in combined WMAP thread for single-obi combo Add copying of *_AVG keywords to blank-sky thread

  9. Docs New CIAO tools gallery automatically generated added to suite of regression tests reproject_aspect thread update using known offset using dax to get net counts add energy filter to source detect thread update dmimgthresh for NaN handling light curve thread updates including plotting delta-T

  10. New - CIAO Tools Gallery

  11. Categories so far: - Binning - Smooth - Detect - Regions - Filter - Misc

  12. Website stats: 2016-09-01 through 2017-08-31

  13. Helpdesk stats CUC Sep 2017

  14. Top Helpdesk Topics Archive FTP server name change (download_chandra_obsid) ChaRT crash Conflicts with other packages Working with 3rd party packages (astropy) Python path Dynamic libraries, ‘LD_LIBRARY_PATH’ Getting an openGL enabled environment for ChIPS (remote display) General PSF questions Does HETG affect PSF Practical limit on EDSER resolution Number of realizations

  15. Examples of bugs Chips smoketests fail on OSX Sierra Plotting in sherpa too Bad Period G blank sky background files Python3 ecf_calc, mktgresp, FOVFile dmimgpick wrong events tagged, bad filtering crash with bad CONTINUE keywords std::erase Abort Problem in CALDB library causing random failures reproject_image_grid parse error for Dec= -0:mm:ss problem running axbary from via runtool (python) repro OBC mode data

  16. Social Media: as of 2017-08-31

  17. Community Summer AAS HEAD 2017 (and Astrostatistics session) Planning invited Pune, India workshop

  18. CALDB Releases • SDS supports all CALDB releases: • test the downloading of the files • test that the files work with CIAO tools • Update threads, add new threads, add “Why” documents etc as needed to reflect changes in calibration data and in methods of applying them • Add a section in the release notes “How CALDB x.x Affects Your Analysis” • Crucial extra help for users: do my data need to be reprocessed because of a given calibration change? How much is the change for a typical user? • In the reporting period: CALDB 4.7.4 to 4.7.6 - New TGAIN - HRC-I QE - HRC-S Badpix and QEU - Time/Temp dependent CTI - Group G blank sky background files - Format fixes to older blank sky background files CUC Sep 2017 CXC-SDS

  19. CIAO 4.9/4.10 and Scripts Overview

  20. CIAO Release CIAO 4.9 released as planned in December, including Python2 & Python3 flavors - Python 3 initial version (scripts, Sherpa, ChiPS) - acis_streak_map: enhancements - dmellipse: table support - dmextract: bug fix involving dead-time file - mkwarf fix to support HRC lower energy boundary - wcs_update: enhancements (e.g. support for filtering input WCS file, quaternion values) - Sherpa updates, including save_all improvements - dmdiff improvements CUC Sep 2017 CXC-SDS

  21. CIAO Release This year, top priority remains Catalog Release 2 support • CIAO 4.9.1 • Dec 2017 patch release. • Obsvis cal data update • Hope to validate support of OS-X High Sierra and CentOS7.5 • Hope to fix Chips display bug on OSX Sierra which is preventing users from using Sherpa (&Chips) on Sierra/High Sierra Macs • CIAO 4.10 • Dec 2017 release postponed to Apr 2018 to accommodate Catalog schedule • High Sierra support • XSPEC model update • Sherpa bug fixes: RMF/ARF header handling for XMM,Swift (AREASCAL, etc) User statistics, sample_flux, parallel_map • CFITSIO and DM fixes (CONTINUE keyword problem, filtering/binning issues, off-image region filter issue) • Caldb index error - Python 3 cleanup Fix to issue with @rpath, LD_LIBRARY_PATH * Later CIAO releases: HRC coordinates work CUC Sep 2017 CXC-SDS

  22. wavdetect duplicate source bug Identified situation where nearby correlation maxima can migrate to the same local flux maximum leading to duplicate detections # POS(X,Y) PSF_SIZE 4906.0868761553 4361.3844731978 13.2910089493 4906.1481481481 4361.3447971781 14.1166629791

  23. dmcontour edge of field dmcontour has an issue when the polygon touches the edge of the image. Left: CIAO 4.9: The contour created actually excludes the bright emission (note polygon traces along outside edge of image. Right: Fixed version

  24. skyfov algorithm review Center: FOV boundary, red=current (corresponding to square points in left plot), cyan=possible (corresponding to square points in right plot). Requires going from a fixed 4-point solution to a variable N-point convex hull.

  25. Script releases 4.9.1 Dec 2016 - specextract: if using background responses, background ARF calculation uses weighting 4.9.2 Apr 2017 - srcflux: fix bug in HRC background calc - combine_spectra: improved error calc - download_obsid_caldb bug fix - blanksky: improve output header to support reprojection - specextract: fix crash in HRC case - Fix Sherpa XSPEC convolution interface to match CIAO4.9 Sherpa 4.9.3 May 2017 - Fix download_chandra_obsid, find_chandra_obsid to use the new ftp server ** cdaftp.cfa.harvard.edu is retired, must now use cda.cfa.harvard.edu ** - Fix some issues with python 3.5 compatibility 4.9.4 Jul 2017 - DS9/DAX CIAO->Statistics->Photometry supports use of different aper-corr methods’ - chandra_repro errors out more gracefully on zero-livetime observations - Maintenance fixes to various scripts

  26. Sherpa CUC Sep 2017 CXC

  27. Sherpa 2017 Development • Sherpa 4.9 for CIAO was released on Dec. 16, 2016. • This is the first CIAO release based on the GitHub-developed version of Sherpa. • Sherpa now runs under Python 2.7 and Python 3.5 . • The highlights include: • the update of the XSPEC models to version 12.9.0d; • the addition of the wstat statistic, for including a background dataset as a model component (it is based on the XSPEC version); • support for background data sets in user statistics created with load_user_stat; • a major upgrade to the Python docstrings (the ahelp pages are still available in this release). • Sherpa 4.9.0 standalone was released on Feb.1, 2017. • This release is for both Python 2.7 and Python 3.5. • It fixes many bugs in the Python 3 support. • It includes a significant refactoring of the Fit and Stat classes that made it possible to fix several bugs related to the recent wstat implementation while making these classes more maintainable and extensible. • Sherpa 4.9.1 standalone was released on Aug.4, 2017 • Full support for Python 3.6 • Bug fixes to non-Chandra response and PHA files. • Improvements to documentation and testing environments. • Detailed release notes on: https://github.com/sherpa/sherpa/releases CUC Sep 2015 CXC

  28. Sherpa GitHub Page Activity summary and contributors: Sherpa GitHub Release Page - August 2017

  29. Using Sherpa in Astronomy Research 927 publications in ApJ, AJ, MNRAS and A&A use Sherpa (since 2001 and including astro-ph abstracts in 2017) - 202 citations to Freeman et al 2001 SPIE paper 66 research papers in 2016, 29 papers in 2017 5 PhD theses listed in ADS that used Sherpa

  30. Sherpa Workshops Session at the AAS 229 Winter Meeting on Jan.4, 2017: Flexible Multi-dimensional Modeling of Complex Data in Astronomy:see theWorkshop Web Page Participated in the DEMOFest Event at CfA on February 14, 2017 Workshop at the CfA Library - two sessions on Jan. 31 and Feb. 3 see the Workshop Web page Astrostatistics session at HEAD meeting, Aug 2017 hea-www.harvard.edu/AstroStat/cfa_workshops/sherpa-iris-20170127

  31. PSF

  32. CUC Sep 2015 CXC ChaRT 2 ChaRT is a web interface to the CXC Optics group's mirror simulator: SAOTrace.

  33. ChaRT Stats

  34. ChaRT stats

  35. MARX simulator MARX: Documentation updates continue, no major changes

  36. CUC Sep 2015 CXC Visualization

  37. CUC Sep 2015 CXC SAOImage ds9 ds9 is an imaging program widely used in the astronomical community, well beyond X-ray astronomy. It was the successor to the earlier, also widely adopted, SAOImage program ds9 is a key part of the Chandra data analysis infrastructure – it directly supports X-ray event files.

  38. CUC Sep 2015 CXC ds9 status ds9 Version 7.5 – release coordinated with CIAO 4.9 as planned. Contents as described in last CUC. - Infrastructure work completed for 2017: Source repo now maintained on Github Test scripts and data now on Github Bug reports, feature requests tracked via Github Tcl/Tk support packages updated to TEA 3.10 Simplified build process unifying Linux, Mac, Windows builds Windows 10 port completed including IRAF IIS support, FITS bintable support 2017 development (Current beta release 7.6b5) New Windows (64 & 32 bit) ports HTTPS support Improved horizontal & vertical graphs Improved FITS compression support Bug fixes Ongoing development - Implementing support for new FITS WCS spectral standards (Paper II, III, IV, naxes > 2) - 178 help requests (41 SAO, 137 external)

  39. CUC Sep 2015 CXC DS9 downloads 2016 Sep – 2017 Aug Total downloads in past year (Sep 2016 to Aug 2017) = 44667 (previous year was 43057)

  40. Gratings

  41. Gratings work Reanalysis of long observations of faint sources and stowed background planned to generate improved estimates of HETG/ACIS and LETG/ACIS backgrounds for the POG. Improved HRC-S QEU (was clipping events near detector plate boundaries).

  42. TGCAT updates TGCAT updates continue - Porting to new host, requires rewriting for modern versions of php/mysql - Interactive plotting improvements being worked on CUC Sep 2017 CXC

More Related