1 / 12

Post-processing Overview

Post-processing Overview. Bryan Butler (& Joe McMullin) (NRAO-AOC). Mission Primary purpose is to provide facilities for post-observational scientific reduction of ALMA/EVLA data Requirement documents: ALMA Offline Data Processing Requirements AIPS++ Audit

deanne
Download Presentation

Post-processing Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Post-processing Overview Bryan Butler (& Joe McMullin) (NRAO-AOC)

  2. Mission Primary purpose is to provide facilities for post-observational scientific reduction of ALMA/EVLA data Requirement documents: ALMA Offline Data Processing Requirements AIPS++ Audit EVLA Data Post-Processing Software requirements Uses the legacy AIPS++code base for development AIPS used for validation, algorithm development (e.g., automated flagging) Requirement Key Areas General Requirements and Interaction Interface Data Handling Calibration and Editing Imaging Data Analysis Visualization Special Features Spreadsheet/tracking on the combined projects’ requirements (common to both projects and the deltas between projects) Science Software Group

  3. Management IPT Organization G. van Moorsel ((e)VLA/VLBA) B. Glendenning (ALMA), Michael Rupen (EVLA) & Debra Shepherd (ALMA) – Project Scientists for Computing J. McMullin, K. Golap (Project Management) S. Myers (SSG Project Scientist) Priorities (ALMA/EVLA) Scientific Completeness/Correctness, Usability Robustness, Performance Organization Resources • NRAO: 6.0 FTEs (including management; split 50/50 with ALMA) • ALMA: 5.5 FTEs (Application development, Data Model, Data Model Interfaces, Data Capture process) • NSF grant: 1.0 FTE (visualization; concludes 2007.8)

  4. Resources: Details

  5. Status Multi-year plan for development of all Priority 1 ALMA science software requirements (2007.5) Project Office Page for development planning, accounting, transparency http://projectoffice.aips2.nrao.edu 18 two-month development cycles completed Established strong feedback loop with internal NRAO scientists Established external scientist testing/verification Infrastructure RPM distribution Framework (slide) Milestones Yearly NRAO VC and UC meetings ASAC, EVLA ACM meetings ALMA CDR1 (Jun 03) - Passed ALMA TST1 (Jan 04) – Passed Single Field Interferometry ALMA CDR2 (Jul 04) - Passed ALMA R2 (Oct 04) - Complete ALMA TST2 (Nov 04) – Passed +Mosaics ALMA TST3 (Apr 05) – Passed +synthesis/single dish combo ALMA R2.1 (Apr 05) - Complete EVLA TST1 (Jun 05) – Passed +Wide Field Imaging ALMA CDR3 (Jul 05) - Passed ALMA R3 (Oct 05) - Complete +ATF simulation (1st CASA test) ALMA R3.1 (Mar 06) - Complete +User Interface Charette Provided testing/detailed requirements SSG Development

  6. 2003.5 Long-term plan 2004 Science: Imaging performance enhancements Science: Flexible continuum subtraction Framework: CVS code management Data Interfaces: Draft AEDF document 2004.5 Science: SF reduction (TST1) Science: Calibrater performance enhancements Science: Calibrater flexibility (interpolations, transfer, etc). Framework: Framework requirements, plan Data Interfaces: AEDF submitted for review, data capture design 2005 Science: Mosaic reduction (TST2) Science: Automated, interactive data editing Framework: Library re-organization Data Interfaces: Data Capture process 2005.5 Science: Single dish/interferometric (TST3) Science: Simulator (telescope modeler, sampler, visibility simulator) Framework: Beta, subset, distribution with ACS Data Interfaces: DRP filler 2006 Science: Complex mosaics (TST4) – changed to ATF (single-baseline) test Science: Full polarized primary beam imaging Framework: Initial CLI, Python interface – added a working group evaluation of the interface 2006.5 Science: Single Dish (TST5) Science: Flexible line fitting Framework: Testing with new framework – 1st done in first half 2006 2007 Science: Simulations I (TST6) Science: Multi-frequency synthesis Framework: GUI development Long-term Schedule (Presented at CDR2 in July 2004)

  7. 2006 NRAO: NAUG testing (AIPS++) Community: Project tests ALMA: external science testing: +single-baseline commissioning test) (CASA) ALMA: external science testing: +single dish reduction (CASA) ALMA: commissioning support (CASA) ALMA/EVLA: user interface review (CASA) EVLA: external science testing: +full polarization imaging; antenna pointing calibration 2006.5 AIPS++ frozen ALMA: Pipeline Heuristics Use of CASA 2007 NRAO: User support (CASA) ALMA: commissiong support (CASA) EVLA: external science testing: + RFI/automated flagging (CASA) ALMA: P1 SSRs complete 2008 EVLA: external science testing: + wide band calibration imaging (CASA) NRAO: user support (CASA) ALMA: commissioning support EVLA: P1 SSRs complete 2009-2010 ALMA/EVLA: commissioning support Community: CASA released/distributed for early ALMA/EVLA science 2011 Community: CASA released/distributed; full user support (ALMA/EVLA: P1/P2s) Coarse Timeline & Staging of Community Usage/User Support of CASA

  8. AIPS++ Glish interface; unknown, unsupported outside of NRAO Tasking system based on Glish GUI system based on Glish/Tk; limited widgets, not robust! Difficult for external developers to contribute Multi-CD binary distribution Large monolithic libraries with cross dependencies No namespace Freeze 2nd half of 2006 CASA Python interface (community standard);IPython Binding to Python, ACS; other frameworks readily possible Hierarchical set of small libraries with clearly defined dependencies Namespace protection for integration with other code RPM distribution mechanism; auto-updates possible Robust Inherits all application code improvements in robustness and performance. Smaller memory footprint/startup time Framework Migration

  9. CASA usage status: • Has been tested internally by NRAO scientists in preparation for the 1st ALMA test. • Early demos provided bi-monthly to the NAUG at: • http://casa.nrao.edu/gettingstarted.shtml • 6 scientists used and reported on it – enabled deployment for the ALMA test. • Was deployed and reviewed by four ALMA testers • http://projectoffice.aips2.nrao.edu/ALMA2006.01/ALMA2006.01.html • The user interface was reviewed and commented on by 8 NRAO scientists • User Interface is being refined/further developed based on the user interface report: • http://projectoffice.aips2.nrao.edu/uiwg_report.pdf • Will be used exclusively by the ALMA Pipeline Heuristics Team (second half of 2006) • CASA will have replaced AIPS++ within NRAO (developers and NAUG testers) this year.

  10. User Interface • User Interface review • IPython interface to functionality, prototype parameter setting interface and in-line help are in the right direction but: • Full needs are documented at: • http://projectoffice.aips2.nrao.edu/uiwg_report.pdf • Revised interface prototype (similar to IRAF epar environment); high priority • Much work needed to provide astronomer-level documentation (collaboration with NAUG to develop this); high priority • Development progress will be reviewed with the Fall EVLA test.

  11. Risks • Staffing • Mitigation strategy: efficient use of existing resources; leverage off of ALMA development as much as sensible • User Support • Mitigation strategy: Work with group of NAUG scientists such that they can provide the front-line help. • User Interface • Mitigation strategy: Cyclic feedback on progress and prototypes (revisited in June-July 2006; will be a review aspect of every formal/informal test) • Requirement rot • Mitigation strategy: Update of EVLA requirements document to reflect current planning/time tables. • ASDM • Mitigation strategy: Multiple evaluations, early testing of needed modifications. • Algorithm development/data rates – SB talk

  12. Overall Risk Mitigation • Backup plan: AIPS • Requires: • Continued support and targeted development • Support current VLA/VLBA users • Develop and deploy automated flagging algorithms to community for testing/review

More Related