Post processing overview
This presentation is the property of its rightful owner.
Sponsored Links
1 / 12

Post-processing Overview PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Post-processing Overview. Bryan Butler (& Joe McMullin) (NRAO-AOC). Mission Primary purpose is to provide facilities for post-observational scientific reduction of ALMA/EVLA data Requirement documents: ALMA Offline Data Processing Requirements AIPS++ Audit

Download Presentation

Post-processing Overview

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Post processing overview

Post-processing Overview

Bryan Butler (& Joe McMullin)


Science software group


Primary purpose is to provide facilities for post-observational scientific reduction of ALMA/EVLA data

Requirement documents:

ALMA Offline Data Processing Requirements

AIPS++ Audit

EVLA Data Post-Processing Software requirements

Uses the legacy AIPS++code base for development

AIPS used for validation, algorithm development (e.g., automated flagging)

Requirement Key Areas

General Requirements and Interaction


Data Handling

Calibration and Editing


Data Analysis


Special Features

Spreadsheet/tracking on the combined projects’ requirements (common to both projects and the deltas between projects)

Science Software Group



IPT Organization

G. van Moorsel ((e)VLA/VLBA)

B. Glendenning (ALMA),

Michael Rupen (EVLA) & Debra Shepherd (ALMA) – Project Scientists for Computing

J. McMullin, K. Golap

(Project Management)

S. Myers (SSG Project Scientist)

Priorities (ALMA/EVLA)

Scientific Completeness/Correctness, Usability

Robustness, Performance



  • NRAO: 6.0 FTEs (including management; split 50/50 with ALMA)

  • ALMA: 5.5 FTEs (Application development, Data Model, Data Model Interfaces, Data Capture process)

  • NSF grant: 1.0 FTE (visualization; concludes 2007.8)

Resources details

Resources: Details

Ssg development


Multi-year plan for development of all Priority 1 ALMA science software requirements (2007.5)

Project Office Page for development planning, accounting, transparency

18 two-month development cycles completed

Established strong feedback loop with internal NRAO scientists

Established external scientist testing/verification


RPM distribution

Framework (slide)


Yearly NRAO VC and UC meetings

ASAC, EVLA ACM meetings

ALMA CDR1 (Jun 03) - Passed

ALMA TST1 (Jan 04) – Passed

Single Field Interferometry

ALMA CDR2 (Jul 04) - Passed

ALMA R2 (Oct 04) - Complete

ALMA TST2 (Nov 04) – Passed


ALMA TST3 (Apr 05) – Passed

+synthesis/single dish combo

ALMA R2.1 (Apr 05) - Complete

EVLA TST1 (Jun 05) – Passed

+Wide Field Imaging

ALMA CDR3 (Jul 05) - Passed

ALMA R3 (Oct 05) - Complete

+ATF simulation (1st CASA test)

ALMA R3.1 (Mar 06) - Complete

+User Interface Charette

Provided testing/detailed requirements

SSG Development

Long term schedule presented at cdr2 in july 2004


Long-term plan


Science: Imaging performance enhancements

Science: Flexible continuum subtraction

Framework: CVS code management

Data Interfaces: Draft AEDF document


Science: SF reduction (TST1)

Science: Calibrater performance enhancements

Science: Calibrater flexibility (interpolations, transfer, etc).

Framework: Framework requirements, plan

Data Interfaces: AEDF submitted for review, data capture design


Science: Mosaic reduction (TST2)

Science: Automated, interactive data editing

Framework: Library re-organization

Data Interfaces: Data Capture process


Science: Single dish/interferometric (TST3)

Science: Simulator (telescope modeler, sampler, visibility simulator)

Framework: Beta, subset, distribution with ACS

Data Interfaces: DRP filler


Science: Complex mosaics (TST4) – changed to ATF (single-baseline) test

Science: Full polarized primary beam imaging

Framework: Initial CLI, Python interface – added a working group evaluation of the interface


Science: Single Dish (TST5)

Science: Flexible line fitting

Framework: Testing with new framework – 1st done in first half 2006


Science: Simulations I (TST6)

Science: Multi-frequency synthesis

Framework: GUI development

Long-term Schedule (Presented at CDR2 in July 2004)

Coarse timeline staging of community usage user support of casa


NRAO: NAUG testing (AIPS++)

Community: Project tests

ALMA: external science testing: +single-baseline commissioning test) (CASA)

ALMA: external science testing: +single dish reduction (CASA)

ALMA: commissioning support (CASA)

ALMA/EVLA: user interface review (CASA)

EVLA: external science testing: +full polarization imaging; antenna pointing calibration

2006.5 AIPS++ frozen

ALMA: Pipeline Heuristics Use of CASA


NRAO: User support (CASA)

ALMA: commissiong support (CASA)

EVLA: external science testing: + RFI/automated flagging (CASA)

ALMA: P1 SSRs complete


EVLA: external science testing: + wide band calibration imaging (CASA)

NRAO: user support (CASA)

ALMA: commissioning support

EVLA: P1 SSRs complete


ALMA/EVLA: commissioning support

Community: CASA released/distributed for early ALMA/EVLA science


Community: CASA released/distributed; full user support (ALMA/EVLA: P1/P2s)

Coarse Timeline & Staging of Community Usage/User Support of CASA

Framework migration


Glish interface; unknown, unsupported outside of NRAO

Tasking system based on Glish

GUI system based on Glish/Tk; limited widgets, not robust!

Difficult for external developers to contribute

Multi-CD binary distribution

Large monolithic libraries with cross dependencies

No namespace

Freeze 2nd half of 2006


Python interface (community standard);IPython

Binding to Python, ACS; other frameworks readily possible

Hierarchical set of small libraries with clearly defined dependencies

Namespace protection for integration with other code

RPM distribution mechanism; auto-updates possible


Inherits all application code improvements in robustness and performance.

Smaller memory footprint/startup time

Framework Migration

Casa usage status

CASA usage status:

  • Has been tested internally by NRAO scientists in preparation for the 1st ALMA test.

    • Early demos provided bi-monthly to the NAUG at:


    • 6 scientists used and reported on it – enabled deployment for the ALMA test.

  • Was deployed and reviewed by four ALMA testers


  • The user interface was reviewed and commented on by 8 NRAO scientists

    • User Interface is being refined/further developed based on the user interface report:


  • Will be used exclusively by the ALMA Pipeline Heuristics Team (second half of 2006)

  • CASA will have replaced AIPS++ within NRAO (developers and NAUG testers) this year.

User interface

User Interface

  • User Interface review

    • IPython interface to functionality, prototype parameter setting interface and in-line help are in the right direction but:

    • Full needs are documented at:


  • Revised interface prototype (similar to IRAF epar environment); high priority

  • Much work needed to provide astronomer-level documentation (collaboration with NAUG to develop this); high priority

  • Development progress will be reviewed with the Fall EVLA test.



  • Staffing

    • Mitigation strategy: efficient use of existing resources; leverage off of ALMA development as much as sensible

  • User Support

    • Mitigation strategy: Work with group of NAUG scientists such that they can provide the front-line help.

  • User Interface

    • Mitigation strategy: Cyclic feedback on progress and prototypes (revisited in June-July 2006; will be a review aspect of every formal/informal test)

  • Requirement rot

    • Mitigation strategy: Update of EVLA requirements document to reflect current planning/time tables.

  • ASDM

    • Mitigation strategy: Multiple evaluations, early testing of needed modifications.

  • Algorithm development/data rates – SB talk

Overall risk mitigation

Overall Risk Mitigation

  • Backup plan: AIPS

  • Requires:

    • Continued support and targeted development

      • Support current VLA/VLBA users

      • Develop and deploy automated flagging algorithms to community for testing/review

  • Login