post processing overview n.
Skip this Video
Loading SlideShow in 5 Seconds..
Post-processing Overview PowerPoint Presentation
Download Presentation
Post-processing Overview

Loading in 2 Seconds...

play fullscreen
1 / 12

Post-processing Overview - PowerPoint PPT Presentation

  • Uploaded on

Post-processing Overview. Bryan Butler (& Joe McMullin) (NRAO-AOC). Mission Primary purpose is to provide facilities for post-observational scientific reduction of ALMA/EVLA data Requirement documents: ALMA Offline Data Processing Requirements AIPS++ Audit

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Post-processing Overview' - deanne

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
post processing overview

Post-processing Overview

Bryan Butler (& Joe McMullin)


science software group

Primary purpose is to provide facilities for post-observational scientific reduction of ALMA/EVLA data

Requirement documents:

ALMA Offline Data Processing Requirements

AIPS++ Audit

EVLA Data Post-Processing Software requirements

Uses the legacy AIPS++code base for development

AIPS used for validation, algorithm development (e.g., automated flagging)

Requirement Key Areas

General Requirements and Interaction


Data Handling

Calibration and Editing


Data Analysis


Special Features

Spreadsheet/tracking on the combined projects’ requirements (common to both projects and the deltas between projects)

Science Software Group

IPT Organization

G. van Moorsel ((e)VLA/VLBA)

B. Glendenning (ALMA),

Michael Rupen (EVLA) & Debra Shepherd (ALMA) – Project Scientists for Computing

J. McMullin, K. Golap

(Project Management)

S. Myers (SSG Project Scientist)

Priorities (ALMA/EVLA)

Scientific Completeness/Correctness, Usability

Robustness, Performance



  • NRAO: 6.0 FTEs (including management; split 50/50 with ALMA)
  • ALMA: 5.5 FTEs (Application development, Data Model, Data Model Interfaces, Data Capture process)
  • NSF grant: 1.0 FTE (visualization; concludes 2007.8)
ssg development

Multi-year plan for development of all Priority 1 ALMA science software requirements (2007.5)

Project Office Page for development planning, accounting, transparency

18 two-month development cycles completed

Established strong feedback loop with internal NRAO scientists

Established external scientist testing/verification


RPM distribution

Framework (slide)


Yearly NRAO VC and UC meetings

ASAC, EVLA ACM meetings

ALMA CDR1 (Jun 03) - Passed

ALMA TST1 (Jan 04) – Passed

Single Field Interferometry

ALMA CDR2 (Jul 04) - Passed

ALMA R2 (Oct 04) - Complete

ALMA TST2 (Nov 04) – Passed


ALMA TST3 (Apr 05) – Passed

+synthesis/single dish combo

ALMA R2.1 (Apr 05) - Complete

EVLA TST1 (Jun 05) – Passed

+Wide Field Imaging

ALMA CDR3 (Jul 05) - Passed

ALMA R3 (Oct 05) - Complete

+ATF simulation (1st CASA test)

ALMA R3.1 (Mar 06) - Complete

+User Interface Charette

Provided testing/detailed requirements

SSG Development
long term schedule presented at cdr2 in july 2004

Long-term plan


Science: Imaging performance enhancements

Science: Flexible continuum subtraction

Framework: CVS code management

Data Interfaces: Draft AEDF document


Science: SF reduction (TST1)

Science: Calibrater performance enhancements

Science: Calibrater flexibility (interpolations, transfer, etc).

Framework: Framework requirements, plan

Data Interfaces: AEDF submitted for review, data capture design


Science: Mosaic reduction (TST2)

Science: Automated, interactive data editing

Framework: Library re-organization

Data Interfaces: Data Capture process


Science: Single dish/interferometric (TST3)

Science: Simulator (telescope modeler, sampler, visibility simulator)

Framework: Beta, subset, distribution with ACS

Data Interfaces: DRP filler


Science: Complex mosaics (TST4) – changed to ATF (single-baseline) test

Science: Full polarized primary beam imaging

Framework: Initial CLI, Python interface – added a working group evaluation of the interface


Science: Single Dish (TST5)

Science: Flexible line fitting

Framework: Testing with new framework – 1st done in first half 2006


Science: Simulations I (TST6)

Science: Multi-frequency synthesis

Framework: GUI development

Long-term Schedule (Presented at CDR2 in July 2004)
coarse timeline staging of community usage user support of casa

NRAO: NAUG testing (AIPS++)

Community: Project tests

ALMA: external science testing: +single-baseline commissioning test) (CASA)

ALMA: external science testing: +single dish reduction (CASA)

ALMA: commissioning support (CASA)

ALMA/EVLA: user interface review (CASA)

EVLA: external science testing: +full polarization imaging; antenna pointing calibration

2006.5 AIPS++ frozen

ALMA: Pipeline Heuristics Use of CASA


NRAO: User support (CASA)

ALMA: commissiong support (CASA)

EVLA: external science testing: + RFI/automated flagging (CASA)

ALMA: P1 SSRs complete


EVLA: external science testing: + wide band calibration imaging (CASA)

NRAO: user support (CASA)

ALMA: commissioning support

EVLA: P1 SSRs complete


ALMA/EVLA: commissioning support

Community: CASA released/distributed for early ALMA/EVLA science


Community: CASA released/distributed; full user support (ALMA/EVLA: P1/P2s)

Coarse Timeline & Staging of Community Usage/User Support of CASA
framework migration

Glish interface; unknown, unsupported outside of NRAO

Tasking system based on Glish

GUI system based on Glish/Tk; limited widgets, not robust!

Difficult for external developers to contribute

Multi-CD binary distribution

Large monolithic libraries with cross dependencies

No namespace

Freeze 2nd half of 2006


Python interface (community standard);IPython

Binding to Python, ACS; other frameworks readily possible

Hierarchical set of small libraries with clearly defined dependencies

Namespace protection for integration with other code

RPM distribution mechanism; auto-updates possible


Inherits all application code improvements in robustness and performance.

Smaller memory footprint/startup time

Framework Migration
casa usage status
CASA usage status:
  • Has been tested internally by NRAO scientists in preparation for the 1st ALMA test.
    • Early demos provided bi-monthly to the NAUG at:
    • 6 scientists used and reported on it – enabled deployment for the ALMA test.
  • Was deployed and reviewed by four ALMA testers
  • The user interface was reviewed and commented on by 8 NRAO scientists
    • User Interface is being refined/further developed based on the user interface report:
  • Will be used exclusively by the ALMA Pipeline Heuristics Team (second half of 2006)
  • CASA will have replaced AIPS++ within NRAO (developers and NAUG testers) this year.
user interface
User Interface
  • User Interface review
    • IPython interface to functionality, prototype parameter setting interface and in-line help are in the right direction but:
    • Full needs are documented at:
  • Revised interface prototype (similar to IRAF epar environment); high priority
  • Much work needed to provide astronomer-level documentation (collaboration with NAUG to develop this); high priority
  • Development progress will be reviewed with the Fall EVLA test.
  • Staffing
    • Mitigation strategy: efficient use of existing resources; leverage off of ALMA development as much as sensible
  • User Support
    • Mitigation strategy: Work with group of NAUG scientists such that they can provide the front-line help.
  • User Interface
    • Mitigation strategy: Cyclic feedback on progress and prototypes (revisited in June-July 2006; will be a review aspect of every formal/informal test)
  • Requirement rot
    • Mitigation strategy: Update of EVLA requirements document to reflect current planning/time tables.
  • ASDM
    • Mitigation strategy: Multiple evaluations, early testing of needed modifications.
  • Algorithm development/data rates – SB talk
overall risk mitigation
Overall Risk Mitigation
  • Backup plan: AIPS
  • Requires:
    • Continued support and targeted development
      • Support current VLA/VLBA users
      • Develop and deploy automated flagging algorithms to community for testing/review