Post processing overview
This presentation is the property of its rightful owner.
Sponsored Links
1 / 12

Post-processing Overview PowerPoint PPT Presentation


  • 123 Views
  • Uploaded on
  • Presentation posted in: General

Post-processing Overview. Bryan Butler (& Joe McMullin) (NRAO-AOC). Mission Primary purpose is to provide facilities for post-observational scientific reduction of ALMA/EVLA data Requirement documents: ALMA Offline Data Processing Requirements AIPS++ Audit

Download Presentation

Post-processing Overview

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Post processing overview

Post-processing Overview

Bryan Butler (& Joe McMullin)

(NRAO-AOC)


Science software group

Mission

Primary purpose is to provide facilities for post-observational scientific reduction of ALMA/EVLA data

Requirement documents:

ALMA Offline Data Processing Requirements

AIPS++ Audit

EVLA Data Post-Processing Software requirements

Uses the legacy AIPS++code base for development

AIPS used for validation, algorithm development (e.g., automated flagging)

Requirement Key Areas

General Requirements and Interaction

Interface

Data Handling

Calibration and Editing

Imaging

Data Analysis

Visualization

Special Features

Spreadsheet/tracking on the combined projects’ requirements (common to both projects and the deltas between projects)

Science Software Group


Organization

Management

IPT Organization

G. van Moorsel ((e)VLA/VLBA)

B. Glendenning (ALMA),

Michael Rupen (EVLA) & Debra Shepherd (ALMA) – Project Scientists for Computing

J. McMullin, K. Golap

(Project Management)

S. Myers (SSG Project Scientist)

Priorities (ALMA/EVLA)

Scientific Completeness/Correctness, Usability

Robustness, Performance

Organization

Resources

  • NRAO: 6.0 FTEs (including management; split 50/50 with ALMA)

  • ALMA: 5.5 FTEs (Application development, Data Model, Data Model Interfaces, Data Capture process)

  • NSF grant: 1.0 FTE (visualization; concludes 2007.8)


Resources details

Resources: Details


Ssg development

Status

Multi-year plan for development of all Priority 1 ALMA science software requirements (2007.5)

Project Office Page for development planning, accounting, transparency

http://projectoffice.aips2.nrao.edu

18 two-month development cycles completed

Established strong feedback loop with internal NRAO scientists

Established external scientist testing/verification

Infrastructure

RPM distribution

Framework (slide)

Milestones

Yearly NRAO VC and UC meetings

ASAC, EVLA ACM meetings

ALMA CDR1 (Jun 03) - Passed

ALMA TST1 (Jan 04) – Passed

Single Field Interferometry

ALMA CDR2 (Jul 04) - Passed

ALMA R2 (Oct 04) - Complete

ALMA TST2 (Nov 04) – Passed

+Mosaics

ALMA TST3 (Apr 05) – Passed

+synthesis/single dish combo

ALMA R2.1 (Apr 05) - Complete

EVLA TST1 (Jun 05) – Passed

+Wide Field Imaging

ALMA CDR3 (Jul 05) - Passed

ALMA R3 (Oct 05) - Complete

+ATF simulation (1st CASA test)

ALMA R3.1 (Mar 06) - Complete

+User Interface Charette

Provided testing/detailed requirements

SSG Development


Long term schedule presented at cdr2 in july 2004

2003.5

Long-term plan

2004

Science: Imaging performance enhancements

Science: Flexible continuum subtraction

Framework: CVS code management

Data Interfaces: Draft AEDF document

2004.5

Science: SF reduction (TST1)

Science: Calibrater performance enhancements

Science: Calibrater flexibility (interpolations, transfer, etc).

Framework: Framework requirements, plan

Data Interfaces: AEDF submitted for review, data capture design

2005

Science: Mosaic reduction (TST2)

Science: Automated, interactive data editing

Framework: Library re-organization

Data Interfaces: Data Capture process

2005.5

Science: Single dish/interferometric (TST3)

Science: Simulator (telescope modeler, sampler, visibility simulator)

Framework: Beta, subset, distribution with ACS

Data Interfaces: DRP filler

2006

Science: Complex mosaics (TST4) – changed to ATF (single-baseline) test

Science: Full polarized primary beam imaging

Framework: Initial CLI, Python interface – added a working group evaluation of the interface

2006.5

Science: Single Dish (TST5)

Science: Flexible line fitting

Framework: Testing with new framework – 1st done in first half 2006

2007

Science: Simulations I (TST6)

Science: Multi-frequency synthesis

Framework: GUI development

Long-term Schedule (Presented at CDR2 in July 2004)


Coarse timeline staging of community usage user support of casa

2006

NRAO: NAUG testing (AIPS++)

Community: Project tests

ALMA: external science testing: +single-baseline commissioning test) (CASA)

ALMA: external science testing: +single dish reduction (CASA)

ALMA: commissioning support (CASA)

ALMA/EVLA: user interface review (CASA)

EVLA: external science testing: +full polarization imaging; antenna pointing calibration

2006.5 AIPS++ frozen

ALMA: Pipeline Heuristics Use of CASA

2007

NRAO: User support (CASA)

ALMA: commissiong support (CASA)

EVLA: external science testing: + RFI/automated flagging (CASA)

ALMA: P1 SSRs complete

2008

EVLA: external science testing: + wide band calibration imaging (CASA)

NRAO: user support (CASA)

ALMA: commissioning support

EVLA: P1 SSRs complete

2009-2010

ALMA/EVLA: commissioning support

Community: CASA released/distributed for early ALMA/EVLA science

2011

Community: CASA released/distributed; full user support (ALMA/EVLA: P1/P2s)

Coarse Timeline & Staging of Community Usage/User Support of CASA


Framework migration

AIPS++

Glish interface; unknown, unsupported outside of NRAO

Tasking system based on Glish

GUI system based on Glish/Tk; limited widgets, not robust!

Difficult for external developers to contribute

Multi-CD binary distribution

Large monolithic libraries with cross dependencies

No namespace

Freeze 2nd half of 2006

CASA

Python interface (community standard);IPython

Binding to Python, ACS; other frameworks readily possible

Hierarchical set of small libraries with clearly defined dependencies

Namespace protection for integration with other code

RPM distribution mechanism; auto-updates possible

Robust

Inherits all application code improvements in robustness and performance.

Smaller memory footprint/startup time

Framework Migration


Casa usage status

CASA usage status:

  • Has been tested internally by NRAO scientists in preparation for the 1st ALMA test.

    • Early demos provided bi-monthly to the NAUG at:

      • http://casa.nrao.edu/gettingstarted.shtml

    • 6 scientists used and reported on it – enabled deployment for the ALMA test.

  • Was deployed and reviewed by four ALMA testers

    • http://projectoffice.aips2.nrao.edu/ALMA2006.01/ALMA2006.01.html

  • The user interface was reviewed and commented on by 8 NRAO scientists

    • User Interface is being refined/further developed based on the user interface report:

      • http://projectoffice.aips2.nrao.edu/uiwg_report.pdf

  • Will be used exclusively by the ALMA Pipeline Heuristics Team (second half of 2006)

  • CASA will have replaced AIPS++ within NRAO (developers and NAUG testers) this year.


User interface

User Interface

  • User Interface review

    • IPython interface to functionality, prototype parameter setting interface and in-line help are in the right direction but:

    • Full needs are documented at:

      • http://projectoffice.aips2.nrao.edu/uiwg_report.pdf

  • Revised interface prototype (similar to IRAF epar environment); high priority

  • Much work needed to provide astronomer-level documentation (collaboration with NAUG to develop this); high priority

  • Development progress will be reviewed with the Fall EVLA test.


Risks

Risks

  • Staffing

    • Mitigation strategy: efficient use of existing resources; leverage off of ALMA development as much as sensible

  • User Support

    • Mitigation strategy: Work with group of NAUG scientists such that they can provide the front-line help.

  • User Interface

    • Mitigation strategy: Cyclic feedback on progress and prototypes (revisited in June-July 2006; will be a review aspect of every formal/informal test)

  • Requirement rot

    • Mitigation strategy: Update of EVLA requirements document to reflect current planning/time tables.

  • ASDM

    • Mitigation strategy: Multiple evaluations, early testing of needed modifications.

  • Algorithm development/data rates – SB talk


Overall risk mitigation

Overall Risk Mitigation

  • Backup plan: AIPS

  • Requires:

    • Continued support and targeted development

      • Support current VLA/VLBA users

      • Develop and deploy automated flagging algorithms to community for testing/review


  • Login