Data processing
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

Data Processing PowerPoint PPT Presentation


  • 79 Views
  • Uploaded on
  • Presentation posted in: General

Data Processing. Progress and Planning June 2014. Jeff Kern. Data Processing Group Lead. Summary Status. CASA-4.2.2 brings capabilities desired for cycle 2 data reduction Currently under test issues with plotMS , otherwise looking good Key deliverables: Weight handling (per SPW)

Download Presentation

Data Processing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Data processing

Data Processing

Progress and Planning

June 2014

  • Jeff Kern

  • Data Processing Group Lead


Summary status

Summary Status

  • CASA-4.2.2 brings capabilities desired for cycle 2 data reduction

    • Currently under test issues with plotMS, otherwise looking good

    • Key deliverables:

      • Weight handling (per SPW)

      • Flag Data Improvement (Tbuff parameter to avoid quacking)

      • Strict Flagging in Applycal

      • Single Dish Interface Update

  • Pipeline

    • Provisional Acceptance in March

    • Long punch list, mostly cosmetic. Largely addressed at this point

    • Doing integration testing against CASA-4.2.2


Planning near future

Planning: Near Future

  • June 27: Prerelease of CASA-4.2.2 and Pipeline to PLWG for testing

  • July 1: Release of CASA-4.2.2 (Public)

    • Both OSX and Linux

  • ~July 1: Release of ALMA Pipeline to project

    • Based on testing from PLWG

  • ~July 15: CASA-4.2.2-Pipeline release available to public

    These dates are all very tight, and are based on not finding issues in the testing. Small slips can have large impact as we are in vacation season


Casa planning

CASA Planning

  • September 2014 CASA-4.3

  • Weight Spectrum

  • Imaging Refactor V1

  • Parallelization (Calibration)

  • SetJy Improvements

  • VO Viewer

  • March 2015 CASA-4.4

  • Imaging Refactor V2

    • Support for Pipeline

  • OTF Cal Application

  • Pipeline Parallelization

  • Enhanced Phase Connection


Pipeline planning

Pipeline Planning

  • From ObsMode Meeting

    • Support for Cycle 2 Operations

      • This could stop all other progress!

    • Data Flagging Heuristics

    • Bandwidth Switching Heuristics

      • First of low SNR Heuristics

    • Science Target Imaging

      • 1 Cube per Science SPW per Science Target

  • In parallel:

    • Automated trigger and cluster exectuion

    • This is NOT a magic bullet.


Weblog example

Weblog Example

Pipeline Regression Server


Plwg user test

PLWG: User Test

  • Pipeline Working Group will perform a user test to receive feedback on Weblog

    • Format TBD either special run and request comments or just survey of Cycle 2 Pis

  • Will need to prioritize feedback from this vs. functional testing


Pipeline single dish

Pipeline Single Dish

  • Well developed and tested

    • Not much knowledge outside of NAOJ so far

  • Hoped to be doing parallel testing at this point

    • Need lots of data (Project wide issue)

  • Concerned about connection between manual reduction, script generator, and pipeline

ICT CPM-3 June 17-19 Socorro New Mexico


Pipeline releases

Pipeline Releases

  • November Update

    • Bandwidth Switching

    • Data Flagging

    • Other Cycle 2 Experience (TBD)

  • March 2015

    • Imaging capabilities V1

    • Long Baseline capabilities

    • Low SNR (?)

  • Q3 2015

    • Continuum Subtraction

    • Self Cal

    • Other Calibration

Infrastructure

&

Automation

Preliminary


Steps to pipeline use in operations

Steps to Pipeline Use in Operations

Cycle 2

Phase C testing

Test commissioned observing modes for Cycle 2 data

PWG full acceptance testing

Test full acceptance criteria

Pipeline documentation review

Send final drafts broad review within Observatory

PWG full acceptance recommendation

Submit report to SciOpsIPT

June/July 2014

Test full acceptance criteria implementation

Test commissioned observing modes for Cycle 2 data

Submit final drafts to SciOpsIPT and DRM WG for review

Submit full acceptance report to Observatory

First Pipeline-assisted deliveries

DRM WG end-to-end testing

Test e-2-e QA2 process, packaging & documentation

Pipeline-assisted QA2 in operations

Pipeline released to users

Test e-2-e QA2 process, packaging & QA2 documentation

For commissioned observing modes, data calibrated by Pipeline and imaged manually

CASA 4.2.2 tarball including Pipeline


Pipeline issues

Pipeline Issues

  • Staffing:

    • Pipeline has exactly 1 full time developer

    • Currently stepdown another 0.5 FTE at end of year

      • Formally 3.0 -> 2.5 but realistically 1.8 -> 1.3 on interferometric

    • “Borrowing” expertise from CASA but not the same as dedicated staff

  • Maintenance / Support

    • Tracking down errors for failed SBs could halt progress on new targets

    • Need robust QA0 process at observatory

  • Balance between Cosmetic and Functional requirements

    • Weblog has grown very large, suspect it may be overspecified at this point


Data processing

CASA and CASA Based Pipelines

Pipeline Weblog: Home Page


Data processing

CASA and CASA Based Pipelines

Pipeline Weblog: Observation Summary


Data processing

CASA and CASA Based Pipelines

Pipeline Weblog: Topics Page


Data processing

CASA and CASA Based Pipelines

Pipeline Weblog: Task Summary


Data processing

CASA and CASA Based Pipelines

Pipeline Weblog: Task Detail


Data processing

CASA and CASA Based Pipelines

Pipeline Weblog: Plot Details


Data processing

CASA and CASA Based Pipelines

Pipeline Weblog: Detail Zoom


Data processing

CASA and CASA Based Pipelines

Pipeline Weblog: Imaging Summary


Data processing

CASA and CASA Based Pipelines

Pipeline Weblog: Imaging Details


  • Login