SOFIA Pipeline Update: Implementing Efficient Data Processing Methods

1. SOFIA Pipeline UpdateSSSC 18 February 2010 Erick Young Director SOFIA Science Mission Operations

2. The SOFIA Data Cycle Reduced Data Archive Archive & Reduce Data Query & Retrieve Data Raw Data Files & Data Manifests Science Data Products Investigator Science Instrument & Observatory Execute Observations FSI Astronomical Observation Requests (AORs) Develop Proposal Detailed Observing Plan Proposal Planning Database Observing Plans (ObsP)

3. Instrument Team Responsibilities • Provide algorithms for data reduction • Can be in “any” format • Teams have software they are using for their own data reduction to do their science, this can be provided “as is” to help build the official pipelines • SMO staff will translate what is provided into pipeline code to run in the DCS • Includes documentation on software and algorithms • Also includes sample test data • Work with SMO staff to validate implemented pipelines • Work with SMO staff to support guest investigators using instrument for observing • This will proceed along the currently established plans • The goal is for the instrument teams to not need to provide substantive additional support over what they are currently contracted to do

4. SMO Responsibilities • Responsible for final implementation of pipeline code • Coding • Integration into the DCS • Documentation • Testing/Validation • Maintenance • Responsible for configuration management of pipeline software • Responsible for ensuring suitable calibration data is taken for pipeline reduction • Operates pipelines and validates output science data • Interfaces with the science community

5. New vs Old SMO Work

6. Timeline • Goal is to have a fully functioning pipeline implemented for each instrument as they start their official commissioning flights after the 2011 down time • We plan to provide pipeline support for all mode outlined in the Science Vision • However, they may not all be available at the same time • Timing of commissioning will drive ramp-up of pipeline development effort and staff over the next two years • Start with re-integration of delivered FORCAST & FLITECAM pipelines • Integrate FIFI LS pipeline (assumes FIFI-LS will be completed) • Using “lessons learned”, expand work to implementation of expanded pipelines and develop pipelines for other instruments • For a given instrument we plan to start implementation ~2 years before commissioning flights are scheduled to begin

7. Current Status • A detailed implementation plan has been drafted, and initial resource estimates have been made. • We are currently working with NASA to adjust the plan in order to balance desirements vs. available resources. • SMO is working on a proposal to cover the SMOR-II additions • I would invite a subgroup of the SSSC to review our plans and to help set priorities regarding the elements of the pipeline effort.