HMI JSOC Science Data Processing
This presentation is the property of its rightful owner.
Sponsored Links
1 / 30

HMI JSOC Science Data Processing Art Amezcua PowerPoint PPT Presentation


  • 82 Views
  • Uploaded on
  • Presentation posted in: General

HMI JSOC Science Data Processing Art Amezcua. AGENDA JSOC-SDP Overview JSOC-SDP Status (H/W, S/W) Pipeline processing Database Level-0, Level-1 and higher levels JSOC-SDP Maintenance and CM Documentation Staffing Summary. JSOC Science Data Processing (SDP) Status.

Download Presentation

HMI JSOC Science Data Processing Art Amezcua

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Hmi jsoc science data processing art amezcua

HMI JSOC Science Data Processing Art Amezcua

AGENDA

  • JSOC-SDP Overview

  • JSOC-SDP Status (H/W, S/W)

    • Pipeline processing

    • Database

    • Level-0, Level-1 and higher levels

  • JSOC-SDP Maintenance and CM

  • Documentation

  • Staffing

  • Summary


Jsoc science data processing sdp status

JSOC Science Data Processing (SDP) Status

  • JSOC-SDP supports both HMI and AIA through Level-1 and HMI through Level-2 science data products

  • JSO-SDP Infrastructure is complete

    • JSOC-SDP Hardware is complete, upgrades in process

      • Database systems – Warm standby system online in September 2009

      • Web Server – Upgrade online in September 2009

    • JSOC-SDP Software

      • Data Record Management System and Storage Unit Management System (DRMS/SUMS) complete as of March 2009

    • JSOC-SDP Archive System is fully operational

  • Software Components needed to support commissioning

    • Level-0 Image processing for both AIA and HMI is ready and was used to support Observatory I&T

    • Level-0 HK, FDS and other metadata merge – complete as of May 2009

    • Level-1 (science observables) – will be completed during commissioning

      • HMI Doppler and LOS Magnetic – 95% complete

      • HMI Vector Field Observables – 90% complete

      • AIA Level-1.5 Images – 50% complete


Jsoc sdp status continued

JSOC-SDP Status (continued)

  • Software components needed to support science mission:

    • Production Pipeline Manager – In development, expected during commissioning

    • HMI Level-2 (Version 1 of science data products)

      • Local Helioseismology – Work in parallel on “rings”, “time-distance”, and “holography” proceeding with basic capability, expected to be ready during commissioning

      • Global Helioseismology – Ready for testing during commissioning

      • Magnetic Field standard products – Ready for testing during commissioning

      • Vector Field disambiguation – 80% complete with preliminary product ready by end of commissioning (requires real data to proceed)

    • Export and Catalog Browse Tools

      • Functional but needs work (http://jsoc.stanford.edu/ajax/lookdata.html)

      • Refinements will continue

    • All science products need flight data during commissioning to complete development

  • AIA Visualization Center (AVC) at Lockheed Martin

    • Higher-level AIA processing and science product generation

    • Heliophysics Event Knowledgebase (HEK)

  • Stanford Summary: On schedule for L – 4 and Phase E – 6 months


Hmi and aia jsoc overview

HMI and AIA JSOC Overview

GSFC

White Sands

housekeeping

LMSAL

MOC

DDS

House-

keeping

Database

HMI & AIA

Operations

Stanford

HMI JSOC Pipeline Processing System

Redundant Data Capture System

Quicklook

Viewing

JSOC-IOC

19-Day

Archive

Primary Archive

Local Archive

AIA Analysis

System

Catalog

JSOC-AVC

High-Level

Data Import

Offline

Archive

Data

Export

& Web

Service

World

Offsite

Archive

LMSAL

Science Team

Forecast Centers

EPO

Public

JSOC-SDP


Jsoc sdp data center

JSOC-SDP Data Center

  • Facility

    • Located in the climate-controlled basement of Physics and Astrophysics building at Stanford

    • Important components on UPS and building back-up power; databases auto-shutdown on power outage

    • Critical components monitored for failure (telephone, email, webpage notification of issues)

  • Components

    • 3 data-capture machines (plus one at MOC)

    • 1 data-processing cluster (512 CPU cores, 64 nodes, queuing system)

    • 1 file- and tape-server machine

    • 3 database machines

    • 2 gateway machines (to MOC, to SDP)

    • 1 web-server machine

    • 2 LMSAL real-time (housekeeping) machines

    • OC3 lines from DDS, ethernet (1 Gbps) connects all components, high-speed (20 Gbps) interconnect between data-processing cluster and file- and tape-server, 10 Gbps link between file- and tape-server and LMSAL.


Hmi jsoc science data processing art amezcua

16

16

16

4

4

4

4

2 Quad Core X86-64

Processors

8

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

8

8

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

2 Quad Core X86-64

Processors

10 TB

10 TB

10 TB

10 TB

13 TB

13 TB

13 TB

50 Slots

50 Slots

50 Slots

SPARC

SPARC

0.5 TB

0.5 TB

1G

Link

JSOC-SDP Major Components

DDS

GSFCAccess(FDS, L0HK)

DRMS & SUMS Database Hosts

Web Server

OutsideWorld

OC3 lines

HMIDB

HMIDB2

Web/Export

Data Capture System

ethernet (1 G)

Pipeline Cluster 512 cores

File/Tape Server

RT Mon

10G

Link

HMI

Spare

AIA

400 TB

Plus

150 TB py

2200-Slot

Tape Library

Plus

12 LTO-4

Drives

HMISDP-mon

Local Science

Workstations

Firewall

LMSAL

AIASDP-mon

10-G ethernet

20-G interconnect

MOC link for real-time housekeeping

X86-64 Processor Cores

Disk

LTO-4 Tape Library


Data capture

Data Capture

  • Telemetry files transmitted to data-capture (DCS) machines via two OC3 lines

    • One line for AIA (data from four cameras over two virtual channels)

    • One line for HMI (data from two cameras over two virtual channels)

  • Three sets of telemetry:

    • DCS machines archive telemetry files to tape, driven to LMSAL twice a week and stored in cabinet

    • Production processes on dedicated cluster node ingest raw telemetry from DCS disk into hmi.tlm and aia.tlm

    • Dedicated cluster node creates Level 0 data from telemetry and stores in DRMS/SUMS as hmi.lev0 and aia.lev0

  • DCS acks (once per day) DDS when offsite tape in place and verified and records in hmi.tlm created


Data capture details

S

S

S

S

C

C

C

C

I

I

I

I

S

S

S

S

C

C

C

C

O

O

O

O

Y

Y

Y

Y

S

S

S

S

T

T

T

T

E

E

E

E

M

M

M

M

S

S

S

S

Data Capture Details

HMI Monitoring

AIA

Monitoring

DDS

HMI

AIA

OC3

MOC link

Switches

Pipeline

Processing

S

I

S

Y

T

S

HMI DCS

Spare DCS

AIA DCS

Back End

ethernet

10gig private link

Spare DCS

Tape Archive

Disk array

13 TB (19 days)

Tape Robot

Operator hand net

UPS

UPS

UPS

Building Power w/ Generator Backup

LMSAL

MOC


Data capture status

Data Capture Status

  • Data capture machines

    • Online as of January 2008

    • Each is capable of caching 19 days telemetry

    • Tape drive is used to generate offsite copy stored at LMSAL

    • Pipeline system is used to generate tlm copy and Level-0 data series

    • All tapes are LTO-4 (800 GB)


Jsoc sdp pipeline hardware

JSOC-SDP Pipeline Hardware

  • Data-processing cluster, file- and tape-server machine, and T950 tape library [COMPLETE as of July 2008]


Data processing pipeline status

Data-Processing Pipeline Status

  • All machines fully tested for integrity with simulated data at realistic data rate and volumes

  • All machines/components under service warranties with vendors

  • Database machines have been online for four years (during DRMS development)

  • Data-processing cluster, file- and tape-server machine, T950 tape robot, and tape systems went online in July 2008

  • Upgrades (new machines onsite)

    • MOC web-access machine

    • solarport (gateway to SDP)

    • ftp server

    • web server

    • two database machines

    • in service in September 2009


Pipeline software drms sums

Pipeline Software – DRMS/SUMS

  • Data Series

    • Related images and metadata stored in “data series”

    • Rows are data records (e.g., one record per time step)

    • Columns are keywords, pointers to data files (in SUMS), pointers to other data series

  • Storage Unit Management System (SUMS)

    • Image files (e.g., FITS files) stored in SUMS

    • Uses PostgreSQL database

    • Sole client is DRMS

  • Data Record Management System (DRMS)

    • Data series minus image files

    • Implemented as C library which wraps PostgreSQL database

    • Has a FORTRAN interface

    • Scientist interact directly with DRMS

  • NetDRMS

    • Network of DRMS/SUMS sites that share DRMS/SUMS data

    • DRMS data shared via RemoteDRMS, which uses Slony-1 to make data logs that are ingested at remote site

    • Data files residing in SUMS shared via RemoteSUMS, which uses scp; integrates with VSO so that data are obtained from least-congested NetDRMS sites


Database configuration

Database Configuration


Level 0 processing

ingest_lev0

VC01

ingest_lev0

VC02

VC02*.tlm

VC02*.qac

VC05*.tlm

VC05*.qac

cl1n001 HMI

hmi.tlm

filename | SUDIR

SUMS

ingest_lev0

VC04

ingest_lev0

VC05

filename | SUDIR

*.tlm *.qac

NFS from dcs1

hmi.lev0

SUMS

fsn | lev0 keys | SUDIR

image.fitsimage_sm.fitsimage.png

fsn | lev0 keys | SUDIR

cl1n001 AIA

aia.tlm

filename | SUDIR

SUMS

filename | SUDIR

*.tlm*.qac

NFS from dcs0

VC01*.tlm

VC01*.qac

VC04*.tlm

VC04*.qac

aia.lev0

SUMS

fsn | lev0 keys | SUDIR

image.fitsimage_sm.fits image.png

fsn | lev0 keys | SUDIR

AIA ~24 images per VC01 or VC04 .tlm file.

HMI ~16 images per VC02 or VC05 .tlm file.

Level-0 Processing

  • Reconstructs images from tlm; no modification of CCD pixels [COMPLETE as of August 2008]


Level 1 processing

Level-1 Processing

fetch

level-0 keywords & segments

get readout mode corrections

read flatfield arguments

interpolate predicted orbit vectors

interpolate spacecraft pointing vectors

hmi.flatfield

fetch level-0 image

remove overscan rows & cols

correct for gain & offset

sdo.fds_orbit_vectors

sdo.lev0_asd_003

ID bad pixels

calculate image center

set quality

hmi.lev0

ancillary-data input

ancillary-data processing

image-data input

image-data processing

hmi.lev1

  • COMPLETE as of September 2009


Hmi observables level 1 products

HMI Observables – “Level-1” Products


Hmi higher level processing status

HMI Higher-Level Processing Status

  • Higher-Level Science Products

    • Internal Rotation Ω(r,Θ) [estimated complete as of August 2009]

    • Internal Sound Speed cs(r,Θ) [estimated complete as of August 2009]

    • Full-Disk Velocity v(r,Θ,Φ) [estimated complete as of December 2009]

    • Sound Speed cs(r,Θ,Φ) [estimated complete as of December 2009]

    • Carrington Synoptic Velocity Maps [estimated complete as of December 2009]

    • Carrington Synoptic Speed Maps [estimated complete as of December 2009]

    • High-Resolution Velocity Maps [estimated complete as of December 2009]

    • High-Resolution Speed Maps [estimated complete as of December 2009]

    • Deep Focus Maps [estimated complete as of July 2010]

    • Far-Side Activity Maps [estimated complete as of December 2009]

    • Line-of-Sight Magnetic Field Maps [COMPLETE as of July 2009]

    • Vector field inversion and direction disambiguation [estimated complete as of March 2010]

    • Vector Magnetic Field Maps [estimated complete as of April 2010]

    • Coronal Magnetic Field Extrapolations [COMPLETE as of July 2009]

    • Coronal and Solar Wind Models [estimated complete as of April 2010]

    • Brightness Images [estimated complete as of August 2009]


Data distribution and export

Data Distribution and Export

  • Scope

    • AIA – Level-0 and Level-1 data

    • HMI – Level-0 data through Level-2 data

  • Web Export

    • http://jsoc.stanford.edu/ajax/lookdata.html

    • Query for desired data, then download via web

    • Supports several data formats (internal files, FITS files, tar files, compressed files)

    • Provides support for special processing (such as extracting regions)

    • Other developers can expand on this export method by writing javascript that is allowed to access our web cgi programs

    • Functional now; enhancements estimated complete as of August 2009

  • NetDRMS

    • Network of DRMS sites

    • Can share DRMS data (not just data files) among sites using RemoteDRMS and RemoteSUMS

    • Scientists can request the same data from one of many sites

    • Functional now; enhancements estimated complete as of August 2009

  • Virtual Solar Observatory (VSO) Integration

    • Provides UI that allows uniform search of disparate types of data

    • Obtains metadata and data files from NetDRMS sites experiencing the least congestion

    • Estimated complete as of December 2009


Maintenance and expansion during mission

Maintenance and Expansion During Mission

  • Hardware

    • Each hardware component is covered under a vendor service plan; as plans expire, they are renewed

    • Planned phased replacement/upgrades throughout Phase-E

  • Software

    • Lead software developers are part of continuing team for Phase-E

  • Storage

    • File Server – 150 TB per year disk

    • Tape Library – filled tapes stored in Data Center, replaced with new tapes as needed; library expansion entails new 1300-slot cabinet when needed

  • Functionality

    • Anticipate continued development of science processing and distribution tools during Phase-E


Documentation

Documentation

  • Wiki - http://jsoc.stanford.edu/jsocwiki

  • Overview of JSOC Data Series, DRMS, and SUMS

  • JSOC Series Definition Files

  • DRMS Names

  • JSOC User’s Guide

  • JSOC Developer’s Guide

  • Doxygen - http://jsoc.stanford.edu/doxygen_html

    • Manual describing DRMS API functions and modules

    • Provides synopsis and describes input parameters, output, return values

    • To date, ~ 1/2 of functions/modules have documentation

  • Flow Diagrams

    • Tree diagrams illustrating connections between various programs, data series, data tables, etc.

    • Diagrammatic view of pipeline processes links to documentation

    • Note stages of development (A – E) and estimated completion date

  • Procedures documented

    • Database maintenance

    • DCS operations

    • Level-0 processing management

    • RemoteDRMS/SUMS installation and maintenance

  • Procedure documentation in progress

    • Calibration Processing (filters, flat fields, etc.)

    • Pipeline Dataflow Management System

    • Export management

    • Weekly data product report generation


  • Summary

    Summary

    • The JSOC-SDP can support:

      • Archive and distribution functions now

      • Analysis for instrument commissioning now

      • Initial science data processing by launch


    Backup slides

    Backup Slides


    Jsoc sdp stages of software development

    JSOC-SDP Stages of Software Development

    • Stage A – Code specification exists, but working code does not exist

    • Stage B – Prototype code exists, but not necessarily on HMI data and not necessarily in the correct language

    • Stage C – Working code exists but cannot run inside JSOC pipeline

    • Stage D – Working code capable of running in JSOC pipeline, but undergoing final testing and not released for general use

    • Stage E – Working code complete and integrated into JSOC pipeline

      Following dataflow charts show status as of FORR with estimated months to complete to stage E shown


    Jsoc sdp dataflow data capture

    JSOC-SDP Dataflow – Data Capture


    Jsoc sdp dataflow hmi level 0 and level 1

    JSOC-SDP Dataflow HMI Level-0 and Level-1


    Jsoc sdp dataflow hmi level 1 detail

    JSOC-SDP Dataflow HMI Level-1 – Detail


    Jsoc sdp dataflow hmi level 2 helioseismology

    JSOC-SDP Dataflow HMI Level-2 Helioseismology


    Jsoc sdp dataflow hmi level 2 los mag

    JSOC-SDP Dataflow HMI Level-2 – LOS Mag


    Jsoc sdp dataflow hmi level 2 vector mag

    JSOC-SDP Dataflow HMI Level-2 – Vector Mag


    Avc dataflow data distribution

    AVC Dataflow – Data Distribution

    Minutes

    Days

    Months


  • Login