Mcaf metadata capture and formatting
Download
1 / 15

MCAF (Metadata Capture and Formatting) - PowerPoint PPT Presentation


  • 103 Views
  • Uploaded on

MCAF (Metadata Capture and Formatting). Rich Moeser. Outline. MCAF overview System diagram Timeline Status Design Possible Reuse (from IDCAF or ALMA) SDM Deployment Summary/Questions. MCAF. MCAF was originally named DCAF

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' MCAF (Metadata Capture and Formatting)' - katima


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Mcaf metadata capture and formatting

MCAF(Metadata Capture and Formatting)

Rich Moeser

Correlator f2f Meeting


Outline
Outline

  • MCAF overview

  • System diagram

  • Timeline

  • Status

  • Design

  • Possible Reuse (from IDCAF or ALMA)

  • SDM

  • Deployment

  • Summary/Questions

Correlator f2f Meeting


MCAF

  • MCAF was originally named DCAF

    • The name was changed to emphasize that it’s “meta” data that’s being captured (as opposed to visibility or monitor data).

  • MCAF’s primary responsibilities are:

    • To collect science metadata from the EVLA system and the correlator.

    • Combine and reorganize the data

    • And write the data in ESDM (EVLA Science Data Model) format

  • It is the successor of IDCAF (Interim Data Capture and Formatting)

    • This is the data capture process currently used by the EVLA system.

    • Required for the retirement of the Modcomp computers.

    • Writes the data in VLA export format to the VLA archive.

  • Differences between IDCAF and MCAF

    • IDCAF = EMCS + VLA Correlator + VLA Export Format

    • MCAF = EMCS + WIDAR PTC + ESDM Format

Correlator f2f Meeting


System diagram
System Diagram

MCAF

EVLA M&C

Correlator f2f Meeting


Phases timeline
Phases/Timeline

  • ALMA deliverables: Q1 2008

  • Typed XML schema

  • Modifications to code generator to eliminate ACS/CORBA dependencies

  • Feb 2008

    • Detailed design complete

  • Mar 2008

    • Schema definitions for all data going into MCAF

  • May (early) 2008

    • Writing the minimal SDM (= mandatory tables)

  • July (mid) 2008

    • Ready to support PTC tests

    • Writing minimal SDM + other required tables (if any)

  • Q1 2009

    • Support for simple observing with WIDAR

    • “commissioning basic observing modes”

  • Q4 2009

    • Writing full-blown SDM, supporting all tables for EVLA/WIDAR

    • “commissioning advanced observing modes”

Correlator f2f Meeting


Status

Currently in the early stages of design and prototype.

Focus has been mostly on infrastructure

Data collection

SDM handling (SDM to Java binding)

Communications

Status

Correlator f2f Meeting


Mcaf design

?/XML

UDP/XML

HTTP/XML

MCAF design

<*/>

UdpMessageHandler

MCAF

Core

<*/>

(Data Aggregation)

HttpMessageHandler

<receipt/>

<alerts/>

OtherMessageHandler

Alert Server

External software process

(HTTP GET)

HttpRequestHandler

notification new data available

<mcaf/>

SDM Archive Interface

SDM

Archive

ui.jsp

(SDM out)

<html/>

Status and Control GUI

HTTP

(data type in)

sdm.xml files (igloo)

TelCal

UDP

(data type out)

Unknown

Correlator f2f Meeting


Mcaf input

This includes all data required to build the ESDM

Data that is either sent to MCAF or data that it retrieves on its own.

MCAFInput

  • EVLA Data Providers

    • Executor

    • CMP

    • MIBs

    • Alert Server or Flagger

      • (depends on internal or external flagging)

    • Dynamic Scheduler (?)

    • TelCal(?)

    • Others?

  • Correlator Data Providers

    • CBE/FF (?)

    • Station Boards (?)

    • Baseline Boards (?)

    • MCCC (VCI) (?)

    • Others?

Correlator f2f Meeting


Mcaf output

Writes SDM output files (to a staging disk)

Sends alerts to the Alert Server

Data to TelCal (the SDM files)

Archive notification (?)

MCAFOutput

Correlator f2f Meeting


Communications

Sending data to MCAF

UDP Multicasts containing XML documents

HTTP POSTs containing XML documents

Unknown data types received by MCAF will be logged and dropped.

MCAF fetching data

What data will MCAF need to collect on its own?

If MCAF needs to get data from the SBs will it need to open a connection to each one?

What is the communications protocol?

Access to MCAF information from clients (Java, python, …)

HTTP interface to retrieve basic status information, e.g. health, number of packets received and processed, errors, start date and time, etc

Data will be an XML <mcaf> document back to the client.

Browser access…

A simple JSP (Java Server Pages) will display MCAF status and possibly present control options (shutdown, restart, etc).

Communications

Correlator f2f Meeting


Possible reuse from idcaf or alma
Possible Reuse?(from IDCAF or ALMA)

  • IDCAF is written in C, MCAF will be written in Java.

    • No chance of reuse there

    • Possibility for reusing parts of the design

  • The infrastructure for sending data to IDCAF exists, e.g. in the Executor, so there’s a good chance the existing XML schema can be used and simply extended.

  • There’s a very good chance of being able to use ALMA’s code for reading and writing SDM.

Correlator f2f Meeting


SDM

  • The SDM is a collection of ~40 “tables” containing “rows” of data of a known type.

  • The “minimal” SDM will create and write the “mandatory” tables of the SDM.

  • The “mandatory” tables:

    • Main, AlmaCorrelatorMode, Antenna, ConfigDescription, DataDescription, Feed, Field, Polarization, Processor, Scan, Source, SpectralWindow, State, SubScan

  • The remainder of the table are considered “optional” tables:

    • Beam, CalDevice, CalAtmosphere, Doppler, Ephemeris, ExecBlock, Focus, FocusModel, GainTracking, History, Observation, Pointing, PointingModel, Receiver, SBSummary, Seeing, SourceParameter, SquareLawDetector, Station, SwitchCycle, TotalPowerData, WVMcal, Weather.

  • Are the mandatory tables sufficient for the PTC tests? If not, what else is needed?

Correlator f2f Meeting


Sdm java binding options

ALMA

CASA

EVLA

Archive

MCAF

UML

Authoring Tool

(MagicDraw)

HLA

Code Generation

Framework

sdm.xmi

ACS/CORBA

dependencies

sdm.xsd

(typed)

ALMA

Generated Source

ALMA jar file

JAXB

(compile and package)

JAXB

(Java API for

XML Binding)

JAXB

Generated Source

JAXB jar file

SDM Java Binding Options

Master SDM

Document

Correlator f2f Meeting


Mcaf deployment for the ptc

MCAF will run on the machine igloo (or its replacement) out at the site.

It will be packaged as a “.war” file and deployed to a running instance of Apache Tomcat.

If the system goes down for whatever reason Tomcat will automatically start and launch all of the applications in its container, including MCAF.

MCAF Deploymentfor the PTC

igloo

Apache Tomcat

Input Data

MCAF

Client Requests

igloo’s

staging disk

Correlator f2f Meeting


Summary questions
Summary/Questions

Summary

  • Still in the early stages of MCAF

  • A detailed examination is needed to find out where all of the data required by the SDM will originate

  • A first cut that writes minimal SDM will be ready by May

  • There’s a pretty good chance we’ll be able to use some of ALMA’s software.

  • Determine whether or not flagging should be built into MCAF or if it should be a standalone external process.

    Questions

  • Does MCAF gather anything directly from the CBE/FF/SBs/BBs/MCCC?

    • What is the communications protocol?

  • Is the “minimal” SDM sufficient for PTC testing?

  • The plan is to have MCAF writing SDM in May. Is that acceptable?

Correlator f2f Meeting


ad