evla software overall architecture science online domains n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
EVLA Software - Overall Architecture, Science & Online Domains PowerPoint Presentation
Download Presentation
EVLA Software - Overall Architecture, Science & Online Domains

Loading in 2 Seconds...

play fullscreen
1 / 36

EVLA Software - Overall Architecture, Science & Online Domains - PowerPoint PPT Presentation


  • 116 Views
  • Uploaded on

EVLA Software - Overall Architecture, Science & Online Domains. Bryan Butler NRAO. Observation Preparation. EVL A. VLBA. ALMA. GBT. VLBA Sched. GBT Sched. ALMA Sched. GBT Control. VLBA Control. ALMA Control. NRAO-wide Design. Mostly Telescope-Independent Common Software.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'EVLA Software - Overall Architecture, Science & Online Domains' - cindy


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
nrao wide design

Observation Preparation

EVLA

VLBA

ALMA

GBT

VLBA

Sched

GBT

Sched

ALMA

Sched

GBT

Control

VLBA

Control

ALMA

Control

NRAO-wide Design

Mostly Telescope-Independent

Common Software

Proposal Submission

And Handling

Observer

Domain

Mostly Telescope-Specific

Project Software

EVLA

Sched

Telescope

Domain

EVLA

Control

Telescope Data Model

GBT Postproc

Feedback to telescope

Data

Capture

Science Data Model

Telcal

Quick Look

Mostly Telescope-Independent

Common Software

Science

Domain

Archive

Pipeline

Export Data Format

Offline

VO

EVLA Advisory Committee Meeting

nrao wide design1
NRAO-wide Design

The EVLA software system needs to conform to the NRAO-wide design to the extent that it has been developed. Unfortunately, the NRAO-wide effort has languished for the past two years (this is changing - a new “e2e Project Manager” [Nicole Radziwill] and “e2e Project Scientist” [Ed Fomalont] have been hired in CV).

EVLA Advisory Committee Meeting

domains models
Domains & Models

The NRAO-wide design introduced the concept of “domains” - different (large) pieces of the system that can be grouped sensibly. It presented three such domains:

  • Observer
  • Telescope
  • Science

It also presented a number of “models” - descriptions of different (smaller) pieces of the system:

  • Observatory
  • Project
  • Observing
  • Archive
  • Telescope Data
  • Science Data

EVLA Advisory Committee Meeting

evla overall design
EVLA Overall Design

In the spring of 2004, the EVLA software group undertook an effort to develop and document a high level design (led by Morgan, Ryan, Sowinski, and Waters). This culminated in a completed design, which was reviewed by the NRAO eOC in June 2004. The design presented in the next few slides is based mostly on that effort, with extension and modification in the past two years.

EVLA Advisory Committee Meeting

evla software requirements
EVLA Software Requirements

The software design and implementation is driven by a number of requirements documents:

  • e2e Science Software Requirements
  • Engineering Software Requirements
  • Real-time System Software Requirements
  • Operations Software Requirements

These do not have everything in them (for instance Proposal Handling and User Database, which are covered in separate [less “requirementy”] documents), but are fairly complete, and notably, the e2e requirements document has extensive requirements for PST and OPT.

EVLA Advisory Committee Meeting

major elements models
Major Elements (“Models”)

The main flow of information (and processes; the “workflow” or “dataflow”) is:

A Scheduling Block is an atomic unit of observing. It is made up of a sequence ofscans; a scan is made up of source(s), resource(s) (hardware definition - both FE and BE), timinginformation, and a “mode”. The mode defines the subscan(s), which are comprised of a single source, resource, and timing information.

Proposal

Data

Project(s)

Commands

Program(s)

Schedule(s)

EVLA Advisory Committee Meeting

evla high level architecture
EVLA High Level Architecture

DATAFLOW

EVLA Advisory Committee Meeting

evla high level architecture1
EVLA High Level Architecture

Note that most of the major subsystems have a direct counterpart in current VLA software - we have a significant amount of experience in what is needed there. What has been lacking is the electronic storage and passage of information between subsystems, and therefore the ability to do much of this automatically.

EVLA Advisory Committee Meeting

slide10

EVLA Design Detail, Pre-Observing Science Domain

Proposal Submission

Tool (PST)

Proposal

Portal

Proposal Handling

Tool (PHT)

Astronomer or Staff

Authenticated

Astronomer or Staff

EVLA

Observing

Heuristics

Project

Observation

Preparation

Tool (OPT)

Program Block

(Set of Scheduling Blocks for one Program)

To Observation Scheduling Tool

EVLA Advisory Committee Meeting

slide11

EVLA Design Detail, Pre-Observing Online Domain

From OPT

Archive

Observation

Scheduling

Tool (OST)

Operator

Environment

Heuristics

Metadata to DCAF

Execution

State

Next SB

Archive

Executor

Operator

Metadata to DCAF

Results from TelCal

Equipment State

Sequence of Configurations

Antenna Delays

From AMCS

& CMCS

To AMCS

& CMCS

EVLA Advisory Committee Meeting

slide12

EVLA Design Detail, Real-Time Domain

From Executor

State Counts

FOTS Receiver

Station, Baseline Boards

EVLA Antennas

RF

CBE

Lag Frames

AMCS

FF

Raw Vis

CMCS

Hardware M&C

Equipment State, Data Addressing Info, Messages, Alerts, etc.

To Archive & TelCal

To DCAF

To DCAF

EVLA Advisory Committee Meeting

slide13

EVLA Design Detail, Post-Observing Online Domain

Astronomer or

Operator

To Executor

And Archive

From AMCS

& CMCS

From CMCS

Portal

TelCal

Results

Data Capture And

Format (DCAF)

TelCal

Authenticated

Astronomer or

Operator

SDM

SDM

M&C

Archive

Observation

Status Monitor

Tool (OSMT)

Quick Look

Pipeline

(QLP)

M&C

Archive

To Archive (?)

To Archive

EVLA Advisory Committee Meeting

slide14

Archive

EVLA Design Detail, Post-Observing Science Domain

Astronomer

From DCAF

From CMCS

Open

Products

Archive Search and

Retrieval Tool

(ASRT)

Portal

Cubes (?)

Existing Proprietary

Products

Image

Cubes

Open

Products

Post-Processing

Default Image

Pipeline (DIP)

Trigger

Authenticated

Astronomer

VO Astronomer

Reprocessed

Proprietary

Products

EVLA Advisory Committee Meeting

detailed subsystem designs prototypes
Detailed Subsystem Designs & Prototypes

For most of the major subsystems, we either have a very advanced prototype (Portal, PST, Executor, OSMT), an early prototype (PHT, OPT, OST, ASRT, TelCal), or at least a much more detailed design (DCAF). The areas where we have done little work and in fact have only preliminary (high level) designs are for the pipelines (QLP, DIP).

EVLA Advisory Committee Meeting

portal user database authentication
Portal, User Database, Authentication

We know that we need some way to restrict access to the various tools, and the information available within them. We do this with a “portal”, through which users access the various tools. It authenticates users, and generates a unique token which is then used to verify a user’s login status. We have our own implementation of this, but recognize that we may need to migrate to the VO method (or have a translation layer).

EVLA Advisory Committee Meeting

portal user database authentication1
Portal, User Database, Authentication

EVLA Advisory Committee Meeting

portal user database authentication2
Portal, User Database, Authentication

Users enter their own User Database information, except:

  • Institution information - they can only select an institution from a pre-set list (we want to use this to automatically generate reports to the NSF, which require precise information on institutions) - if the institute is not there, they indicate so, and we (well, I) add it.
  • We allow so-called “3rd party” user registrations during proposal preparation and submission, adding significant complexity (a sore spot with us, but demanded by operations).

EVLA Advisory Committee Meeting

proposal submission tool pst
Proposal Submission Tool (PST)
  • Mainly a tool to collect form data (web browser)
  • Mostly telescope independent, with “resources” the exception
  • Used to enforce telescope “policy”
  • Coupled to other EVLA tools only loosely, via Portal and databases.
  • Currently also supports GBT.

EVLA Advisory Committee Meeting

pst main processes
PST - Main Processes

Main Processes:

  • Model - retrieve and write data to database
  • Controller - business logic to map user input (from browser) into objects which are then written to database
  • View - the look-and-feel of the interface (done in browser)
  • Validation of various fields - an important and significant part of the tool
  • Help system

EVLA Advisory Committee Meeting

pst model
PST - Model

The Model drives everything, so what’s in there?

  • science information - title, category, “mode”, abstract, scientific justification, and some misc. info.
  • Authors, including which is the PI and “contact author”
  • Sources
  • Resources (telescope hardware setup)
  • “Sessions” (a guide to SB setup)
  • Student Support

Essentially, everything that is necessary to:

  • Referee the proposal
  • Assign telescope time (and money)
  • Automatically generate SBs (mostly for novice users, but experienced users will use this too!)

EVLA Advisory Committee Meeting

pst the view
PST - “The View”

Although the logic is driven by the model, most of the programming complexity here is in the view. How do we do this? 6 main “tabs” in the browser, to represent the 6 main parts of the model. There is also some superstructure, to allow for higher level functions (edit/create, submit, print, choose telescope, etc.), but, again, most of the complexity is in the view for these 6 tabs.

EVLA Advisory Committee Meeting

pst the view1
PST - “The View”

Let’s look at the actual tool…

EVLA Advisory Committee Meeting

pst submit
PST - Submit

We must support both normal “deadline” submissions, as well as “Rapid Response” submissions (this is fundamentally a policy issue). Upon submission, the proposal handling process is initiated. What is stored is a “proposal” (as represented by a Proposal Data Model). This is NOT the Project Data Model, but rather is a guide to the creation of the initial PDM. This is an important distinction, and something we came to a recent agreement on with ALMA (for them this is the Science View of the PDM).

EVLA Advisory Committee Meeting

proposal handling tool pht
Proposal Handling Tool (PHT)
  • Presently, only very initial “wrangling” (view, print), and then splits into GB and VLA specific handling
  • GB and VLA separately support:
    • Adding additional data (edit XML by hand or script)
    • Fixing ‘problems’
    • Pretty-printing, for referees and scheduling committee
  • Future:
    • Referee process
    • Scheduling committee details
  • Points of interest:
    • Requirements for VLA are in hand, but not in the form of the detailed requirements for other areas, rather more like a “User Story”. We need to transform this to real requirements, including the GB process (which have not been written down to our knowledge).
    • Does this go in the PST (with maybe part in the OPT) or as a separate tool?
  • The fundamental output from the PHT is Projects, as represented by a Project Data Model.

EVLA Advisory Committee Meeting

observation preparation tool opt
Observation Preparation Tool (OPT)

This is the process that takes a more generic view of a Project, and turns it into something that can actually be used to command the hardware of the telescope (and run ancillary tasks).

The fundamental output of the OPT is Program Blocks (PBs) (remember that a PB is a collection of SBs, with some extra information - mostly “contingencies”)

It needs to support 3 “levels” of user:

  • Novice (automatic generation of PBs for “standard modes”)
  • Intermediate (this is the tough one!)
  • Expert (allow for script level editing)

EVLA Advisory Committee Meeting

opt commonality with pst
OPT - commonality with PST

Objects in common with PST:

  • Sources
  • Resources (hardware configuration)

Things not in common:

  • Things not in OPT:
    • Front page stuff
    • Authors
    • Sessions (well, kind of - since they “represent” SBs)
    • Student Support
  • Things not in PST
    • Scan builder and organizer
    • PB organizer
    • Detailed correlator setup tool
    • Calibrator tool

EVLA Advisory Committee Meeting

opt detailed design
OPT - Detailed Design

EVLA Advisory Committee Meeting

opt detailed design1
OPT - Detailed Design

Modify PB

EVLA Advisory Committee Meeting

opt detailed design2
OPT - Detailed Design

Modify SB

EVLA Advisory Committee Meeting

opt current status
OPT - Current Status

We currently have an early prototype of the GUI (duplicating the look-and-feel of the PST) in place which will authenticate users and has the most minimal PB functions (create, delete, etc.).

Much of our work here, however, has been on the specification of the details of the objects (the “data models”) for the various parts of the system. We are starting here, knowing that many of the elements in the system will be needed here and will be common. These include the definitions of Proposal, Project, Program Block, and Scheduling Block, and as parts of that, Sources, Resources, and Scans. We start out with an XML document provided by a domain expert, and then turn that into objects. We are currently having detailed discussions with ALMA to attempt to have as much as common in these objects. It is clear that early parts of the process (Proposal) can be common, and that general concepts (major elements of SBs, for instance) can be common; it is not yet clear the level to which true commonality will be achieved.

EVLA Advisory Committee Meeting

opt plan
OPT - Plan

Our plan is to have new major functionality available within the OPT roughly every 3 months. Major milestones are:

  • 30Apr06: framework with minimal functionality
  • 31Jul06: Add VLA calibrator database access, initial spectral setup
  • 31Oct06: Full calibrator setup, more observation setup
  • 31Jan07: VLA mostly supported. Some validation/PB creation
  • 30Apr07: Beginning prototype WIDAR support
  • 31Jul07: VLA fully supported; prototype WIDAR mostly supported
  • 31Oct07: Prototype WIDAR fully supported

EVLA Advisory Committee Meeting

observation scheduling tool ost
Observation Scheduling Tool (OST)

We (well, Barry) have been conducting tests of dynamic scheduling with the VLA during the past 2 (3?) reconfigurations. Again, we consider these as prototypes for the final EVLA subsystem - giving us invaluable information on the practical aspects of dynamic scheduling of a many-element radio interferometer.

EVLA Advisory Committee Meeting

ost block diagram
OST - Block Diagram

EVLA Advisory Committee Meeting

ost lessons learned
OST - Lessons Learned

Here are the lessons learned - I need the full list from Barry. A few things I know:

  • ALMA system didn’t work well (as expected,

per Allen Farris’ email)

  • System is inordinately fond of short SBs
  • Currently effort-intensive (but getting better)

EVLA Advisory Committee Meeting

ost plan
OST - Plan

Here is the plan. I need to get with Barry on this, but my understanding is that he wants the VLA fully dynamically scheduled by the end of ‘06 (yes, he has indeed said that!).

For the EVLA-specific part, we are assigning some effort beginning summer ‘06 to this.

EVLA Advisory Committee Meeting