awips continuous technology refresh ctr awips software ctr n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR PowerPoint Presentation
Download Presentation
AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR

Loading in 2 Seconds...

play fullscreen
1 / 28

AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR - PowerPoint PPT Presentation


  • 123 Views
  • Uploaded on

AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR. T010 TIM Nov. 5, 2008. TIM Topics. CLI update (Mark Fegan) – - textdb work in T010 delivery, Python - uEngine script runner code complete, script management in progress Subscription update (trigger equivalent functionality)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
tim topics
TIM Topics
  • CLI update (Mark Fegan) –
    • - textdb work in T010 delivery, Python
    • - uEngine script runner code complete, script management in progress
  • Subscription update (trigger equivalent functionality)
    • Design in progress
  • Smart Tools "architecture" in AWIPS II
  • Smart Tool migration discussion      - Things that need to be done            - Approach going forward
  • Q&A - open discussion
command line tool suite requirements

Number

Description

CLTS 1

The AWIPS II Command Line Tool Suite shall preserve the command line interface of the legacy textdb tool

CLTS 2

The AWIPS II Command Line Tool Suite shall preserve the command line interface of the legacy handleOUP tool

CLTS 3

The AWIPS II Command Line Tool Suite shall include a command line interface to the µEngine script runners

Command Line Tool Suite Requirements
command line tool suite design concept
Command Line Tool Suite Design Concept
  • This diagram shows the basic interfaces shared by the CLI Suite tools
  • the user runs the CLI tool from the command line
    • data is provided via standard input
    • results are returned via standard output
  • The CLI tool communicates the request to the appropriate EDEX/Mule service endpoint via HTTP
  • The service endpoint performs the required actions and returns the result to the CLI tool
command line tool suite use case
Command Line Tool Suite Use Case
  • User enters a command line including operation flags and operands
  • Tool decodes the command line
  • Tool reads data from standard input
  • Tool converts command line and data to Canonical XML message
  • Tool sends message to EDEX SOA Endpoint
  • Tool receives response from EDEX SOA Endpoint
  • Tool decodes message
  • Tool writes contents of message to standard output
  • Tool returns exit status to User

The User utilizes a Command Line Interface Tool to submit a request to the EDEX Server

command line tool suite sequence diagrams
Command Line Tool Suite Sequence Diagrams
  • This diagram illustrates the execution basic sequence used by the CLI tools
  • This execution sequence leads to a set of components/classes that are shared by the various tools
command line tool suite shared components
Command Line Tool Suite Shared Components
  • This diagram illustrates the expected component design of a CLI Tool
    • The CLI library will be a class library providing common functionality
      • Message: handles conversions to and from XML
      • read/write provides façade for all I/O operations
      • CL Parser provides standard command line parser
      • Comm Link isolates CLI Tool ↔ EDEX SOA Endpoint communications
command line tool shared library
Command Line Tool Shared Library

This is the class diagram for the library shared by all components of the CLI Tools suite. It can be available for other modules as well.

uengine cli tool design concept
uEngine CLI Tool Design Concept
  • This diagram shows the basic interactions for the uEngine tool.
    • The user
      • provides commands via command line
      • provides data (scripts) via STDIN
      • receives responses via STDOUT
    • uEngine communicates via HTTP
      • execution requests are sent to ProductSrv
      • subscription requests are sent to SubscriptionSrv

Standard CLIS interface

Proposed

subscription

endpoint

uengine cli tool design concept1
uEngine CLI Tool Design Concept

The basic idea is to provide a command line interface for the uEngine that has functionality similar to that provided by the textdb CLI Tool. That tool allows clients, either software or human, to perform standard database tasks; inserts, retrieval, certain table updates, via a simple command structure.

The uEngine CLI will allow the user to perform two basic uEngine related functions:

1) Execution of a uEngine script with the results returned

2) Routine maintenance of product subscriptions

Use Cases for these two main scenarios are covered on the next few slides.

uengine cli tool use cases
uEngine CLI Tool Use Cases
  • This diagram illustrates the basic Use Cases for the uEngine CLI Tool
  • There are two main actors
    • User: runs uEngine
    • EDEX: the EDEX server
  • There are two main Use Cases
    • Run Script: User uses uEngine to execute a uEngine script
    • Manage Subscription: User uses uEngine to manage a subscription
uengine cli tool use case 1
uEngine CLI Tool Use Case 1

Run Script

  • User executes the uEngine run script command
  • uEngine decodes the command line arguments
  • uEngine reads the script from STDIN
  • uEngine converts the script into a canonical message
  • uEngine sends the message to EDEX for processing
  • EDEX processes the message and returns the result in canonical XML format
  • uEngine decodes the return message
  • uEngine writes the result to STDOUT
  • uEngine returns status code to User

The User executes uEngine to submit a script to EDEX. EDEX executes the Scripts and returns the result to User.

uengine cli tool use case 2
uEngine CLI Tool Use Case 2

Manage Subscription

  • User executes the uEngine manage subscription command
  • uEngine decodes the command line arguments
  • uEngine converts command line into a canonical message
  • uEngine sends the message to EDEX for processing
  • EDEX processes the message and returns the result in canonical XML format
  • uEngine decodes the return message
  • uEngine writes the result to STDOUT
  • uEngine returns status code to User

User executes uEngine to submit a subscription request to EDEX. EDEX executes the request and returns the result to User.

Note: This scenario assumes no input from STDIN. One variation would have script contents input from STDIN or a file.

uengine cli tool use case 22
uEngine CLI Tool Use Case 2
  • Use case 2, and its variations, depends on implementation of a subscription management endpoint in EDEX
  • Initially, this use case will be stubbed in in the uEngine CLI Tool
uengine cli tool interface
uEngine CLI Tool Interface
  • The basic interface is a combination of command line arguments and standard input
    • see typical command line (below)
    • This table describes the command line flags
    • Only the highlighted options will be initially implemented
  • When required, the uEngine script is read from standard input
    • file redirection may also be used
  • Results are written to standard output; errors are written to standard error
  • Process return values:
    • 0 for successful execution
    • 1 if any error occurs

uEngine -r python -s MESSAGE "Hello from Omaha" < data/HelloWorld.txt

subscription service design concept
Subscription Service Design Concept
  • This diagram shows the existing subscription service
  • The Subscription Manager is a JVM based singleton
  • Script management is handled through ProductSrv
    • Subscriptions are limited to Rhino/JavaScript scripts
  • Script fulfillment is handled by NotifySrv and AutoBldSrv
    • scripts are executed based on arrival of a data product
    • subscription notifications are sent to specific, registered, clients
  • Scripts generally create products in a standard image format.
subscription service design concept1
Subscription Service Design Concept
  • This CONOPS for subscription was based on the need for CAVE to automatically update the display as data was ingested on EDEX
    • This proved inadequate for the auto update capability required by CAVE
  • This CONOPS was replaced in TO6 with CAVE’s auto update facility which is based on data ingest w/o utilizing product creation scripts
subscription service design concept2
Subscription Service Design Concept
  • This design, particularly the (JVM based) singleton implementation of the Subscription Manager) had certain notable problems
    • Product subscriptions tended to fail on clustered installations about 50% of the time
    • Orphaned subscriptions exits since CAVE does not cancel subscriptions on exit
subscription service design concept3
Subscription Service Design Concept
  • This diagram shows the revised design of the subscription management functionality
    • the Subscription Manager is merged into NotifySrv and renamed SubscriptSrv
    • an EDEX client interacts directly with SubscriptSrv to manage subscriptions
    • SubscriptSrv interacts with the data store and passes messages to the script runner (AutoBldSrv)
    • AutoBldSrv will be refactored to allow multiple script runners as well as product and timer based execution
subscription service design concept4
Subscription Service Design Concept

Subscription Fulfillment

  • AutoBldSrv is triggered by system
  • AutoBldSrv obtains subscription list from Subscription Manager
      • Subscription Manager obtains subscription list from database
  • AutoBldSrv obtains script runner from uEngine factory
  • AutoBldSrv triggers the script and returns result to the system

4

3

1

2

subscription service design concept5
Subscription Service Design Concept

Script Management

  • EDEX client creates a subscription management request
  • Subscription request is passed via HTTP to the subscription manager
  • Subscription manager accesses /updates the subscription tables
  • Subscription manager prepares and returns a response to the EDEX client

1

2

4

3

subscription service design concept6
Subscription Service Design Concept

Additional Considerations

  • Subscription management will be provided via the uEngine CLI tool
    • provides a simple command line interface
    • may be supplemented via Python/TK or another GUI interface
  • Initially, we expect to provide 3 uEngine script runners
    • a Python script runner based on the existing Python uEngine
    • a JavaScript script runner based on the existing JavaScript uEngine
    • a system script runner based roughly on the adapter service
      • the idea is to allow running external scripts as if from command line
smart tool architecture layers
Smart Tool Architecture Layers
  • JEP – Java Embedded Python
    • Modified for improved performance and numpy support
  • PythonScript – Java class that wraps JEP calls
  • SmartToolController (new in TO10)‏
    • Java class that extends PythonScript
    • Manages smart tools in a single python interpreter
  • SmartToolInterface – python class that assists the SmartToolController with managing and running smart tools
  • Tool – Java class ported from Tool.py that prepares and executes smart tools through the SmartToolController
awips ii smart tool architecture overview
AWIPS-II Smart Tool Architecture Overview

<Python>

SmartScript

<Python>

BaseTool

Python wrapper

functions (95)

Python utility

functions (29)

<Java>

PythonScript

<JEPP>

Java to cPython

Bridge + numPy

datatypes

cPython

+

Module libs

AWIPS-II interface

To Python by Jepp

<Java>

SmartToolController

<Java>

SmartInitScript

<Java>

ProcedureScript

<Java>

FormatterScript

Runs GFE

SmartTool python

Runs GFE

SmartInit python

Runs GFE

Procedure python

Runs text formatter

python

<Java>

SmartUtil

+runTool()

<Java>

DataManager

<Java>

WEGroupManager

+5 functions

Central Singleton

<Java>

IFPClient

<Java>

ParmManager

<Java>

ReferenceSetManager

<Java>

GFESpatialDisplay..

<Java>

SampleSetManager

+28 functions

+18 functions

+22 functions

+24 functions

+40 functions

137 Functions Implemented In Core GFE Code

smart tool migration
Smart Tool Migration
  • Imports from Numeric must be changed to import from numpy
  • Wx tools must return a tuple of (numpy byte array, keys)‏
    • It appears that with numpy operations sometimes the array comes back as an array of type float32 (e.g. [0, 1, 0] could come back as [0.0, 1.0, 0.0])‏
    • If this occurs, at the bottom of the tool just ensure the wx is still of type byte
      • wxValues = asarray(wxValues, 'int8')‏
  • Any calls on the C objects that did not go through SmartScript will have to be updated to call the correct Java methods
analysis of rnk tools
Analysis of RNK tools
  • RNK delivered Omaha 173 smart tools the other week
  • Analysis (all estimates)‏
    • 74 should work with TO9 with minor changes (previous slide)‏
    • 36 more should work with TO10 delivery with minor changes
    • 20 obsolete - Adjust tool makes others obsolete, some are point-based or parm-based tools
    • 30 incompatible – ISC, hardcoded directory paths, data type not currently supported in AWIPS II
    • Others unsure on compatibility – e.g. tkinter