AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR

1. AWIPS Continuous Technology Refresh (CTR)AWIPS Software CTR TO9: Outbrief September 4, 2008

2. T09 Outbrief Introduction Intro • AWIPS-II T09 Summary • AWIPS-II Architecture Update • AWIPS-II GFE Architecture • AWIPS-II Workstation (CAVE) Capabilities • AWIPS-II Hydro (Advanced Development) • AWIPS-II Stability Testing • AWIPS-II Architecture Evolution • AWIPS-II Demo Scenario • AWIPS-II T09 Delivery ReadMe

3. TO9 AWIPS-II Task OrderSecond Migration Task Order with Emphasis on GFE Summary • Primary task is to migrate Graphic Forecast Editing (GFE) capability to the new AWIPS-II developed architecture • Performed GFE workstation capabilities migration • Performed server side migration to replace IFPServer • Text production generation infrastructure • SmartTool and SmartInit interfaces • Add data ingest capabilities for BUFR, RedBook, and Text Warnings • Added python based derived parameter capability • Migrated AvnFPS application into AWIPS-II • Performed additional risk reduction for Hydro • Hydro View Perspective, SHEF Plug In with live data testing, limited time series display (Initial end-to-end shef ingest to IHFS to timeseries) • Limited T08 DR fixes and numerous infrastructure improvements

4. T09 Made Infrastructure Improvements Summary • Added Python as uEngine scripting language • Developed a JAVA bridge to Python’s “numpy” (Jepp+) • Linked to BLAS for high performance matrix operations • Added Velocity Templates for automatic uEngine script generation (enables both Javascript & Python) • Performed 4D data cube optimization • Improved Grid Tree Data Structure for performance improvements • Improved Localization infrastructure • GFE uses localization dynamically which is a shift from AWIPS-I • Improved CAVE alerting (notification) pattern with a new observer API • Changed CAVE communication to HTTP from JMS for requests • Added a HTTP data retrieve capability for light client operations (Note: IRAD prototype)

5. AWIPS-II: Development Sequence, T09 is GFE+ 3 to 6 month fixed price task orders Summary 2007 2008 2006 2009 2010 Today AWIPS II Development T06 T05 T04 T03 R & D 2 Week Cycles Stability Testing Full Data Load Architecture Development T08 Clean Sheet Start Wx Workstation T09 AWIPS-II Release 1 Forecast Editor T010 Hydrology T011 Comm Deployment Spiral Development with Interim Deliveries

6. AWIPS-II T09 Services and Plug InsESB SOA Plug In Adaptable Future T09 T010 Improve Architecture Airep (22 total) Binlightening Bufrmos Bufrua ccfp 268 Gfe 10808 Goessounding 547 Grib modelsounding 846 Obs Pirep poessounding 598 profiler 560 Radar Recco redbook 730 Satellite Sfcobs shef 3973 Taf Text warning 1115 Eclipse RCP Visualization Plug Ins GfePurgeSrv Bufrmos 801 SmartInit Staging Ingest 14 Core 8 User Interface 22 Library 14 Application GFE 26944 Redbook 619 Python Editor 122 Hydro 10039 aviation(avnFPS) 11539 HdfHttp VTEC Purging Index Archive Notify Adapter Utility AutoBld Product py GFE CAVE Proxy AWIPS-II Services /Light Thick /Light Client Mode Text 678 Enterprise Service Bus SOA Data Type Plug Ins

7. uEngine Scripting Changed to PythonEnables common scripting language, and numpy benefits Architecture uEngine Automatic Script Generation Users (i.e) IFPClient Volume Browser PlotModelGenerator Text Workstation WarnGen CatalogQuery uEngine Script PyProductSrv ScriptCreator JepPool ScriptFactory Jep executeScript Velocity Template Engine Velocity Template Build Response

8. T09 CAVE Eclipse PerspectivesEnables Multiple User Interfaces with High Reuse Architecture Weather Workstation Hydrology Workstation Graphical Forecast Editing Python Editor • Perspectives leverages entire set of CAVE eclipse plug ins • Each perspective can have unique menus, toolbars, and dialogs • User can actively switch back and forth between perspectives

9. MetWatchMonitor IngestSrv IndexSrv T09 Alerting Pattern UpdateAdded Observer Interface to enable multiple users Architecture Data Item EDEX DecisionTree CAVE CAVE Startup Quinlan ID3 Algorithm AlertJobStarter Build Data URI Menu dataURIs URICatalog Message Received MenuUpdater Data URI List ingested dataURIs quartz Timer Data URI Aggregator AvnFPS IAlertObserver topic:edex.alerts

10. T09 Data Cube EnhancementsRedesigned data cube to enhance performance & flexibility Architecture HDF5 Grid Repository Volume Browser RequestJob PlanViewDisplay Loader TimeMatcher BrowserDialog DataCubeContainer BrowserDataCatalog purgeDataCube refreshDerivedParameters examineAttributes getDataRecord performTimeQuery List set|getGridTree <Job> DataCubeManager GridTree Dynamically loaded Python Derived Parameter Python Derived Parameter DerivedParameterGenerator DerivParamScriptsLoader

11. CAVE Localization EnhancementsUses UtilitySrv with a HTTP interface architecture <<SOA Service>> UtilitySrv ESB Http Protocal At CAVE Start Up & On Demand CAVE Config Localization Manager GFE Components that use localization ReferenceSetManager load CAVE Menus save SampleSetManager Notify on updates SampleSetManager Color Maps Base TextFormatter Localization Hierarchy SmartToolManager Local Persistence Of Current Active Localization Site User

12. GFE High Level Implementation ApproachPlugins and Python with Java Bridge GFE GFE Eclipse RCP Plug In Dialogs (60) Perspective Actions (60) Query Grid Edit Tools Core GridManager (51 classes) Interpolation Python Ed Display Resource Sampler Smart Tool Interface Text Formatter Launcher Color Bar Java Python Interface Extended to “numpy” Jepp+ 307 Java Classes 26944Lines SmartTools TextFormatters SmartInit SmartTools TextFormatters SmartInit GfePurgeSrv Jepp+ 114 Java Classes 10808Lines GFE SOA Plug In Services Db access Smart Init Data Types uEngine Tasks History Locking Purging notification ProductSrv SmartInitSrv HDF5 Persistance GFE metadata EDEX Metadata pattern extended to non-partitioned tables

13. T09 GFE Perspective in CAVEGFE Menus & tool bar GFE Pydev Editor in Eclipse Perspective Sample Sets as XML files. Managed by localization. Grid Manager Implemented as Eclipse View Edit Areas stored as XML files. Managed by localization. Notification Pattern enables the updates Spatial Editor Leverages GIS rendering capability of CAVE

14. T09 GFE Server (IFPServer reengineer)Use uEngine by developing a set of GFE tasks GFE Enables MultThreading GetParmListTask GetSiteIdTask GetWeatherVisibilitesTask GetWxDefinitionTask GridLocRequestTask RequestLockChangeTask SaveGfeGridTask SendNotifications StageD2DGridData CommitGridTask GetDbInventoryTask GetDiscreteDefinitionTask GetGridDataTask GetGridHistoryTask GetGridInventoryTask GetGridParmInfoTask GetLockTablesTask GetOfficalDbNameTask ProductSrv ProductSrv HDF5 Persistance uEngine GfeTask Fcst Official Plug In MetaData Concept Extended to handle non-partitioned tables Topo datauri PostgreSQL Metadata Database Table: gfe Table: gfelocktable Table: gfegridhistory Table: spatial_gfe Extracted from AWIPS-I ServerConfig.py

15. AFPSsup.i AFPSDB.i WxMethods.py 35 imports 55lines 57 imports 68 lines 380 lines ConfigurableIssuance.py TextRules.py Grid2DBit.C Grid2DBit.i SiteInfo.py Header.py AFPS.py NOW.py 178 lines 162 lines 735 lines 65 lines 30 lines 65 lines 65 lines 3 lines TimeDescriptor.py ProcessVarList.py GenericReport.py UnitConverter.py CommonUtils.py EditAreaUtils.py TableBuilder.py TextUtils.py SampleAnalysis.py TimeRangeUtils.py 716 lines 202 lines 136 lines 273 lines 884 lines 789 lines 310 lines 127 lines 2775 lines 258 lines ConfigVariables.py ScalarPhrases.py PhraseBuilder.py 1198 lines 2702 lines 4001 lines ForecastNarrative.py IFPImporter.py TextFormatter.py Translator.py 1158 lines 1618 lines 105 lines 664 lines StringUtils.py 399 lines T09 TextFormatters: All AWIPS-I integrated GFE Major Reeng TextFormatter Minor Mods Replace AWIPS-I Interface C++ Python Interface(*.i): 66 Modules, 6767 Lines GIS Interface C++ Modules 25 Modules 6847 Lines MarinePhrases 534 FirePhrases 905 CombinedPhrases.py SimpleTableUtils.py 677 lines 173 lines VectorRelated 954 WxPhrases 1906 DiscretePhrases 2000 * Vtec** GUIs 40 Text 11 Harzard

16. T09 GFE notification designEnables notifications with minimal impact on performance GFE GFE Instance 1 SOA Data Plugin GFE IFPGridDatabase GFE Instance N CommitGridTask LockManager Eclipse RCP Plug In GridManagerView GfePurgeSrv GridMangerVisuals GridParmManager GfeNotification addObserver <Eclipse Job> NotificationManager INotificationObserver sendToObserver msgs <JMS> topic:edex.alerts.gfe <Map> Listeners

17. T09 GFE Parm Locking DesignLeverages RDBMS to improve lock management GFE GFE uEngine Tasks GetLockTablesTask RequestLockChangeTask PostgreSQL Metadata Database LockManager Table: gfelocktable LockDatabase Lock LockNotification <JMS> topic:edex.alerts.gfe

18. T09 GFE Data Type ImplementationsCapabilities of AWIPS-I data types ported into JAVA GFE HDF5 Persistance AWIPS-I Interpolation Functions For all data types ported to JAVA AbstractGridSlice +validTime +gridParmInfo +gridDataHistory Fcst Official • Discrete Slice Reused for Wx Type Topo ScalarGridSlice +scalarGrid Grid2DFloat DiscreteGridSlice +grid Grid2DByte +disceteKey[] DiscreteKey +DiscreteDefintion +subKeys[<DiscreteInstances>] VectorGridSlice +directionGrid Grid2DFloat DiscreteDefinition +modAttrs[] +modifiers[] +nouns[] DiscreteInstance +noun +discreteDefinition +modMap Basic Type for Hazazrds 11 Classes Used to Define Discrete

19. T09 Warning Plug-In Enables GHGMonitorLeverages Warning Plug In MetaData to enable sharing GFE SBN Ingest Endpoint EDEX SOA Plug In plugin-warning (1115L) Extracts Metadata WarningDecoder WarningSeparator parse[SVS,SVRTOR,FFW,FLW, WarngenFLW,RiverproFLW, FLS,WarngenNewFLS, WarngenFollowupFLS, RiverproFLS] ../data/sbn/warning buildGeometry Replaces AWIPS-I VTEC “Active” Table buildPointStr GFE VtecDecoder WarningRecord Extracts Warning Polygon out of Text Warnings Leverages GIS Capabilities of PostGreSQL Table: warning_ugczone MetaData Table: warning

20. T09 GHG Monitor DesignLeverages Warning Plug In MetaData to enable sharing GFE • Common warning • repository feeds both • D2D and GHG. • VTEC elements are • Stored as DB columns D2D Warning Rendering MetaData Table: warning

21. T09 Redbook Graphics CapabilityTwo New Plug-Ins (EDEX+CAVE) D2D SBN Ingest Endpoint CAVE ../viz/redbook (652L) Visualization resource Special Legend Handling Block parsing Rendering EDEX plugin-redbook (890L) Extracts Metadata Decoder*** Message separating ../data/sbn/redbook metadata .viz.ui.personalities.awips uri Redbook hdf5 Persistance Menu XML Adding a Redbook item CAVE Bundle redbook.xml

22. T09 Grid Derived Parameter PatternNew Pattern based on Python Scripts D2D Python Variables Set: Id, Name, Units Core MetLib FORTRAN ported to py Derived Parameters Python Scripts

23. T09 Added Streamline CapabilityPort D2D Fortran into AWIPS-II MeteoLib D2D Right Click Legend Menu Fey Meteolib strmpak.f <Java> Controller <Eclipse Job> ContourManager0 Added D2D FORTRAN to Lib Reuse CAVE Contouring Pattern Dynamic Streamlining While pan and zoom

24. Workstation Warning Generation T09 Delivered T010+ Delivered T08 Delivered D2D WarnGen GUI Life Cycle Rules Dissemination VTEC Database tornado.vm severethunderstorm.vm Templates 3..19 TextDB GIS Capabilities Warning by PolyGon UPDATE LIST Severe Weather Statement Store Set VTEC Generate Text Send to TextWS Text Workstation Editor Restart

25. T09 Hydro Advanced DevelopmentEnd to End Thread from SHEF ingest to TimeSeries Hydro Hydro Eclipse RCP Plugin Dialogs (24) TimeSeries (16 java classes) 101 Java Classes 10039Lines ../sbn/shef (ingest endpoint) 22 Java Classes 3973Lines SHEF SOA Plug In ShefDecoder Shef Posting to IHFS Shef Data Structures EDEX IHFS Tables Token Files

26. T09 Integration Test Bed Cluster MetricsSEDA Load Balancing Performing Well Testing Cluster Balance Remaining Cluster Stability Issues are Network Related and are being worked

27. T09 AWIPS-II Stability Run ResultsBuild 15, 07-Aug to 21-Aug JVM 1.6.0_05 Stability Average Data Ingest Profile Files/hr Garage Collects to a stable level Stable CPU Useage ~100G HDF5 Persistance MBytes

28. T010/11 AWIPS-II Infrastructure Improvements in Progress Arch. Evolution • Improve object serialization (Transition to JAXB and THRIFT) • Improve performance of request/response interactions • Improve persistence of binary objects to HDF5 • Eliminates NFS for data access and enables “light client mode” • Consider Migrating to a pure “C” Grib1/2 Decoder from the Unidata One • GRIB2 decoding using J2000 is inadequate • Use GLUEGEN to generate JAVA interface to “C” decoder • Upgrade ESB and improve ingest performance • Improve HDF5 locking under high transaction rates • SOA Data Plug In Pattern upgraded to use “annotations” (JPA) for MetaData definitions • Eliminates Two XML files from plugin pattern, simplifies development • Raytheon Investment: • Improve separation between EDEX core and Data plugins • Lowers maintance cost and facilitates partial deployment • Improve build process so that only selected set of plugins can be built and deployed • Improve separation of CAVE core plug ins from application level plug ins

29. T09 Demo ScenarioFeatures that illustrate the architecture Demo • D2D Features • Wind Streamlines (Zoom and Pan) • Redbook Graphics • Derived Parameters (Adding {i.e. DpD.py}, modifying) • GFE Features • GridManager (Show eclipse “ViewPart” features) • Running Smart Tools and editing, interpolation • Formatter Launcher (NOW product), Make Hazards • Hydro Perspective • Timeseries from ingested SHEF data

30. T09 Delivery ReadMe • If installing on Linux Fedora 6, the library installed by (yum install compat_lib2c-34) is required for Python module "numpy". • If trying to run disconnected from a network, activemq multicast must be turned off. • Remove the XML element <transportConnector name="discovery"... from activemq.xml in ./activemq/conf • Ready for end to end testing • End-to-end testing will be available in T011 • T09 Complete functionality • GFE: Spatial Editor, Color Legend Menus, Status Bar • GFE: Grid Manager, Interpolation • GFE: Edit Areas, Weather Element Browser, Data Pick Values • GFE: Pencil Tool, Sample Tool, Sample Sets • GFE: Smart Tools, Smart Init • D2D: Streamline rendering for Wind parameters • D2D: WarnGen (Note: TRRs from delivery testing) • Partial implementation • GFE: Formatter Launcher (Products: NOW, PNS, RFD, SPS, ESF) • GFE: GHG Monitor (Requires live warnings to be ingested) • GFE: MakeHazard GUI (Run Button is in T010) • D2D: Derived parameters (dZ, RH, wSp, DpD, DpT, SHx, EPT, PoT, mixRat, and TW) Note: Testing the adding of new derived parameters by just writting Python Scripts and adding them to the CAVE derived parameter folder • D2D: Profiler data, Model Sounding Data, Satellite Sounding, MOS, Redbook Graphics ingest • D2D: Redbook rendering (selected items under the NCEP/Hydro menu) • AvnFPS: Ceiling Vis Dist, Configuration, Metars, Wind Rose GUI, TAFs, Weather Plot GUI

31. T09 Delivery ReadMe • Advanced development • Hydro Perspective, Shef Ingest, Time Series (partial) • Do not test • Anything that is comm related • GFE: ISC and Verify • GFE: Items under Hazard Menu except MakeHazards GUI • Issue: • HDF5 locking performance in cluster for high transaction rates (fix identified and part of T010) • Affects ingest of BUFR MOS