NextGen Network-Enabled Weather (NNEW) “How Things Work”

1. NextGen Network-Enabled Weather (NNEW)“How Things Work” Registry/Repository

2. Registry Q&A • What general functionality does a Registry/Repository provide? • What are the key roles of the Registry/Repository in the NNEW context? • What types of information is stored in the registry? • What are some typical use cases? • Why was ebXML Reg/Rep 4.0 selected as the registry standard for NNEW? • What is the status and roadmap of the ebXML Reg/Rep 4.0 standard? • How many registries are envisioned in the NAS and elsewhere? • What is the topology of the registries (Federation, Tree Hierarchy, Hybrid)? • What is the anticipated load on registries given the usage model? • How will fault-tolerance be addressed? • How will clients be notified of registry changes (i.e., SAS changes) • How will the Wellfleet registry tie in with the NNEW security architecture? • What are the key challenges associated with the ebXML RegRep 4.0 solution?

3. What general functionality does a registry/repository provide? Registration, Discovery, Queries ebXMLRegRep Taxonomies, Classifications, Relationships Federated Queries, Inter-registry links SOA RegistryRepository Federated Information Manage-ment Standard Metadata Cataloging, Validation, Version Control,Lifecycle Support,Extensible Info Model Secure Architecture SOA Governance Digital Signatures,Audit Trail,Access Control,SAML SSO Events • Analog: FAA KSN or NNEW Wiki ‘back-end’ functionality • For SOA environment, focus is more on machine-to-machine interactions, XML-based metadata Content-Based Event Notification

4. What are the Key Roles of the Registry/Repository in the NNEW Context? Build-Time Discovery • Answers the questions: • “What weather datasets are available to me, and what is their native format?” • “What service interfaces exist to store and access weather datasets of interest?” • Results of service interface query include interface descriptions, including sufficient detail to build a software client (WSDL file and associated XML Schemas) • For NNEW, the two key interfaces are Web Feature Service (WFS) for feature data, and Web Coverage Service (WCS) for coverage data • Roughly speaking, features in the ISO terminology correspond to non-gridded data types, while coverages correspond to gridded data types* *Not the full story – weather measurement data along a flight path is a coverage – we refer to it as a TrajectoryCoverage(as opposed to a GridCoverage. See discussion on NNEW Information Models for more details

5. What are the key roles of the NNEW Registry/Repository? (continued) Run-Time Discovery • Answers the questions: • “What dataset(s) are available in domain X (e.g., ‘SAS’) for product type Y (e.g., PIREP)?” • “What service instances (endpoints) are available to access the dataset(s) I want?” • Run-time aspect provides system agility • SAS product set can be redefined and software does not need to be modified • Concept of ‘domains’ not limited to ‘SAS’ – other domainsmay exist for differentcommunities • Single dataset may belong tomore than one domain

6. Viewing the Cube Domain Taxonomy via the RegRep UI • A Taxonomy is a set of classifications organized in a tree hierarchy • A Taxonomy is a restricted version of the more general Ontology, only supporting Parent/Childrelationships between a set of items

7. Build-Time Dataset/Service DiscoveryUse Case “A small flight services company serving the state of Alaska wishes to incorporate a summary of winds information into an existing ‘quick-look’ Web page that they provide to general aviation users. The company browses the NextGen registry to discover what types of wind data are available, and what services exist to access them. An appropriate data access service is located, and members of the company’s software team download the service and data format schema needed to construct a data access client. A thin layer of additional logic is used to convert the raw data to the desired ‘synopsis’ form, and the updated Web application is made available to pilots over the open Internet for flight planning purposes. The data access client software, which conforms to a service standard used to disseminate many types of weather data, provides the company with easy access to numerous other types of weather data in the future.” Source: NextGen Network-Enabled Weather Use Cases, Version 3.1 https://wiki.ucar.edu/download/attachments/17760853/NNEW-UseCases-v3.doc?version=3

8. Run-Time Dataset/Service DiscoveryUse Case “A client application contacts a registry containing dataset information for the weather cube, and requests information about ‘air temperature’ datasets available in the SAS domain. Metadata for the single, primary, ‘air temperature’ data set are returned, along with metadata for zero or more backup datasets, each with a priority indication describing the ordering in which the backups should be used. For each dataset, metadata describing one or more data access services, along with the data formats supported by each service, are returned. The client application selects the most appropriate service for the primary dataset, and begins accessing data.” Source: NextGen Network-Enabled Weather Use Cases, Version 3.1 https://wiki.ucar.edu/download/attachments/17760853/NNEW-UseCases-v3.doc?version=3

9. Creation of 4-D Cube Experimental SAS Domain “NCAR, NASA, NOAA, and MIT Lincoln Laboratory are collaborating on a research project to produce a new weather data product. The 4-D Wx Data Cube data access mechanisms are used by the project due to its distributed nature. The majority of inputs that are used to create the new product are those provided by the SAS, with the exception of a higher-quality surface winds product. A new ‘Experimental SAS’ is generated for the duration of the research project, and made available in a registry. Following the completion of the project, the ‘Experimental SAS’ designation is removed from the product’s metadata registry.” Source: NextGen Network-Enabled Weather Use Cases, Version 3.1 https://wiki.ucar.edu/download/attachments/17760853/NNEW-UseCases-v3.doc?version=3

10. Semantically-Enhanced Discovery of Datasets • Intelligent discovery • Consults knowledge base/ontology to find alternative meanings • Clustered by: synonyms, parent, children • Enables discovery of resources without exact keyword match

11. Semantically Enhanced Discovery of Datasets WCS Dataset: air_temperature ebXML Registry/Repository Ontology Editor WCS air_temperature Service Provider JMBL Dataset: temperatureAir Domain Expert Ontology Mapper JMBL temperatureAir Service Provider WCS Dataset: surface_air_temperature Query Editor SPARQL Query Engine WCS Architect surface_air_temperature Service Provider Service Discovery UI/Client Service Consumer

12. Why ebXML RegRep 4.0 for the NNEW Registry/Repository Solution? • Universal Description, Discovery, and Integration (UDDI) Registry is common registry-only standard, but…. • It is underpowered with respect to NNEW use cases • Limited query capabilities (especially geospatial queries) • Focus on service discovery – not extensible to dataset discovery • Generally not extensible to support ISO/OGC discovery metadata • Industry support on the wane… • IBM, Microsoft (co-creators of UDDI spec) closed public registries in 2006 • IBM now offers more capable (but proprietary) registry/repository solution • OASIS UDDI Technical Committee now inactive • ebXML Registry/Repository • More flexible, extensible solution • ebXML Reg/Rep information model adopted by OGC over the UDDI information model. • It is in use - DOD Metadata Repository uses freebXML ‘under the hood’

13. DOD Metadata Repository Web GUI • Based upon freebXML Registry OS project • Customized web based GUI client • Manages schemas, data elements, attributes, document type definitions, style-sheets, data structures • Not currently used for registration of services for run-time discovery purposes • ebXML reg/rep does support the run-time use case Source: <https://metadata.dod.mil/mdr/about.htm>

14. Why ebXML RegRep 4.0 for the NNEW Registry/Repository Solution? • NNEW Approach • Adopt ebXML registry/repository – extend as needed • Partner with ebXML registry/repository vendor (Wellfleet) • Formalize extensions with OASIS ebXML technical committee

15. NNEW Registry-Repository Implementation • Core registry extended to handle ISO Metadata and OWL for NNEW • Working with OGC and OASIS to standardize these extensions

16. What is the Status and Timeline of the ebXML Version 4.0 Specification? Candidate Draft #1 Candidate Draft #2 Candidate Draft #3 Candidate Draft #4 Target release date for final OASIS draft spec Specification Development Timeline

17. How Many Registries are Envisioned in the NAS and Elsewhere? • Registry usage focuses on slowly changing information (SAS definition and service endpoints won’t change on a timescale of minutes, but days/weeks/months) • Frequency of registry accesses will be limited within the NAS as a result, limiting the need for multiple registry instances. • Notional number of FAA-hosted registries for scaleability/failover in NAS is ~(4-8), located at FTI backbone locations. Each would hold identical information • Additional registries in other domains (e.g., NWS) would be configured as federated registries. Critical information in external registries may need to be replicated (automatically) in NAS-resident registries if reliability of external registries is not matched to FAA requirements

18. Federated Query for the 4-D Cube • Federated model allows for decentralized governance • Assumes that reliability/performance of partner organizations is ‘impedance matched’ • Simple Redundant registries within each domain NOAA Registry FAA Registry NWS Registry NOAA Registry FAA Registry NWS Registry Oceans Register AIM Register Other Register 4-D Cube Register 4-D Cube Register 4-D Cube Register • Query is ‘fanned out’ to all three registries • Duplicate results, if present, are pruned • Issues: Additional latency, infrastructurereliability mismatches, no FAA governancein the loop for partner organization changes FAA Registry Client (Human or machine)

19. Federated Governance, with Local CachingebXML RegRep Remoteable Registers • Preserves federated governance model • Adds caching – remote register contents replicated in local domain • Allows for changes to optionally be reviewed by human before being ‘pulled’ from remote registry • Similar to PC software update model (at least my Mac asks me before installing changes) NOAA Registry FAA Registry NWS Registry NOAA Registry FAA Registry NWS Registry Oceans Register Other Register AIM Register 4-D Cube Register 4-D Cube Register NWS 4-D Cube Register 4-D Cube Register NOAA 4-D Cube Register • Queries satisfied using information from register caches • Issue: Do we explore this for FY 10 demo FAA Registry Client (Human or machine)

20. How Will Registry/Repository Fault-Tolerance be Addressed • Notional design is replicated registries, geographically dispersed • Software clients support failover to alternate registry – every client would have at least one failover registry • Software clients cache retrieved information to handle NAS-wide registry outage (use most recent service endpoints to retrieve weather data)

21. How will Clients be Notified of Registry Changes (e.g., SAS change) • Scaleability issue • Requests from clients are stateless and asynchronous (spread out in time). This naturally scales pretty well. • Notifications to all registry ‘listeners’ in the NAS does not follow this model – can be 1000’s of listeners • ebXML Reg/Rep specification supports notification, but placing the burden on the regrep to notify 1000’s of listeners of changes with a short time interval is not a good solution • Proposed solution • Leverage distribution server mechanism to deliver registry change events. Registry essentially acts as ‘Origin server’ for notifications • Clients receive events, request updated information from registry • Random wait interval for query on scale of minutes to minimize registry query ‘storms’ (Similar to UPnP discovery mechanism)

22. How will the Registry Integrate with the NNEW/SWIM Security Architecture • Registry implementation leverages Spring-Security implementation, and WS-Security standards • Very similar to approach being used by SWIM • Details of how each will be used in NNEW security solution is still evolving, but registry implementation should be able to adapt with relatively little effort • Role-based security based on the XACML standard not yet implemented in registry or NNEW services – this work is ongoing.

23. What are the Key Challenges with the ebXML Reg/Rep 4.0 Solution? • Industry adoption of ebXML Reg/Rep version 4 specification • Registry/Repository solutions still in relatively early stage of development and use • Multiple implementations desirable • Standardization of ebXML Reg/Rep geospatial profile in OGC • Overlaps with older OGC approach – only ½ the community is buying in at the moment • Alternative is to standardize geospatial profile within OASIS • Performance of ebXML Reg/Rep implementation • Not a significant issue at current scale of usage • May require support for higher-performance database (Oracle) on back end for critical NAS use cases (NWS could continue to use open-source Postgres)

24. Backups

25. 4D Weather Data Cube Domains 4D Weather Cube Domains • Single Authoritative Source (SAS) • Regulatory2a – Government2b – Commercial • Both 1 & 2a • Intermediate • 4a – Government • 4b – Commercial • SAS data (supports ATM decisions) to be available on open & unrestricted conditions • Opportunities for commercial entities to provide weather information still exist • Flexible support for different domains needs to be incorporated into architecture

26. Data Discovery and Access using OGC Services (December ’07 Demonstration) MIT Lincoln Laboratory CIWS Forecast Data Catalog (ebxml) Web Coverage Service (OGC WCS) FAA Technical Center National Center for Atmospheric Research (NCAR) “Discover Available Data Sets” “Get data from time T1 to T2 in spatial region A and return in NetCDF format using a Lambert Conformal Projection” Icing Turbulence Data Web Coverage Service (OGC WCS) NOAA Surface Observations Data Requested Data OGC service-enabled display Client Web Coverage Service (OGC WCS) NOAA NCAR MIT/LL Virtual 4-D Weather Cube

27. Datasets and Metadata Metadata Metadata Dataset Metadata Dataset Series e.g. Precip Data for most recent one year period Registry/Repository Dataset Series Metadata Operates On Service Metadata Data Access Service OperatesOn dataset association(s) can be added at RegistrationTime via registry UI if not included in Service Metadata Data Access Service (e.g. JMBL, WCS, WFS)