Operational Deployment of Datasets

1. Operational Deployment of Datasets ATS 690 Jason Burks Holly Allen

2. Introduction • The objective is to show the steps taken to migrate a dataset from a research facility to a weather service operational environment. • Knowledge gained will be beneficial to both the NWS and NASA SPoRT group.

3. Steps Needed • Identifying Dataset for Operational Use • Display and Data Type Implementation Considerations • Setup of Data Ingest • Data Reliability • Testing of Data • Training for Use of Dataset • Use and Evaluation of the Dataset • Application of the Process to Multiple Offices

4. Identify Dataset for Operational Use • Finding Datasets • Dropped in lap • Have a Problem? Find a solution with a new dataset • File Size is Considered • How much bandwidth will it require? • Bandwidth creates ingest limitations • EX: ADAS datasets were pared down because of their size. • How much data can AWIPS hold?

5. Display and Data Type Implementation Considerations • How to Display dataset in AWIPS? • What kind of data is it? • Categorized in terms of how the data will be displayed • Text, Point (obs, profiler), Satellite, RADAR, Gridded (model, LMA) • Does the data make sense? If so,… • Create a color curve, contouring intervals, temporal resolution, projection information, etc. • Add data to menus and volume browser. • Becomes an experimental dataset. • If proven needed, then determine how to get the data.

6. Setup of Data Ingest • How do we get the data? • FTP - push the data to a FTP site and go get it. • LDM - places data in queue. Goes to whoever requests it. • From the LDAD: AWIPS LDAD Regional HQ /Data/Incoming Directory Moved into a local file, where it is stored. Purged off AWIPS. Firewall A script runs from AWIPS to get files from the incoming directory on LDAD.

7. Data Reliability • Once the data is ingested and displayable, is it reliable? • Questions to ask in order to determine if its ready for the next step: • Is it timely? • Does the data look realistic? • High quantity of erroneous data? • Is data consistently available?

8. Testing of Data • It is hard to overcome a ‘tainted mindset’ with forecasters when they are given a bad dataset. • Datasets should be tested and quality controlled before they are viewed by forecasters. • Normally, for a dataset to be implemented, an advocate for that dataset is needed. • The advocate is not a defined person, but usually someone who takes an interest in the new data, and encourages others to look at and use it.

9. Training for Use of Dataset • Normally is done by the Advocate or the Researcher that supplied the dataset. • Researchers might not know how the dataset will be applied operationally, so contact with the forecaster(s) is necessary. • Contact with the Researcher is maintained in order to provide information about data outages, server problems, etc.

10. Use and Evaluation of the Dataset • Qualitative • Immediate feedback (i.e., surveys) • Useful in identifying case studies • Quantitative • Case study analysis. • Shows impact of data: • Was lead time or false alarm ratio improved? • In order to bring a new dataset into full operational use, there must be hard facts and numbers to back up the utility of the dataset. • Not: “Oh, they thought it was great.”

11. Application of the Process to Multiple Offices • Color Curves • Better way to transfer between offices. • Recommend for future AWIPS builds an export/import option. • Compatibility issues between different servers. • Sites with upgrades or builds vs. those not. • Allocate data key for each product • Details for each product is kept under that key. • A way to apply data without getting messed up during each build process. • The extent of the dataset; if it is too large, can make it difficult to send. • After dataset is provided to new offices, information on how to use/modify/troubleshoot data must also be provided. • Written documentation. Ex: If this occurs, then check here, etc. • The process for implementation at the new office begins with training, whether it is done by someone in-house, someone from the previous office, or the initial research team.

12. Summary • Identifying Dataset for Operational Use • Display and Data Type Implementation Considerations • Setup of Data Ingest • Data Reliability • Testing of Data • Training for Use of Dataset • Use and Evaluation of the Dataset • Application of the Process to Multiple Offices • These steps should be done in order to provide a successful transfer from the research facility to operations.