The WWOSC 2014 17 August 2014 Montreal, CA Dr. Mohan Ramamurthy Unidata Program Center

1. Unidata’s Vision for Transforming Geoscience by Moving Data Services and Software to the Cloud The WWOSC 2014 17 August 2014 Montreal, CA Dr. Mohan Ramamurthy Unidata Program Center University Corporation for Atmospheric Research

2. Unidata Niché: Providing comprehensive data services for Earth System sciences. Software: Data Distribution: LDM Remote Data Access: THREDDS, ADDE, and RAMADDA Data Management: netCDF, UDUNITS, and Rosetta Analysis and Visualization: GEMPAK, McIDAS, IDV, and AWIPS II GIS support via TDS (WCS, WMS) and KML and Shapefiles Data: Over 30 data streams provided in real-time Data collection, cataloging, and distribution Both push and pull technologies are used User Support & Training: Direct email support Community mailing lists Annual Training Workshops, Triennial Users Workshops, and Regional Workshops as needed. Community: Equipment Awards to universities; Seminars; Information Commons; Advocacy;

3. The Era of Data-Intensive Science Data is the lifeblood of science, but we need to move from creating data to discovering knowledge.

4. Sea of Data • Given the large data volumes, we must bring the processing to the data – data proximate analysis/visualization

5. “The Cloud” • Private clouds exist within an organization and are often more specialized than a public cloud. • Public clouds are where you will find things like Netflix and Dropbox.

6. Cloud Computing • The “cloud” provides elasticity, virtualization, and on-demand computational, storage, networking, and software services via the Internet. • Any Resource • Any Time • Any Device • Any Place Many types of services running in the cloud

7. Principal Drivers • Diminishing IT infrastructure • Maturity of Cloud computing • Era of Big Data • BYOD: Bring Your Own Device • Increasing Student Expectations

8. Goals • Provide services to our users using one or more public clouds. • Provide software and services to our community which they can run on their own cloud, regardless of the type. • Develop and provide data-proximate processing, analysis and visualization.

9. Unidata Cloud Projects • There are several cloud-related projects currently underway. • Radar and satellite Product generation in the cloud • Migrate Unidata remote data services to a public cloud. • Bring the Integrated Data Viewer (IDV) visualization tool to tablets and post-PC devices via Application Streaming. • Deploy AWIPS II data servers in the Cloud and provide analysis and visualization on AWIPS II “thin clients.” • Provide and maintain a preconfigured computing environment, “Unidata-in-a-Box”.

10. Product Generation in the Cloud • We are now streaming some data streams to Amazon Web Services EC2 and Microsoft Azure cloud instances • We generate products there (radar and satellite composites) • And we stream the resulting products back to Unidata for broader distribution.

11. Remote Data Access Services • Unidata currently provides "Remote Data" services from a single, monolithic server: ‘Motherlode’. • Motherlode • THREDDS • RAMADDA • ADDE • Web

12. Remote Data Access Services • ADDE • Web • RAMADDA • TDS • Unidata is working on providing remote data access services via the cloud, using an Amazon Web Services (AWS) and Microsoft Azure instances.

13. Remote Data Access Services • RAMADDA • TDS • TDS • Web • AWIPS II • AWIPS II • AWIPS II • RAMADDA • RAMADDA • Web • Web • TDS • TDS • AWIPS II • AWIPS II • AWIPS II • By moving to the cloud, we will be able to scale the size and power of the resources needed, leveraging the inherent elasticity of cloud computing. • AWIPS II • AWIPS II • AWIPS II

14. Bringing the IDV to New Devices • The Integrated Data Viewer is used for data analysis and visualization. • It would take a significant resource commitment to bring the IDV to tablet devices. • We are able to avoid this altogether with cloud-based application streaming.

15. Bringing the IDV to New Devices • The interface is streamed to the client device. • All processing is performed “in the cloud”. • In the future, we plan to bring AWIPS II to the cloud by deploying AWIPS II servers on AWS and Azure.

16. Unidata-in-a-Box • We are working to provide a Virtual Machine image containing pre-configured Unidata software that is portable and ready to run.

17. Unidata-in-a-Box • By coupling this with cloud software and services, we are able to remove almost all of the technical overhead of distributing and running these services. • This also scales nicely, allowing us to reach an arbitrarily large number of users.

18. Internal Projects • Unidata is also using the cloud internally. • Software Development • Software Testing • Software Deployment • Benefits: • Reduced hardware requirements • Rapid deployment • Increased security • More efficient workflow

19. Summary • The Unidata community has expressed strong support for us to pursue projects which emphasis cloud functionality and capabilities. • We are making concrete strides on several cloud-related projects that will benefit our users. • Future Unidata cloudefforts will benefit from what we learn from these initial projects. Acknowledgement NSF AGS-1344155