Real-time Data Delivery and Remote Visualization through Multi-layer Interfaces

1. Vinaitheerthan Sundaram, Lan Zhao, Bedřich Beneš,Carol X. Song, Rakesh Veeramacheneni, Peter Kristof Rosen Center For Advanced Computing Department of Computer Graphics Technology Purdue University Real-time Data Delivery and Remote Visualization through Multi-layer Interfaces Work supported by: National Science Foundation

2. NEXRAD II Data • Next Generation Radar (NEXRAD) Level II Data (OR) Weather Surveillance Radar (WSR-88D) Level II Data • This data contains a very fine temporal and spatial resolution of three attributes: reflectivity, Doppler radial velocity and spectrum width • These attributes are vital to understanding, monitoring and predicting severe weather conditions • There are 158 Radar Stations in the country Acknowledgment: Figures are downloaded from websites www.CCSU.edu and www.answers.com.

3. NEXRAD II Data Generation • 3D structure in Radar Data • Radars go through a programmed set of movements, which involve a continuous rotation over 360° in azimuth and a simultaneous increase in elevation by 1° to 3° per complete sweep • Continuous NEXRAD Level II radar data stream • The radar data files vary in size from a few megabytes to tens of megabytes each, depending on the weather conditions. The files are compressed using modified bzip2 • The temporal resolution is 4-5 minutes in severe weather vs. 9-10 minutes in calm weather

4. Availability of NEXRAD II Data in Near Real-time • NEXRAD II Data is available in real-time on the TeraGrid through Purdue resource provider. • Opportunity: • The real-time availability of high-resolution radar data provides an exciting opportunity for a wide spectrum of users ranging from basic ( students) to expert (researchers) if the radar data can be accessed and visualized in 3D in a timely manner. • However, catering to wide spectrum of users presents unique challenges as the requirements for each user differ.

5. Talk Outline • Motivation • Providing real-time data access and remote visualization for a wide spectrum of users • Challenges • A review of challenges in the state-of-the-art systems • A Unique and Versatile System Design • Multi-layer Interfaces • Multiple Service access points • Back-end Architecture – The enabler • Parallel Data Pre-Processing and partial-volume caching • Summary and Future Work

6. Challenges • Limitations in the State-of-the-art • Not handle large amounts of data from many stations over a long time • No direct interaction with the data for users • Not accessible to general public because of complicated interfaces • No access points to third party applications • Challenges • Data management issues: Radar data streams at 50 MB/secs • Native compression format of radar data • Data processing: Computationally expensive processing • Special-purpose hardware (GPU) required

7. Different User Levels and their requirements • Expert Users ( small group ) • Perform in-depth investigation • Examples: Researchers, Emergency Management Personnel • Learners/Casual Users ( large group ) • Access and visualize data for educational or personal purposes • Examples: K12 and College Students, Public • Advanced Users ( small group ) • Explore and evaluate data but don’t have resources • Examples: Graduate Students, Researchers from other domains • Users levels are NOT mutually exclusive • Expert User can be an advanced user when attending conferences and can be a casual user when teaching a class

8. System Design Advanced Users Casual Users Expert Users

9. LiveRadar3D Gadget • Web 2.0 technologies • AJAX, Google Gadgets, Social Networking Applications • Allows rapid dissemination of scientific tools to wide audience • Live Radar 3D • Shows animated Flash movie of 3D visualization of the region near user’s zip • Scalable because movies are pre-generated and stored at the webserver • Granularity: 7 Regions (midwest, south, southwest .. )

10. LiveRadar3D Desktop • Desktop client • Written in C++ • Runs on Linux / Windows • Can be run on standard GPU cards • Uses pre-processed volumes • Leverages Teragrid processing power and local GPU • Advantages • Fast interactive 3D manipulation • Scalable: Supports large number of stations and large intervals of time • Usage Scenario User selects Radar- stations and time period Tool connects to data-access interface and fetch processed volumes Tool renders on the local GPU

11. LiveRadar3D VNC • For users who • Don’t want to download a client • Don’t have the resources such as GPU • Uses VirtualGL/TurboVNC to enable remote 3D visualization • A convenient way to do advanced interactive and collaborative visualization remotely • Browser accessible • Similar to LiveRadar3D Desktop in functionality • Allows full capability available to expert users • Disadvantage – Needs server farm to scale

12. 3rd Party Applications Access • Our architecture is modular and supports fine-grained service access points • Enables developing interesting 3rd party applications such as • Weather prediction application can connect to data access interface • Custom 3D visualizations can be built on pre-processed volumes

13. Services and Backend Data Architecture • In our earlier work, we presented a system that • Accesses NEXRAD II data • Processes it into render-able 3D volumes using Teragrid • Visualizes using Texture-based volume rendering • Disadvantages • On-demand processing => Slow for large amounts of data • Single access point targeted at expert users • Extensions: • Multiple services and access points • Preprocessing data to improve response time and scalability • Volume Caching for easy access and reuse

14. Backend Data Flow Diagram

15. Parallel Data Pre-Processing • Partial 3D volumes • efficient data structure using Hash-maps • spatial/temporal independence property => parallel generation • can be quickly merged to form full 3D volumes that can be rendered • two orders of magnitude smaller than actual data and much smaller than generating full 3D volumes • Generation of partial volumes on Teragrid • Monitors the arrival of new data and pre-processes them and stores the partial-volumes • SRB archives the past-year partial volumes

16. The fully Interactive 3D Viz. Tool for NEXRAD II Data

17. Visualization Images Generated Using Our Tool

18. Hurricane Ike Images (September 14 2008 )

19. Related Work • NEXRAD Data Visualization • Integrated Data Viewer (IDV) by Unidata • National Climate Data Center Java NEXRAD Viewer and Data Exporter • CRAFT Interactive Radar Analysis System ( Java Viewer ) • LEAD Project ( Gateway ) • Remote Visualization • NanoHub – very small data • Insley et al. Parallel RayTracing– slow for 3D interactions • Web Gadgets • Weather.com/Floen.com – simple 2D visualization and animation of radar data only for one station or whole nation

20. Summary and Future Work • Summary of our contribution • A hierarchical and user-oriented design • rich and easy access to NEXRAD II data for a broad range of users. • Improved response time and scalability • parallel data pre-processing and partial volume caching • An integrated end-to-end backend system • radar data retrieval, pre-processing, remote rendering and 3D data visualization. • Future Work • Developing optimized data structures by exploiting the spatial and temporal characteristics of the data

21. Thank you for your attention! Q & A