Testbed for Mobile Augmented Battlefield Visualization: Summing Up May 10, 2006

1. Testbed for Mobile Augmented Battlefield Visualization: Summing Up May 10, 2006

2. Publications in the Last Year • Xin Zhang, Tazama Upendo St Julien, Ramesh Rajagopalan, William Ribarsky, Pramod Varshney, Chilukuri Mohan, and Kishan Mehrotra . Dynamic Decision Support for Mobile Situational Visualization. AppliedVis 2005. • William Ribarsky, co-editor, Special Issue on Haptics, Telepresence, and Virtual Reality, IEEE Transactions on Visualization and Computer Graphics (November, 2005). • Justin Jang, Peter Wonka, William Ribarsky, and C.D. Shaw. Punctuated Simplification of Man-Made Objects. To be published, The Visual Computer. • Tazama St. Julien, Joseph Scoccinaro, Jonathan Gdalevich, and William Ribarsky. Sharing of Precise 4D Annotations in Collaborative Mobile Situational Visualization. Submitted to IEEE Symposium on Wearable Computing. • Remco Chang, Thomas Butkiewicz, Caroline Ziemkiewicz, Zachary Wartell, Nancy Pollard, and William Ribarsky. Using Urban Legibility to Produce Completely Navigable Large Scale Urban Models. To be published, ACM SIGGRAPH 2006 Short Papers. • Remco Chang, Thomas Butkiewicz, Caroline Ziemkiewicz, Zachary Wartell, Nancy Pollard, and William Ribarsky. Hierarchical Simplification of City Models to Maintain Urban Legibility. Submitted to IEEE Transactions on Visualization and Computer Graphics. • Xin Zhang, Tazama Upendo St Julien, Ramesh Rajagopalan, William Ribarsky, Pramod Varshney, Chilukuri Mohan, and Kishan Mehrotra . An Integrated Path Engine for Mobile Situational Visualization. To be submitted.

3. Matrix of Project Activities and Results

4. Transitions • Presented mobile situational visualization and its relation to homeland security at invited talks at a special session of the AAAS meeting on the National Visualization and Analytics Center (February, 2005), at AppliedVis 2005 (May, 2005), and at an invited presentation for the DHS Regional Visualization and Analytics Centers (January, 2006). • Using work in urban terrain analysis begun here as a foundation, began work on a project funded by ARO for eye-point dependent models applied to terrain analysis and applications such as line-of-sight. • Made a proposal for the DTO ARIVA project that will use, among other things, urban infrastructure visualization. The proposal is now in Phase 2 evaluation. • Established the Southeastern Regional Visualization and Analytics Center, funded by DHS. Among other things, the SRVAC will be looking at critical infrastructure simulations for disaster relief planning and emergency response. The terrain visualization and modeling capabilities developed here will be used.

5. IVAC Scholars Consortium Visualization and Analytics Centers A Partnership with Academia, Industry, Government Laboratories RVACUniversity of Washington RVACPenn. State RVACStanford University GVAC RVACPurdueUniversity DHS RVACUniv. of North Carolina Charlotte, Georgia Tech Detecting the Expected -- Discovering the UnexpectedTM http://nvac.pnl.gov/www.pnl.gov/infoviz NVAC: Pacific Northwest National Laboratory

6. IVAC Scholars Consortium Visualization and Analytics Centers A Partnership with Academia, Industry, Government Laboratories Canada Alaska RVACUniversity of Washington PacificRim DrexelUniversity NY EmergencyManagementPort of Authority Hawaii RVACPenn. State Australia Europe RVACStanford University GVAC(s) RVACPurdueUniversity IndianaUniv.Schoolof Medicine DHS New Zealand NSF RVACUniv. of North Carolina Charlotte, Georgia Tech Bank of America Detecting the Expected -- Discovering the UnexpectedTM http://nvac.pnl.gov/www.pnl.gov/infoviz NVAC: Pacific Northwest National Laboratory

7. Mobile Situational Visualization • Mobile Situational Visualization: An extension of situation awareness that exploits and integrates interactive visualization, mobile computing, wireless networking, and multiple sensors: • Mobile users with GPS, orientation sensing, cameras, wireless • User carries own 3D database • Servers that store and disseminate information from/to multiple clients (location, object/event, weather/NBC servers) • Location server to manage communications between users and areas of interest for both servers and users • Ability to see weather, chem/bio clouds, and positions of other users • Accurate overviews of terrain with accurately placed 3D buildings • Ability to mark, annotate, and share positions, directions, speed, and uncertainties of moving vehicles or people • Ability to access and playback histories of movement User equipped with mobile situational visualization system

8. Mobile Situational Visualization Buttons Pen Tool Mobile Team Drawing Area Shared observations of vehicle location, direction, speed collaborators Collaboration Example

9. Everybody has a location in space and time in the Virtual World Geographic server lookup approach Users Location Servers Data Servers Location Server Traffic Server Annotation Server Weather Server User User GeoData Server User Mobile Sitvis Collaborative Environment

10. Everybody has a location in space and time in the Virtual World Geographic server lookup approach Users Location Servers Data Servers Mobile Sitvis Collaborative Environment Location Server Traffic Server Annotation Server Weather Server User User GeoData Server User

11. Testbed Development Georgia Tech campus -Navigable environment Accurate placement of modeled buildings and trees from multiple sources x Detailed urban component Realistic urban buildings

12. Scenarios and Results Scenario 1 A commander, out of sight of his unit, directs it. He creates waypoints and paths through the mobile sitvis system that individuals in the unit move to. Scenario 2 Three individuals in a unit track a moving subject. They must keep it in sight and coordinate their tracking activities. In both scenarios, working with the full mobile sitvis system is compared with a traditional method (individuals with only radio communication and maps). Tracked subjects followed paths (set for about equal length and number of turns) that were not known to unit members.

13. Scenarios and Results Scenario 2: History of all users’ locations and annotations over a 45 minute session. (Tracked subject is in red.)

14. Scenarios and Results • What did we find? • Mobile sitvis works! • It does as well as traditional method (radio + map) for tracking a moving subject. • It is better than traditional method for command operations that direct multiple units. • It provides significant new capabilities: • -Significantly more accurate location than GPS alone. • -Specific digital annotations in space-time that can be shared immediately. • -Overviews of several moving, annotated entities that can be understood all at once • -Histories for tracking and analysis • This work suggest new scenarios of greater impact.

15. Mobile Decision Engine Visualization Decision Support Module 4D Geospatial Server 3D Interaction User Input Sensors Mobile Situational Visualization + Dynamic Decision Support Collaboration with the Syracuse team

16. Mobile Situational Visualization + Dynamic Decision Support

17. Mobile Situational Visualization + Dynamic Decision Support We have fully integrated the dynamic decision support engine with mobile situational visualization, providing the following capabilities: • a structure for shared interaction and collaboration among mobile users, • general methods for heterogeneous spatiotemporal sensor organization and display, • a decision support module supporting activity recognition, response planning, and behavioral modeling that is integrated with the mobile visualization structure and will accept the mobile users as collaborating agents, • a prototype mobile situational visualization system that employs the decision engine to produce meaningful responses in one or more urban scenarios.

18. Dynamic Decision Support: Grid-Based Approach Djikstra’s single source, single destination shortest path algorithm is used. • The region to be traversed is laid out in a grid, balancing computation cost (no. of grid cells) versus accuracy (edges or nodes in the same grid cell). • Edge relaxation is used to choose vertices. Vertices connected by valid edges are considered, and those with the best value of a quality metric, Q, are chosen. • A probability risk model is applied. A simple zero mean Gaussian distribution with a finite range is used to model point risks. Default values are given for grenades, rifles, etc. • If multiple risks overlap in a grid cell, the multiple threat is computed as [1 - (1 - P1) x (1 – P2) x …… x (1 - Pn)] where there are n threats.

19. Dynamic Decision Support: Grid-Based Approach Red boundary shows selected area in dense urban area (mid-town Atlanta).

20. Mobile Situational Visualization + Dynamic Decision Support Our initial urban scenario is route planning under dynamic threats. Threats of different extents and risks are sighted, placed, and shared by mobile users. The decision engine provides a real-time path that balances on a continuous scale between risk and shortest path (where the mobile user can select the balance). Green path (highlighted with green dots): optimal (in this case low risk) path for selected balance between risks and path length.

21. Mobile Situational Visualization + Dynamic Decision Support The mobile decision engine produces fast, accurate, and usable results Safe route Some risk Shortest route

22. Future Directions • Combine dynamic route planning with line-of-sight to take into account obstructions in determining risk. • Scale up to larger areas. • Take into account moving risks or risks that change in other ways. • Support other decisions.

23. Interactive Visualization of Very Large Urban Spaces • How can one freely navigate very large urban spaces? • A medium size city can have several hundred thousand buildings; a large city can have millions of buildings. • Even if the simplest building models are rendered, there could still be an overwhelming amount of geometry and textures. What should be rendered? Apply knowledge from urban planning: urban legibility. View of Xinxiang, China with over 26,000 buildings.

24. Interactive Visualization of Very Large Urban Spaces • Urban legibility embodies concepts from urban planning about what makes an urban space understandable and more easily navigable. (For example, depict the city around the concepts of paths, edges, districts, nodes and landmarks.) • Can we find an automated way to embody these concepts and thus keep the city legible (and recognizable) at all scales?

25. Interactive Visualization of Very Large Urban Spaces Original (textured) District Simplification with our method Yes, we can shape our automated urban analysis to embody the urban legibility principles. Simplification with Qslim Our simplified model with textures applied

26. Interactive Visualization of Very Large Urban Spaces Skyline at full resolution Landmark preservation Skyline with 7% polygons View-dependent rendering Interactive view with 18% polygons and greatly simplified textures Full resolution Perceptual errors are not very noticeable because conceptual structure (i.e., what’s important) is retained.

27. Q Q Q Q Q Q Q Q Q Q View-Dependent Rendering of Very Large Collection Q • Hierarchical multiresolution organization • View-Dependent LOD for large collections of 3D models Q Q Q Q N Levels Linked Global Quadtrees Q Bounding box Viewpoint Screen Selected LOD

28. Knowledge Visualization: Very Large Urban Spaces Video

29. Organizing Large Collections of 3D Models for Interactive Display Q Q Q Q Q Q Q Q Q Q • Merging of different types and formats • Automated replacement of lower resolution duplicate structures Common format and organization for different types Q Q Q Q Q Linked Global Quadtrees Q Q Q Q

30. Paging, Culling, and Fast Rendering Linked global quadtree Block Block Block Block Block Q Q Q Q Out-of core Storage Block Quadcell

31. Hierarchical, Multiresolution Organization Quadtrees LODs

32. Hierarchical, Multiresolution Organization Quadtree interior nodes Quadtree leaf nodes tree2 sg3 model1 collection3 collection2 collection1 Urban Legibility Detailed Hierarchical Simplification Collections of simple geometry

33. Questions? www.viscenter.uncc.edu

34. Publications from Previous Years • Ernst Houtgast, Onno Pfeiffer, Zachary Wartell, William Ribarsky, and Frits Post. Navigation and Interaction in a Multi-Scale Stereoscopic Environment. IEEE Virtual Reality 2004. • Nickolas Faust and William Ribarsky. Integration of GIS, Remote Sensing, and Visualization. Invited paper, Proc. Remote Sensing 2003 (Barcelona, 2003). • William Ribarsky, editor (with Holly Rushmeier). 3D Reconstruction and Visualization of Large Scale Environments. Special Issue of IEEE Computer Graphics & Applications (December, 2003). • David Krum, Olugbenga Omoteso, William Ribarsky, Thad Starner, and Larry Hodges. Evaluation of a Multimodal Interface for 3D Terrain Visualization. pp. 411-418 IEEE Visualization 2002. • Justin Jang, William Ribarsky, Christopher Shaw, and Peter Wonka. Appearance-Preserving View-Dependent Visualization. IEEE Visualization 2003, pp. 473-480. • William Ribarsky, Zachary Wartell, and Nickolas Faust. Precision Markup Modeling and Display in a Global Geospatial Environment. Proceedings SPIE 17th International Conference on Aerospace/Defense Sensing, Simulation, and Controls (2003). • William Ribarsky. Virtual Geographic Information Systems. The Visualization Handbook, Charles Hanson and Christopher Johnson, editors (Academic Press, New York, 2003). • Zachary Wartell, Eunjung Kang, Tony Wasilewski, William Ribarsky, and Nickolas Faust. Rendering Vector Data over Global, Multiresolution 3D Terrain. Eurographics-IEEE Visualization Symposium 2003, pp. 213-222. • Peter Wonka, Michael Wimmer, Francois Sillion, and William Ribarsky. Instant Architecture. Siggraph 2003, pp. 669-678 (2003). • Tony Wasilewski, William Ribarsky, and Nickolas Faust. From Urban Terrain Models to Visible Cities. Vol. 22(4), pp. 10-15, IEEE CG&A (2002). • David Krum, Rob Melby, William Ribarsky, and Larry Hodges. Isometric Pointer Interfaces for Wearable 3D Visualization. ACM CHI 2003. • William Ribarsky, “Towards the Visual Earth,” Workshop on Intersection of Geospatial Information and Information Technology, National Research Council (October, 2001). • William Ribarsky, Christopher Shaw, Zachary Wartell, and Nickolas Faust, “Building the Visual Earth,” to be published, SPIE 16th International Conference on Aerospace/Defense Sensing, Simulation, and Controls (2002).

35. Publications from Previous Years • David Krum, William Ribarsky, Chris Shaw, Larry Hodges, and Nickolas Faust. Situational Visualization. pp. 143-150, ACM VRST 2001 (2001). • David Krum, Olugbenga Omoteso, William Ribarsky, Thad Starner, and Larry Hodges. Speech and Gesture Multimodal Control of a Whole Earth 3D Virtual Environment. Eurographics-IEEE Visualization Symposium2002. Winner of SAIC Best Student Paper award. • “Acquisition and Display of Real-Time Atmospheric Data on Terrain,” T.Y. Jiang, William Ribarsky, Tony Wasilewski, Nickolas Faust, Brendan Hannigan, and Mitchell Parry, Proceedings of the Eurographics-IEEE Visualization Symposium 2001, pp. 15-24. • “Client-Server Modes of GTVGIS,” Nick Faust, William Ribarsky, and Frank Jiang, Vol. 4368A, SPIE 15th Annual Conference on Aerosense (2001). • “Hierarchical Storage and Visualization of Real-Time 3D Data,” with Mitchell Parry, Brendan Hannigan, William Ribarsky, T.Y. Jiang, and Nickolas Faust, Proc. SPIE 15th Annual Conference on Aerosense 2001, Vol. 4368A. • “Semiautomatic Landscape Feature Extraction and Modeling,” Matthew Grimes, Tony Wasilewski, Nickolas Faust, and William Ribarsky, Proc. SPIE 15th Annual Conference on Aerosense (2001), Vol. 4368A. • “Real-Time Global Data Model for the Digital Earth,” William Ribarsky, Nickolas Faust, William Ribarsky, T.Y. Jiang, and Tony Wasilewski, Proceedings of the INTERNATIONAL CONFERENCE ON DISCRETE GLOBAL GRIDS (2000). • Development of Tools for Construction of Urban Databases and Their Efficient Visualization,” Nickolas Faust and William Ribarsky, Modeling and Visualizing the Digital Earth, Mahdi Abdelguerfi, Editor (Kluwer, Amsterdam, 2001). • Computers & Graphics, Special Issue on Data Visualization (Vol. 24, no. 3, June, 2000), Editors Eduard Groeller, William Ribarsky, and Helwig Loeffelmann.

36. Students Who Worked on Project • Remco Chang • Tom Butkiewicz • Caroline Ziemkiewicz • Xin Zhang • Justin Jang • Tazama St. Julien • David Krum • Olugbenga Omoteso • Jaeil Choi • Weidong Shi • Guoquan (Richard) Zhou • Eunjung Kang • Brendan Hannigan • Mitchell Parry • Matthew Grimes • Ernst Houtgast • Onno Pfeiffer • Joseph Scoccinaro • Jonathan Gdalevich