1 / 49

Grid Data Transport & Grid-based Visualization Utility

This research overview discusses the problem domain, previous work, current development, and related research agendas in the field of scalable visualization for grid-based data transport and visualization utility. It covers topics such as data pipelines, tiling, collaboration, live data, and network-optimized data formats.

mastudillo
Download Presentation

Grid Data Transport & Grid-based Visualization Utility

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid Data Transport & Grid-based Visualization Utility Marcus Thiébaux Hongsuda Tangmunarunkit Carl Kesselman Center for Grid Technologies Information Sciences Institute University of Southern California

  2. Overview • The Problem Domain • Scaling, modalities, pipelines, complexity • Previous work • Active storage, point-clouds, limitations • Related Research Agendas • Rendering, work-flow, scheduling, data services • Current Development • GDT, GVU • Tasks, Tiers, extensions, plug-ins, control Thiebaux/Tangmunarunkit/Kesselman

  3. The Big Gorilla: Scalable Visualization • Larger volumetric spatial extents • Long, fast time-series animation • Tiling, immersion, collaboration • Live data • Validation, steering • Grid resources • Community archives • Distributed computing • Visualization centers • Desktop access Thiebaux/Tangmunarunkit/Kesselman

  4. Browsing Modalities • Structured browsing • Batch-processed movies • Remote rendering, structured streaming • Latency cannot be hidden • Exploratory (unstructured) browsing • On-demand, at-a-glance validation • Debugging sim codes • Visual analysis/searching • Human inference • Communicating results in real-time • image as evidence Thiebaux/Tangmunarunkit/Kesselman

  5. Data Pipelines/Control • Pipelines for visualization tasks • Distributed analysis/rendering stages • The visualization ontology • Dataset types, filter-operations, drawables Thiebaux/Tangmunarunkit/Kesselman

  6. Pipe Complexity • Tiling with multiple channels • Distribution for M x N sorting Thiebaux/Tangmunarunkit/Kesselman

  7. (Pipe Complexity) • Structured collaboration • Negotiating shared controls Thiebaux/Tangmunarunkit/Kesselman

  8. Previous Work • DVC: ‘Data and Visualization Corridors’ • CACR: Technology Road Maps • GVC: Grid Visualization Components • Network-optimized data format • Minimal copies/translation • Active storage model • Dataset staging (semi-persistent storage) • Interactive data access • Parameterized reduction • Browsing path for low viewpoint-latency • Transmit viewpoint independent primitives Thiebaux/Tangmunarunkit/Kesselman

  9. Network-Optimized Data • Linear format for speed • Use memory-mapped files • Translate-copy foreign data • Zero-copy receive • Flexible field directory • Manage fields individually • Wrap external data arrays • Supports scatter/gather send • Multiple content types • Fully support VTK field types • Multiple coordinate and cell types • Optimized point-list rep. Thiebaux/Tangmunarunkit/Kesselman

  10. Active Storage • File-Set Transfer • Custom striping • Data transformation • Reassembly at client Thiebaux/Tangmunarunkit/Kesselman

  11. Reduction Filters Thiebaux/Tangmunarunkit/Kesselman

  12. Parallel I/O Reduction Sorting Transport Rendering TCP/IP Receive Buffer ... ... ... ... Volume Browsing Path • Remote filtering / local rendering • Transmit 3D drawable data • Allow low-latency rotations & viewpoint • Leverage high-performance OpenGL Thiebaux/Tangmunarunkit/Kesselman

  13. Point Sampling • Suited to Volumetric time-series’ • Culled fields are not reused • Block-based rendering wasteful • SGI-Volumizer, VolumePro • Interest ratios of 5-10% the norm, 1% for larger • Features may be temporal • Static detection: LT ~10 Hz • Motion fusion: GT ~10 Hz • Adapted to tet-meshes • Assists stereo fusion Thiebaux/Tangmunarunkit/Kesselman

  14. Monte Carlo Sampling • Minimizes aliasing artifacts • Robert Cook 1986 Thiebaux/Tangmunarunkit/Kesselman

  15. Density vs. Range 256^3 512^3 1 % interest range 5 % interest range Thiebaux/Tangmunarunkit/Kesselman

  16. Spatial Quantization • Nearest neighbor • Lowest cost • Shows actual cell quantities • Tri-linear interpolation Thiebaux/Tangmunarunkit/Kesselman

  17. Cricket Thiebaux/Tangmunarunkit/Kesselman

  18. GVC Limitations • Scheduling • Need to co-locate data/processing • Challenging for scheduling services • Staging • Need tools to adapt to other data formats • Staging tools are clunky • Efficiency/flexibility • Depending on deprecated Globus-Nexus • Need to construct more complex pipelines • Supports only 32bit int, float (VTK limitation) • Need plug-in/custom filters • Adapt to domain-specific analysis Thiebaux/Tangmunarunkit/Kesselman

  19. GV Research • High performance dataset standards • Portable filtering models • VTK • Real-time analysis algorithms • Scalable analysis algorithms • Pre-packaged services • WireGL, Chromium • Focus on end-stage (rasterization) • Granular (OpenGL primitives) • Limited scalability Thiebaux/Tangmunarunkit/Kesselman

  20. Visualization Research • Visual quality/performance trade-offs • Volume rendering not solved • Visual simulation vs. data exploration • Perceptual limitations • Psychophysics/cognition • How we see, understand Thiebaux/Tangmunarunkit/Kesselman

  21. Perception • Cloudy interiors hard to interpret • Lauer/Hanrahan 1991 • Defined surfaces may be misleading • Kindlmann/Durkin 1998 • Fuzziness is meaningful • Perceptual psychology • Transparency is ambiguous without cues • Stereopsis and/or motion necessary • Stereo fusion needs correspondence • Occlusion useful but limiting Thiebaux/Tangmunarunkit/Kesselman

  22. Collaboration Research • Work-flow design • What gets shared, what does not • Structured, remote visual collaboration • Defining the collaborative paradigm • Independent/shared control paths • Negotiating control • Dynamic configuration • Annotation • Multimedia Thiebaux/Tangmunarunkit/Kesselman

  23. Grid Research • Application defined scheduling/staging • Active storage • Co-located data and processing • Spatial decomposition/striping • Domain-specific solutions • Meta-cells • Leverage cache-effects • Fine-grained remote data filtering • Virtual data services • AI planners to synchronize data accesses • Based on data layout, access patterns Thiebaux/Tangmunarunkit/Kesselman

  24. Current Development • Configurable work-flow framework • Flexible topology construction • Default/custom data transport • Extensible data/control encoding • Dataset representations • Task-control paradigm • Selection, filtering, rendering • Custom filters • Plug-in support • Leverage existing VTK tools • Scaling for CPU intensive operations • View-dependent sorting • Multivariate volumes • Complex feature analysis GDT GVU Thiebaux/Tangmunarunkit/Kesselman

  25. Grid Data Transport (GDT) • Globus gt2 • Globus GSI/GRAM/DUROC • Globus-IO • Active Storage • Co-located processing/storage • Distributed streaming • Multi-tiered filter stages • Automatic connectivity • Plug-in support • Blocking/asynchronous reception • Custom allocation/filtering/sending Thiebaux/Tangmunarunkit/Kesselman

  26. GDT Task Nodes Thiebaux/Tangmunarunkit/Kesselman

  27. Node Taxonomy Thiebaux/Tangmunarunkit/Kesselman

  28. GDT SEND Handle • Parallel recipients • Parallel channels per recipient Thiebaux/Tangmunarunkit/Kesselman

  29. Tiers • GDT implements inter-tier communication • Convenient abstraction of pipeline stage • M x N connectivity • Extensible to other data-flow tasks • Parallel File-Set Transfer • Other control paths considered • Performance stats • Flow feedback • Asynchronous commands • View frustum Thiebaux/Tangmunarunkit/Kesselman

  30. Flow & Control Thiebaux/Tangmunarunkit/Kesselman

  31. Grid Visualization Utility (GVU) • Dataset management/transport • Class hierarchy • VTK-style data classes • Extensible field types • Pipeline control fields • Adapted to active storage • Memory mapped file access • RECV/SEND handles • Automatic byte-swap, in-place pre-swap • Multi-thread parallel sending Thiebaux/Tangmunarunkit/Kesselman

  32. GVU Pipeline Thiebaux/Tangmunarunkit/Kesselman

  33. Client/Master/Server • Client • Single point of control • Master • ‘Storage expert’ • Maps client abstraction to physical resource • Located near Server nodes • Server • Active storage work-horse • Synthesizes query-specific dataset Thiebaux/Tangmunarunkit/Kesselman

  34. Master Node • Storage controller • Handles lightweight request • Or ‘smart client’ Thiebaux/Tangmunarunkit/Kesselman

  35. Master Variants Thiebaux/Tangmunarunkit/Kesselman

  36. Active Storage Mapping Thiebaux/Tangmunarunkit/Kesselman

  37. Staging • Active storage is a shared resource • Semi-persistent storage • Custom ‘striping’ of data geometry • Master defines layout • Round-robin/hash Thiebaux/Tangmunarunkit/Kesselman

  38. Staging v. Browsing Thiebaux/Tangmunarunkit/Kesselman

  39. Display Mapping Thiebaux/Tangmunarunkit/Kesselman

  40. Culling Thiebaux/Tangmunarunkit/Kesselman

  41. Filters • Transform/reduce data selection • Down-sampling, culling • Map raw data to drawable primitives • 3D texture bricks • Iso-surfaces • Point clouds • Glyphs • Sprites Thiebaux/Tangmunarunkit/Kesselman

  42. Conversion Filters Thiebaux/Tangmunarunkit/Kesselman

  43. Reduction Filters Thiebaux/Tangmunarunkit/Kesselman

  44. Sampling Filters Thiebaux/Tangmunarunkit/Kesselman

  45. Rendering Thiebaux/Tangmunarunkit/Kesselman

  46. Controls • Control-set is a special case of dataset • Uses same transport interface • Travels with data • Records transformation history • Extensible to new filters • VTK marching-cubes of interest Thiebaux/Tangmunarunkit/Kesselman

  47. Filter Commands Thiebaux/Tangmunarunkit/Kesselman

  48. Coming Soon • GVU to OGL renderer • Out-of-core data rendering • Out-of-frame non-linear snapshots • Glut-based GUI • Buttons, sliders, crystal-ball, PIP… • GWiz Math Lib • Quaternions, eulers, matrices, splines • Performance Instrumentation • Optiputer testbed Thiebaux/Tangmunarunkit/Kesselman

  49. END (6/2/03)

More Related