1 / 29

Grids for Real-time and Streaming Applications

Grids for Real-time and Streaming Applications. I-Light Symposium IUPUI September 22 2005 Geoffrey Fox Computer Science, Informatics, Physics Pervasive Technology Laboratories Indiana University Bloomington IN 47401 http://grids.ucs.indiana.edu/ptliupages/presentations/

sykesc
Download Presentation

Grids for Real-time and Streaming Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grids for Real-time and Streaming Applications I-Light Symposium IUPUI September 22 2005 Geoffrey Fox Computer Science, Informatics, Physics Pervasive Technology Laboratories Indiana University Bloomington IN 47401 http://grids.ucs.indiana.edu/ptliupages/presentations/ gcf@indiana.eduhttp://www.infomall.org

  2. Four Data Streaming Application Areas • Data Assimilation applied to link the data deluge (satellites, sensors, seismometers) in real time to large scale parallel simulations • Use in Earthquake Science • Department of Defense (and Homeland Security) have built the Global Information Grid with a target architecture NCOW (Network Centric Operations and warfare) • They submit no jobs; rather stream data to brokers from they are filtered and distributed • Includes their rather dated distributed simulation HLA • Audio-Video Conferencing implemented with services and Grid messaging • Hand-held Grid linking PDA/cell-phones to Grids

  3. Key Concepts • Grid of Grids (System of Systems) allows “library” approach to composing Grids • Service Oriented architectures (Web or Grid services) are attractive for many/most distributed systems • There are many applications that are NOT best considered as jobs and files (classic Grid) but rather as streams and filters (services) • Services exchanging messages becomesServices exchanging streams (sets of messages) • Publish-Subscribe messaging gives better QoS and management than point to point messaging with negligible performance loss

  4. Typical use of Grid Messaging in NASA Sensor Grid GIS Grid Grid Eventing Datamining Grid

  5. GIS and Sensor Grids • OGC has defined a suite of data structures and services to support Geographical Information Systems and Sensors • GML Geography Markup language defines specification of geo-referenced data • SensorML and O&M (Observation and Measurements) define meta-data and data structure for sensors • Services like Web Map Service, Web Feature Service, Sensor Collection Service define services interfaces to access GIS and sensor information • Grid workflow links services that are designed to support streaming input and output messages • We are building Grid (Web) service implementations of these specifications for NASA’s SERVOGrid

  6. Grid Eventing via NaradaBrokering • The Scripps Orbit and Permanent Array Center (SOPAC) GPS station network data published in RYO format is converted to ASCII and GML

  7. Grid Messaging with Grid of Grids Architecture Filter or Datamining Grid DoD Principles Sensor Grid Post afterProcessing Post beforeProcessing Web Feature Service NaradaBrokering Notify WFS (GIS data) Database Archives Subscribe HPSearch Manages GIS Grid WS-Context Stores dynamic data GeographicalInformation System

  8. Earthquake Faults from the WMS Client

  9. WMS uses WFS that uses data sources <gml:featureMember> <fault> <name> Northridge2 </name> <segment> Northridge2 </segment> <author> Wald D. J.</author> <gml:lineStringProperty> <gml:LineStringsrsName="null"> <gml:coordinates> -118.72,34.243 -118.591,34.176 </gml:coordinates> </gml:LineString> </gml:lineStringProperty> </fault> </gml:featureMember>

  10. Google Map Client Archived Real Time Databases withSERVOGrid Faults Sensor Grid Google Central HTTP WFS2 WFS1 Google Map Client Helper Services SOAP DoD and Homeland Security can in a crisis combine custom geo-referenced data with that available from hundreds of thousands of computers from Microsoft, Yahoo and Google Just build simple services using Interoperability standards! UDDI

  11. Real Time GPS and Google Maps Subscribe to live GPS station. Position data from SOPAC is combined with Google map clients. Select and zoom to GPS station location, click icons for more information.

  12. Integrating Archived Web Feature Services and Google Maps Google maps can be integrated with Web Feature Service Archives to filter and browse seismic records.

  13. Google Maps as Service accessed from our WMS Client

  14. Google plus GIS Grid Integrated with Los Alamos Critical Infrastructure Simulations for DHS Natural Gas Layer Energy Power Layer

  15. Traditional NaradaBrokering Features

  16. Mean transit delay for message samples in NaradaBrokering: Different communication hops 9 hop-2 hop-3 8 hop-5 7 hop-7 6 5 Transit Delay (Milliseconds) 4 3 2 1 0 100 1000 Pentium-3, 1GHz, 256 MB RAM 100 Mbps LAN JRE 1.3 Linux Message Payload Size (Bytes)

  17. Average Video Delays for one broker – Performance scales proportional to number of brokers Multiple sessions One session Latency ms 30 frames/sec # Receivers

  18. Consequences of Rule of the Millisecond • Useful to remember critical time scales • 1) 0.000001 ms – CPU does a calculation • 2a) 0.001 to 0.01 ms – Parallel Computing MPI latency • 2b) 0.001 to 0.01 ms – Overhead of a Method Call • 3) 1 ms – wake-up a thread or process (do simple things on a PC) • 4) 10 to 1000 ms – Internet delay • 2a), 4) implies geographically distributed metacomputing can’t in general compete with parallel systems • 3) << 4) implies a software overlay network is possible without significant overhead • We need to explain why it adds value of course! • 2b) versus 3) and 4) describes regions where method and message based programming paradigms important

  19. Databases and/or Sensors Data Data Filter Filter Filter Data Filter Data OGC or OGSA-DAIGrid Services AnalysisControl Visualize Grid Data Filter This Type of Grid integrates with Parallel computing Multiple HPC facilities but only use one at a time Many simultaneous data sources and sinks HPC Simulation Grid Data Assimilation Other Gridand Web Services Distributed Filters massage data For simulation SERVOGrid (Complexity) Computing Model

  20. DoD Core Services and WS-* plus OGSA I

  21. DoD Core Services: WS-* and OGSA II

  22. Gateway Gateway Gateway Gateway XGSP Media Service WS-Context Collaboration Grid NaradaBroker Audio Mixer HPSearch Video Mixer UDDI NaradaBroker Transcoder Thumbnail WS-Security Replay NaradaBroker Record Annotate SharedWS SharedDisplay WhiteBoard

  23. GIS TV GlobalMMCS SWT Client Chat Video Mixer Webcam

  24. PDA Download video (using 4-way video mixer service) Desktop PDA

  25. NB Features Released 2005-2006 • Production implementations of WS-Eventing, WS-RM and WS-Reliability. • WS-Notification when specification agreed • SOAP message supportand NaradaBrokers viewed as SOAP Intermediaries • Active replay support: Pause and Replay live streams. • Stream Linkage: can link permanently multiple streams – using in annotating real-time video streams • Replicated storage support for fault tolerance and resiliency to storage failures. • Management: HPSearch Scripting Interface to streams and brokers (uses WS-Management) • Broker Topics and Message Discovery: Locate appropriate • Integration with Axis2 Web Service Container (?) • Support of IBM MQSeries functionality and Legacy MQSeries Systems as a Grid of Grids gateway • Better Security tracking endless changes of WS-Security • High Performance Transport supporting SOAP Infoset

  26. What to Remember • Grids are Services exchanging Messages • Developing messaging paradigm for Grids using Message Oriented Middleware or Software Overlay Network or Grid Service Bus • Web Service container replaces computer • Service replaces process • A stream is an ordered set of messages • NaradaBrokering replaces MPI with different applications and different requirements • Service Internet replaces Internet: messages replace packets • (Sub)Grids replace Libraries in Grid of Grids Architecture • Use W3C OASIS OGC GGF standards

  27. Location of software for Grid Projects in Community Grids Laboratory • htpp://www.naradabrokering.org provides Web service (and JMS) compliant distributed publish-subscribe messaging (software overlay network) • htpp://www.globlmmcs.org is a service oriented (Grid) collaboration environment (audio-video conferencing) • http://www.crisisgrid.org is an OGC (open geospatial consortium) Geographical Information System (GIS) compliant GIS and Sensor Grid (with POLIS center) • http://www.opengrids.org has WS-Context, Extended UDDI etc. • The work is still in progress but NaradaBrokering is quite mature • All software is open source and freely available

More Related