PTLIU Laboratory for Community Grids Geoffrey Fox, Wenjun Wu, Ahmet Uyar, Hasan Bulut

1. CTS04 San Diego 19 January 2004 Design and Implementation of Audio/Video Collaboration SystemBased on Publish/subscribe Event Middleware PTLIU Laboratory for Community Grids Geoffrey Fox, Wenjun Wu, Ahmet Uyar, Hasan Bulut Computer Science Department Indiana University, Bloomington IN 47404 http://www.globalmmcs.org gcf@indiana.edu uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

2. A/V Collaboration Systems • H323 H.323 is defined as an umbrella standard specifying the components to be used within an H.323-based environment. • SIP The Session Initiation Protocol (SIP) defines how to establish, maintain and terminate Internet sessions including multimedia conferences • Access Grid enhanced Mbone A/V tools ( VIC, RAT ) Internet 2 network ( Multicast support ) uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

3. XGSP Collaboration Framework • To integrate heterogeneous systems into one collaboration system • A unified, scalable, robust “overlay” network is needed to support A/V and data group communication over heterogeneous networking environments. • A common A/V signaling protocol has to be designed to support interactions between different A/V collaboration endpoints. • Different A/V endpoints should collaborate in the same collaboration session. uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

4. Group Communication in Collaboration • Collaboration applications usually need a group communication service for both multipoint data and control information delivery • Centralized conferencing systems usually depend upon a single conference server • Distributed conferencing systems use IP multicast Access Grid uses Internet2 multicast for audio/video transmission. • Problems: Centralized conferencing systems don’t have good scalability IP multicast has not become ubiquitously available uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

5. Data base Narada Broker Network (P2P) Community For message/events service Broker Broker (P2P) Community Resource Broker Broker Broker (P2P) Community Software multicast Broker (P2P) Community uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

6. Narada Brokering • Narada Brokering from the Community Grid Labs is adopted as a general event brokering middleware • supports publish-subscribe messaging model with a dynamic collection of brokers • provides services for TCP, UDP, Multicast, SSL and raw RTP clients • Supports the communication through firewalls and proxies • operate either in a client-server mode like JMS or in a completely distributed JXTA-like peer-to-peer mode. uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

7. Narada Brokering • Covers the heterogeneity of network transportation and provides unified multipoint transportation API • Software multicast • Communication over firewalls and proxy boundaries • Communication over multiple transports • Application level Quality of Service • Filter messages to slow (collaborative/real-time) clients • Provides robust, scalable and high efficient multipoint transportation services • Availability and scalability • Efficient routing and bandwidth utilizations uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

8. A/V Collaboration over publish/subscribe Middleware • video streams (VS) VS {v1, v2, … vm } • audio streams (AS) AS { a1, a2, … an } • A/V endpoints: E1, E2, … En Each endpoint may send a single or multiple video streams, but only send an audio stream • Different types of A/V endpoints have different collaboration capabilities. • Multicast endpoints are able to receive multiple video and audio streams, display all the video streams in their screens, and mix all the audio streams by themselves • Unicast endpoints like Polycom Via Video can only receive and play a single video and audio stream uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

9. Publish/subscribe of A/V • A stream in VS and AS is regarded as a “topic” • Each RTP packet from this stream is regarded as an “event” for this topic • Only the sender of this stream can “publish” A/V events to this topic • Other endpoints need to subscribe to this topic in order to receive the stream data • Create mixed video and audio stream topics for unicast endpoints uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

10. Major Events for each stream topic uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

11. A/V Collaboration Architecture uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

12. RTP packets encapsulation • RTPLink: RTP Packets ~ Narada Brokering Event • Every legacy A/V client needs one corresponding RTPLink to be set up at a broker within broker network. • Unicast RTPLink: Integer Topic Numbers for RTP and RTCP • Multicast RTPLink: A reflector between NaradaBrokers and multicast groups, encapsulating raw RTP packets from a multicast IP address to RTP events, publishing these events to NaradaBrokers, and forwarding the data it receives from broker network on the same IP address uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

13. Audio Mixer, Video Mixer, Image Grabber • Audio Mixing The audio mixer creates a mixed audio stream from all the audio streams in the session • Video Mixing Video mixing makes the unicast users watch the pictures of multiple participants in a meeting through one video stream • Video Thumbnail visualize the VS set in the session, embedded into the control panel of each endpoint, which Image grabbers capture video streams and save them as static JPEG files. • All the media processing components can be distributed among the pool of the media servers connected to NaradaBrokering infrastructures. uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

14. H.323, SIP Gateway Servers, A/V Session Server • H.323 and SIP gateway transform these protocol specific messages into XGSP signaling messages so that H.323 and SIP A/V endpoints could communicate with the XGSP A/V session server • The session server implements session management logics • creating/destroying A/V sessions • allowing endpoints to join/leave session • Allowing users to make audio/video selection, managing A/V application components uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

15. Implementation • Java Media Framework (JMF) : • Openh323 is basis of H323 Gateway • NIST SIP stack is basis of SIP Gateway • NaradaBrokering is open source messaging from Indiana uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

16. Demo Pics ( I ) uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

17. Demo Pics ( II ) uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

18. Demo Pics ( III ) uri="http://www.globalmmcs.org" email="gcf@indiana.edu"

19. Thank you! Please watch our demo at 5:30 PM ! uri="http://www.globalmmcs.org" email="gcf@indiana.edu"