1 / 35

Final Year Project 2003/2004 LYU0302 PVCAIS – Personal Video Conference Archives Indexing System

PVCAIS is a system that provides convenient searching and browsing support for videoconferencing users on past videoconference archives. It extracts and indexes media files, including audio, video, text chat, whiteboard, and shared applications.

geraldg
Download Presentation

Final Year Project 2003/2004 LYU0302 PVCAIS – Personal Video Conference Archives Indexing System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Final Year Project 2003/2004LYU0302PVCAIS – Personal Video Conference Archives Indexing System Supervisor: Prof Michael Lyu Presented by: Lewis Ng, Philip Chan 2 December 2003

  2. Outline • Introduction • Motivation • Architecture of PVCAIS - Media Acquisition Module - Archive Indexing Module - Videoconference Accessing Module • Implementation in First Term • Conclusions • Future Work

  3. Introduction • PVCAIS stands for Personal Video Conference Archives Indexing System • A system that provides the convenient searching and browsing support for videoconferencing users on past videoconference archives

  4. Introduction • What is video conference? A real-time communication technology which combines different media: audio, video, text chat, file transfer, whiteboard and shared applications - More precisely is “multimedia conference” - Standard of video conference : ITU-T H.323

  5. Motivation • Videoconference is becoming popular in education, business and personal communication • Participants wish to keep videoconference archives for later references • Normal video and audio files are neither searchable nor helpful to recall their contents • Indexing of videoconference archives has not been investigated till now

  6. Architecture of PVCAIS • Consists of 3 modules: - Media Acquisition Module - Archive Indexing Module - Videoconference Accessing Module

  7. Architecture of PVCAIS Archive Indexing Media Acquisition Videoconference Accessing

  8. Architecture Media Acquisition Module • Extracts channel data and forms media files • Videoconferencing physically contains 4 types of channels: Audio, Video, Data and Control • Audio and Video channels: transmit incoming/ outgoing audio and video information • Data channel: carries information for user application such as Text Chat, Whiteboard and File Transfer • Control channel: transmits system control information such as Member Information

  9. Architecture Media Acquisition Module • Video-in and Video-out channel • Reduce redundancy : just store key-frames • Detect scene change in real time • Each key frame picture is stored with a timestamp

  10. Architecture Media Acquisition Module • Audio-in and Audio-out channel • Mixed into one stream after videoconference • Will be used for Speech Recognition

  11. Architecture Media Acquisition Module • Text Chat channel • sender / receiver • message • store with timestamp

  12. Architecture Media Acquisition Module • Whiteboard channel • Consists of a text-based index file and a number of snapshot pictures • Index file records timestamp for each whiteboard update event and the path of the corresponding snapshot picture • Update of this channel happens in a period of time -> need to detect when update begins and ends by monitoring data transfer in this channel

  13. Architecture Media Acquisition Module • File Transfer channel • Make a copy of the sent/received files to the directory of archive • Index file includes sender’s / recipient’s user names and the path of the files

  14. Architecture Media Acquisition Module • Control channel • Contains timestamp and information of each event such as member joined and member left

  15. Video_in Video_out Audio_in Audio_out One line One line Two lines One line Two lines Three lines One line Two lines Three lines Four lines One line Two lines Three lines Four lines Five lines Text_chat Whiteboard File_in File_ out i i i i Control Video_in archive Video_out archive Audio archive Text chat archive Whiteboard archive Document archive Time 0:00:00 Control archive Paradigm of storing the videoconference archives. Architecture Media Acquisition Module

  16. Architecture Archive Indexing Module • 7 raw files are extracted in Media Acquisition Module • Need to implement some multimedia indexing functions to retrieve more information • These includes: Face Detection, Face Recognition, Speech Recognition, OCR, Time-based Text Merging, Keyword Selection, Title Generation

  17. Architecture Archive Indexing Module • Face Detection - Distinguish between Slides and Faces - If face is detected, find out the face region Face Detection Slide Face Detection Face

  18. Architecture Archive Indexing Module • Face Recognition - Associate human faces in Video-in with name - Need to keep a face base - If no match in the face base, ask remote user to enter the name

  19. Architecture Archive Indexing Module • Speech Recognition -Generate speech script from audio archive-Speech of a videoconferencing contains the most information - Can use commercial library: Microsoft SAPI, IBM Via Voice • OCR - Take the slide archive as input and recognizes text from them - Need to identify and localize text on the complex background

  20. Architecture Archive Indexing Module • Time-based Text Merging - Merge the Speech transcript, Chat script, Whiteboard script and slide text archive into the Text Source according to their timestamp • Keyword Selection -Take the Text Source as input - Generate keyword for the videoconference

  21. Architecture Archive Indexing Module • Title Generation -Take the Text source as input - Automatically generate a title for the videoconference • Generate XML index file - Integrate all the archives - Store all the related files of a videoconference into a single directory

  22. Architecture Videoconference Accessing Module • Provides an interface for user to manage, search and review all indexed conference archives. • Allows user to modify the content of a conference, such as editing title or keywords, or delete a conference. • Allows user to search for a conference by different criteria, such as period of meeting, member name or keyword. • Allows user to review a conference by playing back different media in a synchronized way.

  23. Implementation • NetMeeting 3.0 • A Windows feature that provide Internet conferencing function. • Support video, audio and data conferencing including application sharing, chat, whiteboard and file transfer. • Other features include remote desktop sharing.

  24. Implemetation • NetMeeting 3.0 SDK • An extension of NetMeeting, provides an interface for programmers and Web developers to integrate conferencing capabilities into their applications. • API is in the form of COM interfaces and functions.

  25. Implementation • A simple NetMeeting compatible videoconference program built on top of the NetMeeting 3.0 SDK. • Support: • Video • Audio • Text message • File Transfer • Whiteboard

  26. Implementation Media Acquisition Module • By directly using the functions of the API, the following raw data can be obtained: • the members information • file transfer record • text messages record • Video, audio and whiteboard data cannot be directly obtained.

  27. Implementation Media Acquisition Module • Video • create a thread to check the display of the video windows • if scene change is detected, the video will be captured and stored as a still image. • the stored images are key frames of the conference.

  28. Implementation Media Acquisition Module • Audio • create a thread to record the local audio from the microphone. • members of the conference will continuously exchange the audio data. • all the received audio files and locally recorded audio files will be combined to generate a single audio file.

  29. Implementation Media Acquisition Module • Whiteboard • cannot capture the NetMeeting whiteboard information because the format of the data is not stated in the API. • solution: create our own whiteboard function and data format.

  30. Implementation Archive Indexing Module • The stored key-frames will be used for face detection and recognition after the conference. • The final audio file will be used for voice recognition, the voice engine used is Microsoft SAPI.

  31. ImplementationVideoconference Accessing Module • An interface for conferences management. --- search conference by member name or chatting content. --- review conference by playing back the content of the conference, including audio, key-frames, member information, file exchange record and chatting content.

  32. Conclusions • We developed a videoconferencing agent • All channel data except whiteboard can be collected. • Speech Recognition and Face Detection & Recognition is integrated into the system but accuracy needs to be improved • Simple searching can be performed on stored archives

  33. Future Work • Whiteboard • Improve accuracy of Voice Recognition • Improve performance of Face Detection • XML • Better searching method • OCR for slide in video • Improve User Interface

  34. Q & A Session

More Related