1 / 18

TeleMorph & TeleTuras: Bandwidth determined Mobile MultiModal Presentation

IntelliMedia . TeleMorph & TeleTuras: Bandwidth determined Mobile MultiModal Presentation. Student: Anthony J. Solon Supervisors: Prof. Paul Mc Kevitt Kevin Curran School of Computing and Intelligent Systems Faculty of Engineering University of Ulster, Magee. Aims of Research.

RexAlvis
Download Presentation

TeleMorph & TeleTuras: Bandwidth determined Mobile MultiModal Presentation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. IntelliMedia TeleMorph & TeleTuras:Bandwidth determined Mobile MultiModal Presentation Student: Anthony J. Solon Supervisors: Prof. Paul Mc Kevitt Kevin Curran School of Computing and Intelligent Systems Faculty of Engineering University of Ulster, Magee

  2. Aims of Research • To develop an architecture, TeleMorph, that dynamically morphs between output modalities depending on available network bandwidth: • Mobile device’s output presentation (unimodal/multimodal) depending on available network bandwidth • network latency and bit error rate • mobile device display, available output abilities, memory, CPU • user modality preferences, cost incurred • user’s cognitive load determined by Cognitive Load Theory (CLT) • Utilise Causal Probabilistic Networks (CPNs) for analysing union of constraints giving optimal multimodal output presentation • Implement TeleTuras, a tourist information guide for city of Derry

  3. Objectives of Research • Receive and interpret questions from user • Map questions to multimodal semantic representation • Match multimodal representation to knowledge base to retrieve answer • Map answers to multimodal semantic representation • Monitor user preference or client side choice variations • Query bandwidth status • Detect client device constraints and limitations • Combine affect of all constraints imposed on system using CPNs • Generate optimal multimodal presentation based on bandwidth constraint data

  4. Wireless Telecommunications • Generations of Mobile networks: • 1G - Analog voice service with no data services • 2G - Circuit-based, digital networks, capable of data transmission speeds averaging around 9.6K bps • 2.5G (GPRS) - Technology upgrades to 2G, boosting data transmission speeds to around 56K bps. Allows packet based “always on” connectivity • 3G (UMTS) - digital multimedia, different infrastructure required, data transmission speeds from 144K-384K-2M bps • 4G - IP based mobile/wireless networks, Wireless Personal Area Networks (PANs), ‘anywhere and anytime’ ubiquitous services. Speeds up to 100Mbps • Network-adaptive multimedia models: • Transcoding proxies • End-to-end approach • Combination approach • Mobile/Nomadic computing • Active networks

  5. Please select a parking place from the Map Mobile Intelligent MultiMedia Systems • SmartKom (Wahlster, 2003) • Mobile, Public, Home/office • Saarbrücken, Germany • Combines speech, gesture and facial expressions on input & output • Integrated trip planning, Internet access, communication applications, personal organising • VoiceLog (BBN, 2002) • BBN technologies in Cambridge, Massachusettes • Views/diagrams of military vehicles and direct connection to support • Damage identified & ordering of parts using diagrams • MUST (Almeida et al., 2002) • MUltimodal multilingual information Services for small mobile Terminals • EURESCOM, Heidelberg, Germany • Future multimodal and multilingual services on mobile networks

  6. Intelligent MultiMedia Presentation • Flexibly generate various presentations to meet individual requirements of: 1) users, 2) situations, 3) domains • Intelligent MultiMedia Presentation can be divided into following processes: • determination of communicative intent • content selection • structuring and ordering • allocation to particular media • realisation in specific media • coordination across media • layout design • Key research problems: • Semantic Representation • Fusion, integration & coordination

  7. Semantic representation - represents meaning of media information • Frame-based representations: - CHAMELEON - REA • XML-based representations: - M3L (SmartKom) - MXML (MUST) - SMIL - MPEG-7 • Fusion, integration & coordination of modalities • Integrating different media in a consistent and coherent manner • Multimedia coordination leads to effective integrated multiple media in output • Synchronising modalities • Time threshold between modalities E.g. Input - “What building is this?”, Output - “This is the Millenium forum” • Not synchronised => side effect can be contradiction • SMIL modality synchronisation and timing elements

  8. Intelligent MultiMedia Presentation Systems • Automatically generate coordinated intelligent multimedia presentations • User-determined presentation: • COMET (Feiner & McKeown, 1991) • COordinated Multimedia Explanation Testbed • Generates instructions for maintenance and repair of military radio receiver-transmitters • Coordinates text and 3D graphics of mechanical devices • WIP (Wahlster et al., 1992) • Intelligent multimedia authoring system • presents instructions for assembling/using/maintaining/repairing devices (e.g. espresso machines, lawn mowers, modems) • IMPROVISE (Zhou & Feiner, 1998) • Graphics generation system • constructive/parameterised graphics generation approaches • Uses an extensible formalism to represent a visual lexicon for graphics generation

  9. Intelligent MultiMedia Interfaces & Agents • Intelligent multimedia interfaces • Parse integrated input and generate coordinated output • XTRA • Interface to an expert system providing tax form assistance • Generates & interprets natural language text and pointing gestures automatically; relies on pre-stored graphics • Displays relevant tax form and natural language input/output panes • Intelligent multimedia agents • Embodied Conversational Agents (e.g. MS Agent, REA) • Natural human face-face communication - speech, facial expressions, hand gestures & body stance • MS Agent • Set of programmable services for interactive presentation • Speech, gesture, audio & text output; speech & haptic input

  10. Project Proposal • Research and implement mobile intelligent multimedia presentation architecture called TeleMorph • Dynamically generates multimedia presentation determined by bandwidth available; also other constraints: • Network latency, bit error rate • Mobile device display, available output abilities, memory, CPU • user modality preferences, cost incurred • Cognitive Load Theory (CLT) • Causal Probabilistic Networks (CPNs) for analysing union of constraints giving optimal multimodal output presentation

  11. Implement TeleTuras, a tourist information guide for city of Derry providing testbed for TeleMorph incorporating: • route planning, maps, spoken presentations, graphics of points of interest & animations • Output modalities used & effectiveness of communication • TeleTuras examples: • “Where is the Millenium forum?” • “How do I get to the GuildHall?” • “What buildings are of interest in this area?” • “Is there a Chinese restaurant in this area?”

  12. Architecture of TeleMorph

  13. Data flow of TeleMorph High level : Media Analysis :

  14. Comparison of Mobile Intelligent MultiMedia Systems

  15. Comparison of Intelligent MultiMedia Systems

  16. Software Analysis • Client output: • SMIL media player (InterObject) • Java Speech API Markup Language (JSML) • Autonomous agent (MSAgent) • Client input: • Java Speech API Grammar Format (JSGF) • J2ME graphics APIs • J2ME networking • Client device status: • SysInfo MIDlet - (type/memory/screen/protocols/input abilities/CPU speed) • TeleMorph server tools: • SMIL & MPEG-7 • HUGIN (CPNs) • JATLite/OAA

  17. Project Schedule

  18. Conclusion • A Mobile Intelligent MultiModal Presentation Architecture called TeleMorph will be developed • Dynamically morphing between output modalities depending on available network bandwidth in conjunction with other relevant constraints • CPNs for analysing union of constraints giving optimal multimodal output presentation • TeleTuras will be used as testbed for TeleMorph • Corpora of questions to test TeleTuras (prospective users/tourists)

More Related