telemorph teleturas bandwidth determined mobile multimodal presentation
Download
Skip this Video
Download Presentation
TeleMorph & TeleTuras: Bandwidth determined Mobile MultiModal Presentation

Loading in 2 Seconds...

play fullscreen
1 / 18

TeleMorph & TeleTuras: Bandwidth determined Mobile MultiModal Presentation - PowerPoint PPT Presentation


  • 272 Views
  • Uploaded on

IntelliMedia . TeleMorph & TeleTuras: Bandwidth determined Mobile MultiModal Presentation. Student: Anthony J. Solon Supervisors: Prof. Paul Mc Kevitt Kevin Curran School of Computing and Intelligent Systems Faculty of Engineering University of Ulster, Magee. Aims of Research.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'TeleMorph & TeleTuras: Bandwidth determined Mobile MultiModal Presentation' - RexAlvis


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
telemorph teleturas bandwidth determined mobile multimodal presentation
IntelliMedia

TeleMorph & TeleTuras:Bandwidth determined Mobile MultiModal Presentation

Student: Anthony J. Solon

Supervisors: Prof. Paul Mc Kevitt

Kevin Curran

School of Computing and Intelligent Systems

Faculty of Engineering

University of Ulster, Magee

aims of research
Aims of Research
  • To develop an architecture, TeleMorph, that dynamically morphs between output modalities depending on available network bandwidth:
    • Mobile device’s output presentation (unimodal/multimodal) depending on available network bandwidth
      • network latency and bit error rate
      • mobile device display, available output abilities, memory, CPU
      • user modality preferences, cost incurred
      • user’s cognitive load determined by Cognitive Load Theory (CLT)
    • Utilise Causal Probabilistic Networks (CPNs) for analysing union of constraints giving optimal multimodal output presentation
    • Implement TeleTuras, a tourist information guide for city of Derry
objectives of research
Objectives of Research
  • Receive and interpret questions from user
  • Map questions to multimodal semantic representation
  • Match multimodal representation to knowledge base to retrieve answer
  • Map answers to multimodal semantic representation
  • Monitor user preference or client side choice variations
  • Query bandwidth status
  • Detect client device constraints and limitations
  • Combine affect of all constraints imposed on system using CPNs
  • Generate optimal multimodal presentation based on bandwidth constraint data
wireless telecommunications
Wireless Telecommunications
  • Generations of Mobile networks:
    • 1G - Analog voice service with no data services
    • 2G - Circuit-based, digital networks, capable of data transmission speeds averaging around 9.6K bps
    • 2.5G (GPRS) - Technology upgrades to 2G, boosting data transmission speeds to around 56K bps. Allows packet based “always on” connectivity
    • 3G (UMTS) - digital multimedia, different infrastructure required, data transmission speeds from 144K-384K-2M bps
    • 4G - IP based mobile/wireless networks, Wireless Personal Area Networks (PANs), ‘anywhere and anytime’ ubiquitous services. Speeds up to 100Mbps
  • Network-adaptive multimedia models:
    • Transcoding proxies
    • End-to-end approach
    • Combination approach
    • Mobile/Nomadic computing
    • Active networks
mobile intelligent multimedia systems
Please select a parking place from the MapMobile Intelligent MultiMedia Systems
  • SmartKom (Wahlster, 2003)
    • Mobile, Public, Home/office
    • Saarbrücken, Germany
    • Combines speech, gesture and facial expressions on input & output
    • Integrated trip planning, Internet access, communication applications, personal organising
  • VoiceLog (BBN, 2002)
    • BBN technologies in Cambridge, Massachusettes
    • Views/diagrams of military vehicles and direct connection to support
    • Damage identified & ordering of parts using diagrams
  • MUST (Almeida et al., 2002)
    • MUltimodal multilingual information Services for small mobile Terminals
    • EURESCOM, Heidelberg, Germany
    • Future multimodal and multilingual services on mobile networks
intelligent multimedia presentation
Intelligent MultiMedia Presentation
  • Flexibly generate various presentations to meet individual requirements of:

1) users, 2) situations, 3) domains

  • Intelligent MultiMedia Presentation can be divided into following processes:
    • determination of communicative intent
    • content selection
    • structuring and ordering
    • allocation to particular media
    • realisation in specific media
    • coordination across media
    • layout design
  • Key research problems:
    • Semantic Representation
    • Fusion, integration & coordination
slide7
Semantic representation - represents meaning of media information
    • Frame-based representations:

- CHAMELEON

- REA

    • XML-based representations:

- M3L (SmartKom)

- MXML (MUST)

- SMIL

- MPEG-7

  • Fusion, integration & coordination of modalities
    • Integrating different media in a consistent and coherent manner
    • Multimedia coordination leads to effective integrated multiple media in output
    • Synchronising modalities
      • Time threshold between modalities E.g. Input - “What building is this?”, Output - “This is the Millenium forum”
      • Not synchronised => side effect can be contradiction
    • SMIL modality synchronisation and timing elements
intelligent multimedia presentation systems
Intelligent MultiMedia Presentation Systems
  • Automatically generate coordinated intelligent multimedia presentations
  • User-determined presentation:
    • COMET (Feiner & McKeown, 1991)
      • COordinated Multimedia Explanation Testbed
      • Generates instructions for maintenance and repair of military radio receiver-transmitters
      • Coordinates text and 3D graphics of mechanical devices
    • WIP (Wahlster et al., 1992)
      • Intelligent multimedia authoring system
      • presents instructions for assembling/using/maintaining/repairing devices (e.g. espresso machines, lawn mowers, modems)
    • IMPROVISE (Zhou & Feiner, 1998)
      • Graphics generation system
      • constructive/parameterised graphics generation approaches
      • Uses an extensible formalism to represent a visual lexicon for graphics generation
intelligent multimedia interfaces agents
Intelligent MultiMedia Interfaces & Agents
  • Intelligent multimedia interfaces
    • Parse integrated input and generate coordinated output
    • XTRA
      • Interface to an expert system providing tax form assistance
      • Generates & interprets natural language text and pointing gestures automatically; relies on pre-stored graphics
      • Displays relevant tax form and natural language input/output panes
  • Intelligent multimedia agents
    • Embodied Conversational Agents (e.g. MS Agent, REA)
    • Natural human face-face communication - speech, facial expressions, hand gestures & body stance
    • MS Agent
      • Set of programmable services for interactive presentation
      • Speech, gesture, audio & text output; speech & haptic input
project proposal
Project Proposal
  • Research and implement mobile intelligent multimedia presentation architecture called TeleMorph
  • Dynamically generates multimedia presentation determined by bandwidth available; also other constraints:
    • Network latency, bit error rate
    • Mobile device display, available output abilities, memory, CPU
    • user modality preferences, cost incurred
    • Cognitive Load Theory (CLT)
  • Causal Probabilistic Networks (CPNs) for analysing union of constraints giving optimal multimodal output presentation
slide11
Implement TeleTuras, a tourist information guide for city of Derry providing testbed for TeleMorph incorporating:
    • route planning, maps, spoken presentations, graphics of points of interest & animations
    • Output modalities used & effectiveness of communication
  • TeleTuras examples:
    • “Where is the Millenium forum?”
    • “How do I get to the GuildHall?”
    • “What buildings are of interest in this area?”
    • “Is there a Chinese restaurant in this area?”
data flow of telemorph
Data flow of TeleMorph

High level :

Media Analysis :

software analysis
Software Analysis
  • Client output:
    • SMIL media player (InterObject)
    • Java Speech API Markup Language (JSML)
    • Autonomous agent (MSAgent)
  • Client input:
    • Java Speech API Grammar Format (JSGF)
    • J2ME graphics APIs
    • J2ME networking
  • Client device status:
    • SysInfo MIDlet - (type/memory/screen/protocols/input abilities/CPU speed)
  • TeleMorph server tools:
    • SMIL & MPEG-7
    • HUGIN (CPNs)
    • JATLite/OAA
conclusion
Conclusion
  • A Mobile Intelligent MultiModal Presentation Architecture called TeleMorph will be developed
  • Dynamically morphing between output modalities depending on available network bandwidth in conjunction with other relevant constraints
  • CPNs for analysing union of constraints giving optimal multimodal output presentation
  • TeleTuras will be used as testbed for TeleMorph
  • Corpora of questions to test TeleTuras (prospective users/tourists)