1 / 69

Agenda

Multimodal Interaction in Smart Environments A Model-based Runtime System for Ubiquitous User Interfaces. Dipl.-Inf. Marco Blumendorf Fakultät IV – Elektrotechnik und Informatik Technische Universität Berlin. Agenda. Motivation Current Trends Fundamentals User Interface Models Approach

akasma
Download Presentation

Agenda

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multimodal Interaction in Smart EnvironmentsA Model-based Runtime System for Ubiquitous User Interfaces Dipl.-Inf. Marco BlumendorfFakultät IV – Elektrotechnik und InformatikTechnische Universität Berlin

  2. Agenda • Motivation • Current Trends • Fundamentals • User Interface Models • Approach • Runtime Concepts • UI Description Language Extensions • Architecture • Implementation • Does it work? • Conclusion • Some issues remain A Model-based Runtime System for Ubiquitous User Interfaces

  3. “Recent” Trends World Wide Web & Ubiquitous Computing A Model-based Runtime System for Ubiquitous User Interfaces

  4. World Wide Web A Model-based Runtime System for Ubiquitous User Interfaces

  5. World Wide Web - Basics • Websites hosted on servers • Accessible from everywhere • HTML language • Browser as HTML interpreters A Model-based Runtime System for Ubiquitous User Interfaces

  6. Web Page Development in 4 Steps • Write HTML code • Add some CSS (style) • Add some JavaScript • Deploy it on a webserver Model, interpreted by the browser A Model-based Runtime System for Ubiquitous User Interfaces

  7. Ubiquitous Computing A Model-based Runtime System for Ubiquitous User Interfaces

  8. Computers are everywhere! A Model-based Runtime System for Ubiquitous User Interfaces

  9. “Ubiquitous Computing” (Mark Weiser, 1991) “In the 21st century the technology revolution will move into the everyday, the small and the invisible.” A Model-based Runtime System for Ubiquitous User Interfaces

  10. Smart Objects Smart Devices Smart Networks Smart Environments A Model-based Runtime System for Ubiquitous User Interfaces

  11. Interaction happens everywhere! A Model-based Runtime System for Ubiquitous User Interfaces

  12. Ubiquitous User Interfaces (UUIs) A Model-based Runtime System for Ubiquitous User Interfaces

  13. Utilizing Ubiquitous User Interfaces in Smart Environments used interaction modalities A Model-based Runtime System for Ubiquitous User Interfaces

  14. Compared to a web page … How would you develop a Ubiquitous User Interface? A Model-based Runtime System for Ubiquitous User Interfaces

  15. Language and Infrastructure for Ubiquitous User Interfaces A Model-based Runtime System for Ubiquitous User Interfaces

  16. Agenda • Motivation • We need a language and infrastructure for UUIs • Fundamentals • User Interface Models • Approach • Runtime Concepts • UI Description Language Extensions • Architecture • Implementation • Does it work? • Conclusion • Some issues remain. A Model-based Runtime System for Ubiquitous User Interfaces

  17. User Interface Models A Model-based Runtime System for Ubiquitous User Interfaces

  18. HTML as a Model User Interface HTML HTML A Model-based Runtime System for Ubiquitous User Interfaces

  19. Models to Maximize Common Code User Interface Model A Model-based Runtime System for Ubiquitous User Interfaces

  20. UIDLs use Multiple Levels of Abstraction Concepts and Task Model Abstract Interface Concrete Interface Final User Interface See also Szekely 1996, Cameleon Reference Framework (Calvary et al. 2003) Similar concepts can be found e.g. in UsiXML (Limbourg et al. 2004) and TERESA (Berti et al. 2004) A Model-based Runtime System for Ubiquitous User Interfaces

  21. UIDLs use Multiple Levels of Abstraction Concepts and Task Model Abstract Interface Concrete Interface Final User Interface See also Szekely 1996, Cameleon Reference Framework (Calvary et al. 2003) Similar concepts can be found e.g. in UsiXML (Limbourg et al. 2004) and TERESA (Berti et al. 2004) A Model-based Runtime System for Ubiquitous User Interfaces

  22. What if you want to change anything at runtime? A Model-based Runtime System for Ubiquitous User Interfaces

  23. You need the models! A Model-based Runtime System for Ubiquitous User Interfaces

  24. Problems • Current approaches have a design time focus • At runtime, knowledge is hidden within code • Design intentions are not explicit anymore • Design decisions can not be revised • This makes UIs static and limits adaptivity! => No Ubiquitous User Interfaces A Model-based Runtime System for Ubiquitous User Interfaces

  25. Keep the UI models at runtime! A Model-based Runtime System for Ubiquitous User Interfaces

  26. Model Interpretation A Model-based Runtime System for Ubiquitous User Interfaces

  27. Model Interpretation A Model-based Runtime System for Ubiquitous User Interfaces

  28. Model Interpretation A Model-based Runtime System for Ubiquitous User Interfaces

  29. Model Interpretation A Model-based Runtime System for Ubiquitous User Interfaces

  30. Agenda • Motivation • We need a language and infrastructure for UUIs • Fundamentals • We need models interpretation at runtime • Approach • Runtime Concepts • UI Description Language Extensions • Architecture • Implementation • Does it work? • Conclusion • Some issues remain. A Model-based Runtime System for Ubiquitous User Interfaces

  31. Framework Building Blocks • Executable Models • UI Description Language • Server Architecture A Model-based Runtime System for Ubiquitous User Interfaces

  32. How to make Models Executable? • Preserve design information at runtime • Add state information • Add execution logic A Model-based Runtime System for Ubiquitous User Interfaces

  33. Extending Current Design Models Dynamic Executable Model See e.g. Breton & Bézivin 2001 for details A Model-based Runtime System for Ubiquitous User Interfaces

  34. A Task Tree Example Abstract Task Interaction Task >> [] Temp. Operators A Model-based Runtime System for Ubiquitous User Interfaces

  35. Metamodel of Executable Task Models A Model-based Runtime System for Ubiquitous User Interfaces

  36. Metamodel of Executable Task Models A Model-based Runtime System for Ubiquitous User Interfaces

  37. Metamodel of Executable Task Models A Model-based Runtime System for Ubiquitous User Interfaces

  38. Metamodel of Executable Task Models A Model-based Runtime System for Ubiquitous User Interfaces

  39. Framework Building Blocks • Executable Models • UI Description Language • Server Architecture A Model-based Runtime System for Ubiquitous User Interfaces

  40. How to build a language for such models? • Extend existing models with runtime concepts • Add state & execution logic • Synchronization of models through mappings • Interaction with external entities • Additional concepts for Ubiquitous User Interfaces • Multimodal and distributed interaction • Context-sensitivity • Adaptation A Model-based Runtime System for Ubiquitous User Interfaces

  41. Mediating between human and computer 10011010101111100100 “turn the light on” A Model-based Runtime System for Ubiquitous User Interfaces

  42. Mediating between human and computer ? 10011010101111100100 “turn the light on” A Model-based Runtime System for Ubiquitous User Interfaces

  43. Mediating between human and computer ? 10011010101111100100 “turn the light on” A Model-based Runtime System for Ubiquitous User Interfaces

  44. Mediating between human and computer 10011010101111100100 “turn the light on” A Model-based Runtime System for Ubiquitous User Interfaces

  45. Net of UI Models A Model-based Runtime System for Ubiquitous User Interfaces

  46. Mediation Process A Model-based Runtime System for Ubiquitous User Interfaces

  47. Adding State and Execution Logic A Model-based Runtime System for Ubiquitous User Interfaces

  48. Synchronizing Models at Runtime Task Model Service Model Mapping Link Target Source A Model-based Runtime System for Ubiquitous User Interfaces

  49. A UUI is defined by a set of linked & stateful executable models. Ubiquitous User Interface A Model-based Runtime System for Ubiquitous User Interfaces

  50. Framework Building Blocks • Executable Models • UI Description Language • Server Architecture A Model-based Runtime System for Ubiquitous User Interfaces

More Related