Post Desktop User Interfaces

1. Post Desktop User Interfaces Multimedia Architectures Ines Färber and Alexander G.M. Hoffmann Advisor: Thorsten Karrer

2. Motivation • multimedia applications are of growing importance • have to satisfy hard real-time and interaction requests •  good architectures are required

3. Introduction • “multimedia architecture” is not clearly defined • multimedia is an integration of multiple digital media into one big media • at least one combined media has to be time-dependent

4. Introduction (2) • typical applications: video games, video conferencing systems, speech recognition, animation, internet • dealing with creation, manipulation, presentation, storage or communication • high requirements concerning real-time, low latency and precise synchronisation • all influences have to be well-regulated

5. Outline • SAI- unifying approach to the distributed implementation of algorithms and their easy integration into complex systems • Fran - Architecture for functional reactive animation generation • Multimedia Information Service Enabling • Conclusion

6. SAI - Software Architecture for Immersipresence SAI is a new software architecture model for designing, analyzing and implementing applications performing distributed, asynchronous parallel processing of generic data streams. (Francois 2005) • general formalism • high bandwidth • low latency A. R.J. Francois. Software Architecture for Immersipresence, IMSC 2005

7. SAI (2) • problems in actual complex systems • integration • resource intensive activity • countless unforeseen problems • SAI • distributed implementation • easy integration

8. SAI (3) • dataflow architectures • simple and intuitive design • support parallel and distributed processing • problems: • limitation of efficiency and modelling power • shared data excess • blackboards • but: not adapted to online and real-time processing

9. SAI – Fundamental principles • explicit account of time both in data and processing models • Cells • Sources • distinction between persistent and volatile data • asynchronous parallelism

10. SAI – graphical approach • data encapsulated in pulses • cells=processing centres • sources=persistent information repository • persistent data (constant) • volatile data (changing) • VisualSAI

11. Architecture • conceptual level • set of sources and cells • inter-connections • descriptions of tasks • logical level • cell: active and passive filters • source: structure of passive pulse

12. MuSA.RT – Music on the Spiral Array. Real-Time (ACM ‘03) These computations are defined by four independent streams: • MIDI input and event processing • tonal analysis • rendering of the Spiral Array structures • control device (game pad) input and camera manipulation

13. Screenshot • The active triad is indicated by a coloured triangle • red = major • blue = minor • grey dots = actual key(s) • green line = main melody

14. Video http://www-rcf.usc.edu/~mucoaco/

15. Outline • SAI- unifying approach to the distributed implementation of algorithms and their easy integration into complex systems • Fran - Architecture for functional reactive animation generation • Multimedia Information Service Enabling • Conclusion

16. Problems of Creating Animations • missing differentiation between presentation and modelling • animation is conceptually continuous and parallel, whereas a PC works discretely and sequentially  visual authoring tool  change of language  Fran Conal Elliott and Paul Hudak. Functional reactive animation IFCP’97

17. Fran (Functional Reactive Animation) • implemented in Haskell

18. Pascal vs. Haskell • procedure quicksort(l,r : integer); var x,i,j,tmp : integer; begin if r>l then begin x:=a[l]; i:=l; j:=r+1; repeat repeat i:=i+1 until a[i]>=x; repeat j:=j-1 until a[j]<=x; tmp:=a[j]; a[j]:=a[i]; a[i]:=tmp; until j<=i; a[i]:=a[j]; a[j]:=a[l]; a[l]:=tmp; quicksort(l,j-1); quicksort(j+1,r) end end. • quicksort [] = [] quicksort [x:xs] = quicksort [n | n<-xs, n<x] ++ [x] ++ quicksort [n | n<-xs, n>=x]

19. Fran (Functional Reactive Animation) • implemented in Haskell • afford a declarative instead of an imperative kind of programming • modelling-oriented • high level vocabulary of data types, functions and primitive graphic routines

20. Fran: key concepts • behaviours: time-varying, reactive values • events: sets of arbitrarily complex conditions, carrying possibly rich information • modelling approach is established by four features: • temporal modelling • event modelling • declarative modelling • polymorphic media • implicit treatment of time • method for event detection based on interval analysis

21. Fran – examples behaviours upDownPat = moveXY 0 waggle pat pat = importBitmap "../Media/pat.bmp" leftRightCharlotte = moveXY wiggle 0 charlotte charlotte = importBitmap "../Media/charlotte.bmp" Conal Elliott and Paul Hudak.1997 Modeling Interactive 3D and Multimedia Animation with an Embedded Language

22. Fran – examples behaviours charlottePatDance = leftRightCharlotte `over` upDownPat

23. Fran – example event redBlue u = buttonMonitor u `over` withColor c circle where c = red `untilB` lbp u -=> blue Fran – example 3D teapot = stretch3 2 (importX "../Media/tpot2.x") redSpinningPot = turn3 zVector3 time ( withColorG red teapot)

24. Outline • SAI- unifying approach to the distributed implementation of algorithms and their easy integration into complex systems • Fran - Architecture for functional reactive animation generation • Multimedia Information Service Enabling • Conclusion

25. Multimedia Information Services Enabling • 3-level Multimedia-Search Architecture • extendable and scaleable multimedia information management systems • extendable: easily adjusted to new data types • scaleable: no performance lose while growing workload Erik Boertjes, W. Jonker, and J. Wijnands. Multimedia information services enabling: An architectural approach. ACM 2001

26. Metadata Module User Profiles and Personalization Module Communication and Data Transfer User Interfaces and Presentation Module Raw Data Servers Modules

27. Three – layer model

28. Peggy • Internet movie search • retrieval service • movie trailers are stored (rawdata) • MPEG-7 format (metadata)

29. 0900 - SPOT “Turn left, the empty spot Is between the black car and the red car” Spot – a – spot • camera find free parking slot • Feature - and Conceptual database • PDOM search engine • generates voice messages • call-service

30. Outline • SAI- unifying approach to the distributed implementation of algorithms and their easy integration into complex systems • Fran - Architecture for functional reactive animation generation • Multimedia Information Service Enabling • Conclusion

31. Conclusion - SAI • universal framework • distributed asynchronous algorithms • integration into complex systems • graphical interface VisualSAI • components, connectors and constraints

32. Conclusion Fran • embedded language • simplifies the animation programming process via declarative programming techniques • presentation details are left to the underlying implementation • possible improvements: more features for 2D, 3D and sound, improving performance • the underlying ideas of Fran also formed the basis of Microsoft’s DirectAnimation

33. Conclusion MISE • 5 modules • 3 layer model with new concept of raw- and metadata databases • universal search • extendable and scaleable • modify and adjustable

34. Conclusion Multimedia Architectures • no universal architecture • various approaches • different approaches • SAI is a graphical approach • FRAN is animation oriented • MISE handles Interactive Multimedia retrieval requests • incompatibility if we exchange media

35. Thanks for your attention Multimedia Architectures Ines Färber and Alexander G.M. Hoffmann Advisor: Thorsten Karrer