1 / 56

Imagineering Creative Technologies

Imagineering Creative Technologies. Professor Paul Mc Kevitt Intelligent Systems Research Centre School of Computing & Intelligent Systems Faculty of Computing & Engineering University of Ulster, Magee Derry/Londonderry Northern Ireland. Background.

isleen
Download Presentation

Imagineering Creative Technologies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Imagineering Creative Technologies Professor Paul Mc Kevitt Intelligent Systems Research Centre School of Computing & Intelligent Systems Faculty of Computing & Engineering University of Ulster, Magee Derry/Londonderry Northern Ireland

  2. Background • Intelligent MultiMedia & MultiModal Systems • Integration of natural language, speech/audio, graphics/VR, vision • Semantic representation, fusion & synchronisation • Affective computing • Integration & reuse of existing software (Thórisson, 2007) • Relations with cognitive science • Deployment on mobile devices (Dell Axim, Smartphones, Nintendo DS) • Applications in arts: music, drama, dance, design, computer games, film, TV/radio, art, storytelling

  3. CHAMELEON:a platform for developing intelligent multimedia applications Tom Brøndsted, Paul Dalsgaard, Lars Bo Larsen, Mike Manthey, Paul Mc Kevitt (CPK, Aalborg University, Denmark) • Speech input/output, image processing & laser • IntelliMedia Workbench campus information system application • Semantic representation with frames (Minsky, 1975) • Blackboard for semantic representation • Bielefeld DACS distributed communications platform (Fink et al., 1996) • Implemented with C, C++, DACS, Infovox TTS, grapHvite • Applications in multimodal systems development

  4. Architecture of CHAMELEON

  5. IntelliMedia Workbench

  6. CHAMELEON Semantic Representation • Semantic Frames (Minsky, 1975)

  7. CONFUCIUS:automatic conversion of natural language to 3D animation Minhua (Eunice) Ma, Paul Mc Kevitt • Lexical Visual Semantic Representation (LVSR)Use of Jackendoff’s (1990) Lexical Conceptual Structure (LCS) & Schank’s (1977) scripts • Use of Fellbaum’s (1998) semantic classes in WordNet • Implementation with Java, VRML, WordNet, Connexor parser, FreeTTS (Java) • Applications in education, computer games, animation production, digital storytelling

  8. Architecture of CONFUCIUS

  9. MultiModal semantic representation

  10. CONFUCIUS examples • Contact verb example • (e.g. hit, collide, scratch, touch) “The car collided with a wall.” • Collision detection • Nonspeech sound effects • No human role involved • Human action verb examples • “Ipke gave Brigitte a loaf of bread.” • “Bernie put a cup of coffee on the table.” • H-Anim Site node • locative tags of object (on tag for table object) • “Paul pushed the door.” • “Gert ate the bread.”

  11. “Paul sat on the chair.” “Teri was waving goodbye.” “Helge went to the gym.” “Gert put on his hat.” States “Anke is sad.” “Anke is ashamed.”

  12. SceneMaker:automatic visualisation of screenplays Eva Hanser, Joan Condell, Tom Lunney, Paul Mc Kevitt • Automatically generates affective virtual scenes from screenplays/play scripts • Realistic visualisation of emotional aspects • Extraction of genre/moods from screenplays • Use of OCC (Ortony-Clore-Collins) affective model, ConceptNet, Open Mind Common Sense (OMCS), WordNet-Affect • Enhance believability of virtual actors and scene presentation • Multimodal representation with 3D animation, speech, audio & cinematography • Applications in assisting directors, actors, drama students, scriptwriters

  13. Architecture of SceneMaker

  14. MemoryLane:a mobile digital storytelling companion Sheila Mc Carthy, Heather Sayers, Mike McTear, Paul Mc Kevitt • Multimodal stories (speech, text, video) from life-cached memory data (poems, songs, photos, home videos) • Extensive HCI requirements analysis with focus on older people • Multimodal preferences and abilities of users determines story modalities and user interface configuration • Prudent on sensitive life-cached data • Creates new stories on the fly • Applications in medical informatics and digital care • Implemented with Microsoft Visual Studio • Deployed on Dell Axim X51v PDA

  15. Architecture of MemoryLane

  16. MemoryLane Prototype

  17. AmbiLearn:an ambient intelligent multimodal learning environment for children Jennifer Hyndman, Tom Lunney, Paul Mc Kevitt • Educational games & virtual learning environments • Child - Computer Interaction (ChiCI) • AmbiLearn architecture & TreasureLearn application • Implementation with DevKit Pro (Nintendo homebrew) • Deployment on mobile device (Nintendo DS) • Application to collaborative learning for children

  18. Architecture of AmbiLearn

  19. PlayPhysics:an emotional games learning environment for teaching Physics Karla Munoz Esquivel, Tom Lunney, Paul Mc Kevitt • Virtual Learning Environment (VLE) for teaching Physics • Incorporating serious games & intelligent tutoring systems (ITS) • Modelling of emotions & personality of student • Use of Bayesian Networks for modelling student knowledge • Affective feedback through audio-visual effects • On-line web application with Student, Lecturer, Head of Department, Administrator views • Implemented with Elvira Bayesian Network tool, Flash, Java, Tomcat Apache Server, MySQL database, ConceptNet, 3D Studio Max & Maya • Applications in educational software & computer games

  20. Olympia architecture

  21. PlayPhysics interface (astronaut game)

  22. TeleMorph & TeleTuras:bandwidth determined mobile multimodal presentation Tony Solon, Kevin Curran, Paul Mc Kevitt • Multimedia limitations of mobile networks, mobile device capabilities • User preferences on modalities • Transmoding between modalities with cross-modality adaptation • Use of Fuzzy (logic) Inference system (FIS) • TeleMorph platform & TeleTuras tourist application • Implemented with MATLAB • Applications in streaming media, computer networks, multimodal presentation

  23. Architecture of TeleMorph Fuzzy Inference System (FIS)

  24. TeleTuras GUI

  25. Available media qualities in 6 sample test scenarios

  26. Adaptation trace data from test scenario 6

  27. MediaHub:Bayesian decision-making in an intelligent multimedia distributed platform hub Glenn Campbell, Tom Lunney, Aiden McCaughey, Paul Mc Kevitt • Interpret/generate semantic representations of multimodal input/output • Fusion & synchronisation of multimodal data • Decision-making with Bayesian Causal Probabilistic Networks (CPNs) • Implemented with Psyclone platform, HuginLite Bayesian modelling tool, Java, NetBeans IDE • Applications in decision-making for key problems in multimodal systems: anaphora resolution, domain knowledge awareness, multimodal presentation, turn-taking, dialogue-act recognition & parametric learning

  28. Architecture of MediaHub

  29. Decision-making layers

  30. psyProbe view of MediaHub Whiteboard

  31. Alternative `Turn-taking' Bayesian network in Hugin

  32. SoFI:SOng Form Intelligence for repairing streaming musicacross wireless bursty networks Jonathan Doherty, Kevin Curran, Paul Mc Kevitt • Large packet loss dropouts of 5 - 15 s. • Use audio pattern recognition & matching to repair gaps • K-means clustering classifier determines self-similarity matches • Match current sections of song received with previous sections played • Typicality of Western Tonal Format (WTF) • Use of MPEG-7 Audio Spectrum Envelope (ASE) multimedia content description standard giving features for extraction • Implemented with G-streamer for steaming media, Ices2, Icecast2 for audio broadcast & MPEG-7 Audio Spectrum Envelope (ASE) • Applications in repair of streaming music

  33. Architecture of SoFI

  34. Spectrogram representation

  35. Steganoflage:strengthening steganography in digital images Abbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt • Hiding/embedding data in transmission medium • Use of 1D hash algorIthm coupled with 2D iFFT (irreversible Fast Fourier Transform) • Encryption of digital documents in 2D spatial domain • Use of steganography to add security layer resistant to noise & JPEG compression • Exploitation of Jarvis kernel (Jarvis et al., 1976) to generate error diffusion signal • Use of Wavelet-based Inverse Halftoning via De-convolution (WInHD) to recover approximate original signal • Implemented with MATLAB & PHP • Applications in security, forensics, forgery, digital watermarking

  36. Architecture of Steganoflage

  37. Securing confidential data Application The traditional method of securing individuals’ data with the notion of security by obscurity that Steganography introduces

  38. National & international collaboration • InstitutoTecnológico de Estudios Superiores de Monterrey, Campus Ciudad de México (ITESM-CCM), Mexico • Aalborg University, Denmark • CADIA, Reykjavik University, Iceland • University of Bielefeld, Germany • LIMSI (CNRS), Orsay, Paris, France • Sheffield University, England • Stanford University, California, USA

  39. Noel Sharkey thinks AI is an illusion

More Related