1 / 79

e-Science Collides with the Performing Arts

e-Science Collides with the Performing Arts. Dr R P Fletcher University of York. So what is this e-Science??. At the inception of the UK e-Science programme John Taylor said that:

penda
Download Presentation

e-Science Collides with the Performing Arts

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. e-Science Collides with thePerforming Arts Dr R P Fletcher University of York

  2. So what is this e-Science?? • At the inception of the UK e-Science programme John Taylor said that: • “e-Science is about global collaboration in key areas of science and the next generation of infrastructure that will enable it” • Another source says it is: • “science increasingly done through distributed global collaborations enabled by the Internet, using very large data collections, terascale computing resources and high performance visualisation “ • Malcolm Atkinson has said that: • “e-Science is the development and exploitation of advanced informatics methods to support collaborative research in all disciplines “

  3. OVERVIEW • Academic Background – am I a real scientist? • Involvement in Performing Arts – am I a fraud? • Involvement with technology – am I a technologist? • Technology • Bespoke hardware and software • Computers • Networks • Software • Engaging with this technology • Right people in the right place • Future – some ideas • Questions – whatever you want to throw at me!

  4. My Academic background • 1968 Degree in Biology • General • 1971 D.Phil in Neurophysiology • Information processing in single sensory nerve cells • 1975 Research Fellow in Psychology • Risk of schizophrenia • Psychophysiology • 1981 Computing Service • Graphics • Database systems • Programming • Mathematical and statistical systems and libraries

  5. Academic Background • Involvement in Performing Arts • My involvement with technology • Technology • Bespoke hardware • Computers • Networks • Software • Engaging with this technology • Right people in the right place • Future • Questions

  6. My “Performing Arts” background • 1965 Staged school bands • 1968 Staged University bands • 1971 President of the Electronic Arts Society • 1972 Recording engineer York Musicians Cooperative • 1972 Worked with Trevor Wishart • 1973 Recording/mixing engineer campus band • 1974 Built own synthesiser system • 1976 Programmed FX on PDP-11 • 1978 Recording/mixing engineer campus band (again) • 1991 SGI Indigos bootstrap Music Technology • 1996 Chaotic Constructions • 1999 RIMM • 2005 UKLight for Collaborative Composition/Performance • 2006 PEACHES committee of TERENA • 2007 Content Creation tools for Digital Cinema?

  7. Technology Input and Output • Specialised software to edit synthetic speech • User interfaces for Music Technology • Courses in graphics for Music Technology • External Examiner MA Digital Arts (Middlesex) • Midi I/O systems for Sun Workstations • Developer for GIMP (video plugins for Unix) • Developer for XMX (collaborative X-Windows) • Developer for .ogg audio formats • Still ambisonic developments • Tracking technology for Chaotic Constructions • Graphics system designer/developer for RIMM • Real-time Interactive Multiple Media • PEACHES committee at TERENA • Panel of Experts for Arts, Culture and Humanities E-Science

  8. Academic Background • Involvement in Performing Arts • My involvement with technology • Technology • Bespoke hardware • Computers • Networks • Software • Engaging with this technology • Right people in the right place • Future • Questions

  9. Technologies • Aim to lead you through some technologies • Ones I have worked on over the last 15 years (or so!) • At some point technology overlaps e-Science • Need to engage with the new technologies • Need to help drive their development • Performing Arts need new/different facilities • either from the existing infgrastructures • or via new facilities • But, unless we know what’s possible • How do we know what to ask for?

  10. YORKTALK • Worked with the Language and Linguistics staff • Their technology allowed about six synthesis runs • PER DAY! • Data transferred via Kermit file transfer program! • Real informed trial and error to fix problems • I created a graphical interface to the data • Ran on a then high performance system • Could interact with the data • Synthesis was up to near real time • Combined visualisation to “see” errors • Used interactive graphics to “fix” the problems

  11. YORK TALK • Research vehicle for speech synthesis • DecTalk example (may recognise the voice) • YorkTalk examples • Same words as DecTalk • A sentence (with a cold!)

  12. Engaging with “artists • Next project was with staff in Music • Needed help to create interfaces for a system • Needed some lateral thinking • Definitely moving into the Arts domain • High performance computing was needed • Multiple tasks by same computer • Video • Audio • Graphics • Midi

  13. Chaotic Constructions - 1994 • Certain amount of serendipity • Peter Fluck talking on Radio 4 about Chaotic motion in kinetic scuplture • Tony Myatt talked a few weeks later about Chaos in Music • Phoned each other • classic “WE SHOULD TALK” moment • By 1996 developed a project • to use the chaotic motion of a kinetic sculpture to drive a music composition system • PROBLEM – how to capture the motion • ANSWER – ask me

  14. Chaotic Constructions – the Tracking System • Request came to me • How can we do this? • and … • It will be installed in Tate Modern in March 1997 • and … • It was now January 1997! • First thoughts were to use shape tracking systems • Problems: • most systems were proprietary and expensive • shapes to be tracked changed because sculpture was 3 dimensional • could not use trajectory prediction due to the chaotic motion • ANSWER – simply track unique coloured items

  15. VIDEO EXAMPLE The Making of CHAOTIC CONSTRUCTIONS

  16. Stage 1 – Define a Colour “Fingerprint” • Program creates a set of “unique fingerprints”

  17. Tuning the “Fingerprints” • Check motion does not lose sync with colour

  18. Peter Fluck Hand Cranking the Sculpture

  19. Running with Tracking and Music Program • “Fingerprint” used by second program • Motion tracking is just one element

  20. Changing a Tracking Point • What are the best points to track? • Due to chaotic motion some trial and error!

  21. And finally …the prototype • … we get there … a small excerpt

  22. And the real thing

  23. Credits and Awards • Exhibited to great acclaim in Tate Modern • Other UK Art Galleries • Travelled to Hong Kong and Corsica • Herald Angel Award by the Glasgow Herald for the best piece of visual art at the 1997 Edinburgh Festival • International Critics Award as part of the 1997 Demarco European Art Foundation exhibition

  24. Not really “Performing Arts” I hear you say • In St. Ives staff gave children coloured “bats” • Switched off the sculpture • Children waved the bats around • And, of course, “made music” • Security was called once to “remove” a visitor • Would not leave at closing time • Transfixed by the whole thing • Some people came 2 or 3 times a week • For a “fix”

  25. How can this Technology be used … • … for Performing Arts • Mount a camera above a stage • Give actors coloured hats • Now you have their position on stage • Link their movement around the stage • To … • Orchestrate lighting changes • Hotspots • Move to one place on stage and make a sound • Detect “collisions” • Allow actors to fully interact with their environment

  26. …or … • Point the camera at a tropical fish tank • Let the fish make music

  27. Bringing e-Science into the Frame • So far York work done on a single modest computer • Scale up • To multiple specialist computers • Networks • Wireless devices • Complex tracking • 3D sound • Interactive graphics • All controlled by … • The PERFORMER

  28. Real Time Interactive Multiple Media - RIMM • “Construction 3” • Realised by the Real-time Interactive Multiple Media Project • University of York Music Department (UK), IRCAM (France) and SIMPK (Germany) • European Commission Information Society Technologies Programme • Incorporates a live performer on soprano saxophone • Three-dimensional sculpture • Computer generated graphics • Ambisonic surround sound

  29. construction 3 is a synergetic composition • The musical and graphical elements of the piece have been composed concurrently • Both the computer graphics and music are triggered by the saxophonists natural gestures • Through infra-red video tracking • Use i/r device and i/r filter on camera • Electronic sensors • Footswitches, accelerometers • Machine listening • Score following • Gesture recognition using neural networks • e.g. “throw” sound into the audience

  30. Block Diagram of System

  31. System Diagram

  32. The “Team” • Lots of people needed to make this happen • Project manager to keep us in line • The Composer/Sound stage designer • The Graphic Designer • The Graphics Programmer • The Audio Programmer • The Electronics technical wizard • The Local site organiser • The commercial suppliers • And … • THE PERFORMER

  33. Team Photo

  34. Screens and Performer

  35. Examples of the RIMM Computer Graphics

  36. Technology becomes e-Science • Clearly a lot of technology to make it work • Some is bespoke electronics • Majority of tasks were programmed • The SGI Origin 2000 • 8 fast processors • Highly parallel • jMax • 24-bit optical output for audio • High speed network interconnect to graphics computer • SGI O2 graphics • Optimised for OpenGL • Texture memory drawn from main memory

  37. Academic Background • Involvement in Performing Arts • My involvement with technology • Technology • Bespoke hardware • Computers • Networks • Software • Engaging with this technology • Right people in the right place • Future • Questions

  38. Technologies for All • Many of these technologies are available • Many of the ideas are now embodied in software • Many are free • Pure Data – aka PD (with GEM add-in) • Eyesweb • VJ-ing is growing • Many VJ products around • Grid Computing is available • Not easily accessible • Lacks some functionality for Performing Arts • Photonic networks are available • High speed links point to point • Up to 10 gb/s possible now • 1 gb/s across the globe

  39. jMax, Pure Data/GEM • Graphics Programming • Lots of modules • Connect up inputs and outputs • Uses the modular synthesiser metaphor • GEM • Graphics Environment for Musicians • Link audio to graphics • Limited by modules unless you write your own

  40. eyesweb • Video programming with modules • Huge number of modules • Includes “man” tracking capability • Can link to rudimentary audio • But … • Can also link to PD • using Open Sound Control protocols - OSC

  41. REACTABLE • Collaborative electronic music instrument • Tabletop tangible multi-touch interface • Several simultaneous performers • Share complete control over the instrument • by moving and rotating physical objects • on a luminous round table surface. • Objects represent components of a classic modular synthesizer • can create complex and dynamic sonic topologies • a kind of tangible modular synthesizer or • graspable flow-controlled programming language

  42. REACTABLE

  43. REACTABLE - demo Click on Image to go to site with Videos [will open a browser]

  44. More Video Tracking and Performance Messa di Voce Tmema,Blonk,La Barbara http://www.tmema.org/messa/messa.html

  45. Messa di Voce - demo Click image to go to the Host site [will try to play Quicktime in a browser]

  46. And just for fun … another demo

  47. Academic Background • Involvement in Performing Arts • My involvement with technology • Technology • Bespoke hardware • Computers • Networks • Software • Engaging with this technology • Right people in the right place • Future • Questions

  48. So what is this Grid Computing? • The term originated in the early 1990’s • Metaphor for making computer power as easy to access as an electric power grid • Many years on and still mainly in the domain of “BIG SCIENCE” • Most Grids are run like 80’s batch style computing • Submit a job description to a scheduler • It decides when and where the job runs – more or less • This is a computational grid • Many computers and clusters are interconnected • Can be globally distributed via photonic networks

  49. What is this Data Grid? • A grid computing system that deals with data • The controlled sharing and management of large amounts of distributed data • These are often, but not always, combined with computational grid computing systems • Finding data is not easy • Accessing it is not easy • Mainly in the domain of “BIG SCIENCE” • Some shift to Arts and Humanities • We have lots of data too! • More later in context of National Grid Service

  50. What are these Photonics Networks • A worldwide network of optical fibres • Speeds are 1gbit/s to 40 gbit/s • Run by different organisations • UK - Janet • Europe - GEANT2 • USA - StarLight • USA - National Lambda Rail • Canada – CANARIE • etc … • A connection is usually point-to-point • Very exclusive • Lots of bandwidth • Low latency and jitter (more of this later)

More Related