The NESS Project

1. The NESS Project MUSICA Seminar Series University of Edinburgh 13 February 2013 Stefan Bilbao, Paul Graham, Alan Gray, Brian Hamilton Kostas Kavoussanakis, James Perry, Alberto Torin, and Craig Webb Acoustics Group EPCC Work supported by the European Research Council

2. Background 2

3. Background 3

4. The HPC Centre of the University of Edinburgh • 70 staff • £4M turnover (almost) all from external sources 4

5. Case Study: FilmGrid Photo by Brad & Sabrina • Distributed digital asset management system • Accurate supervision of the progress of a film production • Secure asset transfer between entities working on the film • Complements existing film editing software (e.g. Nucoda) • Software support for current day-to-day manual workflow processes 5

6. Case Study: FilmGrid Photo by Jeremy Keith 6

7. Academic: • Permanent staff with degrees in various backgrounds: • Physics, Computer Science, Mathematics, Chemistry, Biology… • Education and Training provider: • MSc in HPC • Bespoke training on HPC, accelerators, multi-core computing • Strengths: • Project management • Interdisciplinary projects: small pilots, distributed programmes • User support • Technical: • Software development for academia and industry • Code optimisation (serial, parallel, accelerators) • Simulation and modelling • Wide language, operating system and computer architecture expertise http://www.epcc.ed.ac.uk/ 7

8. NESS • An ERC funded project (PE6: Informatics), running at Edinburgh, 2012—2017. • The goal: digital sound synthesis… …of musical sound • Today: • Digital sound synthesis and physical modeling • Ness project: structure and activities • Group member presentations

9. “Sampling” “Synthesis” Digital Sound Synthesis • A longstanding attempt to move away from the use of recorded sound (sampling) • Analogous to computer graphics rendering? • At the philosophical level…yes • At the technical level…not really!

10. Abstract Digital Sound Synthesis • Early synthesis methods (1950s—1960s): based on simple heuristic building blocks---efficient and easy to program… Wavetables: Sinusoidal Oscillators: • Frequency modulation synthesis: a very successful variant! • Still extremely popular! • Can be difficult to control (lots of user input), and sound quality is generally very artificial! FM trumpet FM bell

11. A fair degree of hybridization abounds: • physical modeling + sampling is analogous to, say, motion capture! Physical Modeling Sound Synthesis Physical models: • based on physical descriptions of musical “objects”… • can be computationally demanding… • potentially very realistic* sound • control parameters: few in number, and perceptually meaningful * realism needs a good definition, if there is not a real-world reference!

12. Methods: Lumped Mass Spring Networks Earliest large scale attempt at physical modeling synthesis: • networks of masses/springs/dampers + simple ODE solver • basis for “Cordis” and “Cordis Anima” systems (Cadoz and collaborators, Grenoble, from 1979 to present!) • generally abstract (modular), but if put in a regular arrangement, it is possible to simulate distributed objects (strings, membranes, etc.) Cymbal Timpani

13. Methods: Modal Synthesis • A different approach: decompose dynamics of vibrating object into modes… • basis for “Modalys” synthesis system (IRCAM, 1985—present) • geared towards linear objects (with interesting extensions to the nonlinear case…Volterra series, e.g.) • a lot of offline precomputation (modal shapes, frequencies)

14. Methods: Digital Waveguides • Yet another approach: decompose dynamics of vibrating object into traveling waves… • developed at CCRMA, Stanford University, 1985—present • roots in early scattering-based speech synthesis methods (Kelly Lochbaum, 1962) • basis for many synthesis systems, including Yamaha VL1 (1994) • meant for simulating distributed linear objects in 1D (strings, acoustic tubes)…extremely efficient! Guitar

15. Methods: Time-stepping Methods (FDTD, FVTD, etc.) • The obvious approach: represent dynamics of vibrating object on a grid and integrate using direct solvers… • distinct roots in musical acoustics investigations, and a few early synthesis attempts (Ruiz, vibrating string, 1969!) • tools exist to handle virtually any system…much more general than other methods • but…a lot of specialization work for audio applications…

16. Electromechanical Instruments NESS: Target Systems • Trying to span the full range of musical acoustic systems… • difficult to approach using other physical modeling techniques… Brass Instruments Nonlinear Plate and Shell Vibration Gong Trumpet Cymbal

17. NESS: Target Systems Modular Synthesis Environments Room Acoustics Modelling Embeddings and Spatialization Snare Drum Excerpt: Orbit, G. Delap, 2009

18. Sample rates and bandwidth • A basic constraint for audio: choice of sample rate Fs, and time step k = 1/Fs • Constraint 1 (necessary): Need to be able to fully render audio up to limits of human audio perception, so: • Fs≥ 40 000 Hz, k ≤1/40000 s • Constraint 2 (desirable): Don’t want to render audio above this range, for efficiency reasons, so: • Fs≈ 40 000 Hz, k ≈ 1/40000 s • Time step is small…lots of computational work to do… • Some aspects of time stepping methods need to be reconsidered in this light!

19. Suppose operation at a given sample rate…need to choose the grid carefully, for perceptual reasons: Increase in grid spacing decrease in operation count Grids and Bandwidth Limitation Simple string model… x 2 decrease in output bandwidth x 4 • an additional constraint in numerical design…certain techniques (grid refinement) are dangerous in an audio context

20. Audibility of Numerical Dispersion Numerical Dispersion: speed of wave propagation is incorrect, numerically! Phase velocity Exact and Numerical Example: thin bar, simple explicit FD method: Mistuning! Exact: 44100 Hz: Careful design necessary…methods with free parameters allow a means of tuning the scheme…at the price of linear system solutions (hard on GPU!)

21. Numerical Instability • A critical concern in synthesis design for non expert users… Linear membrane instability Nonlinear shock wave instability • Not too hard to deal with… • Harder to deal with without compromising audio output…

22. Modular Instability • Consider two rudimentary systems Ideal String Mass/ spring • Problems can appear here if the connection is not handled properly: Unstable Connection Stable Connection • Difficulties are compounded for more complex systems…

23. Energy-based Stability Numerical energy conserved to machine accuracy: …giving a stability guarantee Extremely useful in debugging, and in designing complex modular systems:

24. arithmetic operations/second output, at 48 000 Hz: 1015 108 1016 109 106 1014 107 1012 1013 1010 1011 1017 optimal realtime performance on commercially available single core Nonlinear plates/shells Electromechanical Instruments Brass instruments Concert Halls Small rooms Small embeddings Computational Costs and HPC • Audio sample rates are high: 44 100 Hz, 48 000 Hz, 92 000 Hz… • Flop rates/memory requirement scale as power of sample rate (2,3,4)… • Musical use/experimentation: reasonable compute time (no overnight jobs!) • Solutions: Parallel implementations (GPU, e.g.) • New algorithmic issues: parallelizability, memory management, stability in finite precision

25. 3-D Room Acoustics (Brian Hamilton, Acoustics Group) 25

26. Simulating 3-D Room Acoustics • A wave is a spatial field that changes over time • Sound propagates as a pressure wave • Simulating sound wave propagation: • Pick some 3-D lattice (grid) of points • Calculate sound pressure at each point • Iterate in time... • Problem to solve: • How to do this as efficiently as possible? • Any audible artifacts? How to minimise them? 26

27. Spatial Lattices Which to choose? 27

28. Spatial Lattices Many choices... 28

29. Spatial Lattices • Waves should propagate uniformly in every direction • Symmetry is key! Stacking fruit 29

30. Numerical Dispersion • Numerical dispersion → wave speed error • Simulated waves propagate along axes of grid Wave speed depends on grid orientation!

31. Example: Without Dispersion 31

32. Example: With Dispersion 32

33. Wave Speed Error • We want the error to be isotropic (direction independent) • Delicate cancellation of error in space and time 33

34. Wave Speed Error 34

35. Wave Speed Error 35

36. Audible Artifacts: Examples • Castanet (clean) • Castanet (dispersive) • Guitar (clean) • Guitar (dispersive) 36

37. Percussion Instruments (Alberto Torin, Acoustics) L = Linear, NL = Non-linear

38. Linear Plates w = transverse displacement  = density, H = thickness, D = stiffness parameter • There is no interaction between different modes!

39. Non-linear Plates • Add non-linear terms to previous equation von Kármán equations for non-linear plates F = Airy’s function, E = Young’s modulus

40. Non-linear Plates • Energy exchange between different modes is allowed! • - Crashes, Pitch glide effects

41. Air coupling • Add the pressure on the plate • Introduce the acoustic field , that obeys the wave equation • Add coupling conditions between the air and the plate

42. Numerical schemes • Stability and Energy conservation • Need for a Fast algorithm • Bottleneck of the code is the solution of a sparse linear system (matrices involved have a few non-zero entries) • We can use iterative solvers… • works well for the simple plate • needs extra work when air coupling is present

43. Example: MultiPlate3D Roll gesture Several strikes

44. What is a GPU? • Graphics processing unit • Originally designed for rendering 3D graphics fast • Now also used for general purpose computations (GPGPU) • Very well suited for problems like ours • Especially the 3D ones • Same simple computation required for huge number of points 44

45. Porting Process Matlab C CUDA Optimized CUDA • Port from Matlab to C • Faster than Matlab, will run anywhere • Easier to debug and modify than CUDA • Good basis for CUDA port • Port from C to CUDA • Some code (e.g. setup code) remains in C • Time critical main loop is ported to CUDA • Optimize CUDA code • Gain high performance (as far as possible) 45

46. Example • 3dabc code • Simulates a 3D box of air with various boundary conditions • Run times: Matlab version not optimised Small simulation size - would expect larger speed-up from C to CUDA for large size 46

47. Large-scale 3D virtual acoustics(Craig Webb, Acoustics) • Computing 3D wave propagation in a virtual space. • Dynamic simulations, with full wave behavior. • Can inject dry audio to produce reverberation. • Or embed virtual instruments.

48. Computation Size At a sample rate of 44.1kHz : • 1 cubic metre requires 422 thousand grid points. • 1 second of output requires 185G floating-point operations. • 2,000 cubic metres – 370T operations. • A 10 second simulation – 3.7P operations, that’s 3,700,000,000,000,000. Use multiple GPU cards to accelerate the model. • With 4 cards, speedup over serial C code is in the range x100 ~ x140. • Under an hour per second of simulation, instead of 5 days in serial C. • Requires 10Gb of memory at single precision.

49. Hall Simulation: Dry Audio Input • Audio examples of 2,000 cubic metre hall • Dry guitar input : Output : • More guitars : • Opera singer (anechoic) : Output : • Can move sound around during runtime :

50. Embedded Instruments Timpani Drum The timpani drum is a good test case for 3D physical modeling. We use a non-linear circular membrane, attached to a parabolic shell with fixed boundaries. This is then placed inside the room simulations, and we can model any number of timpani inside the space. These can then be played together, by specifying the timing and type of strikes on each drum. Audio examples One Timpani : Two Timpani : Three Timpani : Four Timpani :