Neuromodulation in Artificial Neural Networks

1. Neuromodulation in Artificial Neural Networks Andy Philippides

2. Outline • Biological background • GasNet background • Why increased evolvability? 3 hypotheses from neuromodulation • Hypothesis 1: Different temporal scales • Hypothesis 2: Spatial embedding • Hypothesis 3: Modulation • Different temporal scales (esp. GasNets vs CTRNNs) • Coupled networks through spatial embedding • Conclusions and caveats

3. Point-to-point transmission at synapses Short temporal scale Overriding metaphor is electrical nodes connected by wires ClassicalNeurotransmission • Inspiration for standard connectionist ANN 

4. We now know it’s not quite that simple • Single neurons are highly complex • Brains involve many interacting dynamical systems • Discovery of gaseous diffusible neurotransmitters (NO,CO,H2S – all highly toxic!) • Allow interactions between synaptically unconnected neurons • Gases are neuromodulatory: “Any communication between neurons caused by the release of a chemical that is either not fast, or not point-to-point or not simply excitation or inhibition” (Katz, 1999)

5. Freely diffusing therefore can act as a volume signal Acts over wide range of temporal scales (ms to years) Involved in learning, vasodilation, modulation of sensory input etc Modulatory: up-regulates/down-regulates neuron gains (sometimes both at the same neuron) Highly reactive so cannot measure directly To understand its function must model it: combination of mathematical modelling and more abstract GasNet model Neuromodulation by nitric oxide (NO)

6. NO in the locust Mushroom Body Mushroom Body is the seat of associative learning Made up of ~50k parallel tubular Kenyon Cells (KCs) Concentric organisation: NO-synthesising KCs surround NO-receptive KCS [SHOW MOVIE]

7. How does this morphology affect the NO signals? Single KCs are ineffective but NO from multiple KCs summates: serves as a low-pass filter (also seen in evolved GasNets) Signal variability reduced by segregated organisation

8. Inspiration for new form of ANN: GasNets Positive and negative electrical connections + diffusing modulatory gas Node emits gas due to high electrical or chemical activity 2 distinct but linked signalling systems: electrical and chemical Space and time crucial, local processes SHOW DEMO

9. Diffusion in GasNets (1) Receiver Emitter 1. Gas cloud centred on emitter with a (variable) radius of action 2. Gas concentration at receiver dependent on distance to emitter i.e. Conc_Receiver = f(Distance) Conc_Emitter Space nodes exist in is crucial to action of gas

10. Diffusion in GasNets (2) 3. Gas concentrations at emitter rise and fall over time at a (1-10 times) slower rate than electrical changes 4. Gas alters slope of transfer functions of receivers Receiver Emitter Rate of rise = 3 Ojt = tanh[kjt(ΣwijOit-1 + Ij) + bj] Chemical and Electrical on different temporal scales

11. GasNets evolvability • GasNet controllers developed by evolutionary search • Experiments focussed on the speed of evolution of different networks which is taken as a proxy for evolvability • Idea is that networks that are evolvable will also be adaptable to new environments and suitable for the generation of autonomous behaviour • Evidence of faster (evolutionary) search for GasNet controllers on a number of robot (and other) tasks • New variants developed which further improve evolvability (twice and 10 times as fast) • Analysis of overlap of chemical/electrical connections shows looser coupling for new variants

12. Why is evolvability increased? No explanation for GasNet evolvability in terms of fitness landscape properties (eg neutrality, epistasis etc) Return to neuromodulation definition: “… communication … that is either not fast, or not point-to-point or not simply excitation or inhibition” Suggests 3 (linked) hypotheses: 1. Action of gas over different (and separate) temporal scales 2. Spatial embedding of network: serves to (flexibly) couple 2 interacting signalling systems? Other effects? 3. Modulatory effects

13. GasNet vs ‘Instant’ GasNet Test a new variant where rise/fall of gas is on the same temporal scale as electrical connections Instant Plexus worse (roughly twice as bad) than original on robot and CPG task (Philippides, Cherian) Seemed to be bad at tuning behaviour to get highest fitnesses ‘Tuning time’ of 467 generations compared to 100 for standard

14. CTRNN vs GasNet If availability of different temporal scales is useful, if a network’s electrical connections can be ‘slow’, is it equivalent to a GasNet? • Eg in a CTRNN (leaky integrator), time-scale of electrical activation set by t • = 1, nodes are integrate-and-fire, as in the GasNet • > 1, nodes integrate activity over time, as the gas does Work by Chris Buckley suggests the explicit separation of time-scales, as in nervous systems could be key

15. Sven Magg: GasNet vs CTRNN (2) • Compared CTRNN and GasNets on 2 CPG tasks • On 4:1 GasNets were better and quicker than CTRNNs • Lots of variants of connection scheme etc

16. However on 7:5 pattern GasNets were worse • Why the differences for different patterns? Similar to work of Gary McHale evolving gaits in quadrupeds and bipeds: GasNets to be as good as and more ‘general’ than CTRNNs

17. Different temporal dynamics • Patterns have a different balance of on-off ones and zeroes • CTRNNs have the same on and off dynamics • Combination of modulation and temporal dynamics inherent in GasNets lead to different on-off dynamics

18. Sven Magg: GasNet vs CTRNN (3) Next used Beer’s diamond catching, ball avoiding robot: a reactive task? If timing unimportant will GasNets be bad? • CTRNNs better but so are integrate and fire neurons • However, NoGas worse than GasNet… • By stripping down elements of GasNets, addition of weights helped network NOT timing elements • Is it simply timing that aids GasNet evolvability?

19. GasNet vs CTRNN (4): Mark Pratley • Range of (simulated) robot walking tasks (pattern generation) • Used GasNet CTRNN: slope of transfer function affected by gas: • Non-spatial: extent of gas effect determined by weighted connections (both elec. and chem. nets fully connected) • CTRNN with gas outperforms standard: modulation is key? • But different performance of GasNet variants suggests it is not all temporal dynamics and modulation: spatial embedding?

20. Temporal scales summary • Different temporal scales aid evolvability and temporal adaptibility • In both real and artificial nervous systems, slow scales can be used to filter noisy inputs • Can match world time-scales longer than those inherent in the network, but it’s harder to tune mechanisms • Explicit separation of different mechanisms’ time-scales may be useful • High transfer slopes can easily lead to oscillatory networks • Certainly differences in GasNets and CTRNNs which may be due to the differences in on-off dynamics

21. Evolvability and coupling • Original GasNets had tight coupling between electrical and chemical processes as ‘connections’ were specified over the same space • New versions allow for more flexibly coupled processes with distinct characteristics (electrical, chemical) • May allow non-destructive “tuning” of functionality of one system against the other (Conrad, 90)

22. What if we alter the (spatial) coupling by changing gas diffusion? 1. Gas spreads through the entire network (more destructive interference?) – [Red line] 2. Different 2d spaces for chemical and electrical (less coupling?) –[Green line] Median run lengths over 20 runs: Plexus = 512, Whole = 1130, Decoupled = 638) Summary: Whole is like the original GasNet. No significant difference between Plexus and Decoupled

23. Modularity (Peter Fine) Also, changing the spatial embedding of the network leads to differences in their modularity Plexus Original Here the more connected network evolved faster

24. Mark Pratley: Does coupling change through evolution? Examined the correlation of electrical and chemical weight matrices during evolution Most runs inconclusive (though definite clusters) However, 1 (/100) run produced the following graph of coupling against evolutionary time An accident? Or proof that coupling has an effect? Also, constraining the coupling to different degrees affects fitness depending on degree: unconstrained and 15% best

25. Conclusions • Different and explicitly separate temporal scales seems useful • CTRNNs vs GasNets: no clear winner – especially if weights are used but there are differences – maybe due to different off-on temporal dynamics • Flexible coupling of 2 systems aids evolvability • Some evidence of coupling changing during the course of evolution and changing the outcome of evolutionary search • What about modulation? Intuitively, seems powerful to have 2 different modes of connection: additive and multiplicative • Maybe the combination of 2 signalling channels: fast, additive electrical connections and slow, modulatory/mutliplicative gases allows independent behaviour on different time-scales and rich adaptive dynamics in general (cf Buckley and scale)

26. Future work Spatially Embedded Complex Systems Engineering (SECSE) will look in more detail at issues raised (Patricia Vargas, Lionel Barnett, Xiaobing Hu, Dave Harris (SPRU)) • Better measures of coupling • Measures for on-line dynamics in networks • Systematic investigations of impact on evolvability of different temporal scales, spatial embedding and modulation Visually guided insect learning algorithms to develop more complex GasNets • GasNets and learning • More complex architectures (eg spiking network Dan Bush)

27. Acknowledgements • Phil Husbands • Michael O’Shea • Sven Magg • Peter Fine • Mark Pratley • Philip Cherian • Chris Buckley • Tom Smith • Complex Networks Workshop (which led to SECSE) • EPSRC/BBSRC funded work

28. Original spatial distribution based on a single neuron Plexus has a uniform distribution Gas cloud centred on a genetically specified position in the network plane Plexus GasNet

29. Receptor GasNet • In nervous systems, only neurons with one of the receptors for NO will be affected • Each node may have quantities of receptors (none, medium, maximum) • Modulation depends on amount of receptor Gas concentration Receptor concentration

30. 0 < k<= 1 Stability Analysis Transfer function is tanh(kx). For a single neuron with +ve recurrency stability of fixed points determined by |k sech2(kx)| >1 ie Stability and number of fixed points governed by |k|

31. k > 1 -1<= k < 0

32. k < -1

33. Leads to various ‘interesting’ dynamics for 2 neurons with different ‘phases’ of stable/semi-stable behaviour cf Buckley: showed gas allows ‘re-setting’ of network from a stable to a unstable state

34. Conclusions: Mark Pratley Tanh activation function performs better than sigmoidal Top group tanh variants, bottom group sigmoids Adding a function modifying variable k can give improvement, However effect is dependent on whether gas is used and which activation function

35. Gas increases fitness by increasing sensitivity of neurons thus increasing leg oscillation size Type of gas needed depends on range of k’s allowed Transplanting neurons evolved with gas into NoGas and vice versa show electrical system is robust Electric system performs leg movement with a smooth phase-locked horizontal and vertical rhythm, while gas tunes the size of oscillations.