Tutorial Session: The Bayesian brain, surprise and free-energy

1. Predictive Coding: Whatever Next?University of Edinburgh, January 19th, 2010 Tutorial Session: The Bayesian brain, surprise and free-energy Value-learning and perceptual learning have been an important focus over the past decade, attracting the concerted attention of experimental psychologists, neurobiologists and the machine learning community. Despite some formal connections; e.g., the role of prediction error in optimizing some function of sensory states, both fields have developed their own rhetoric and postulates. In work, we show that perception is, literally, an integral part of value learning; in the sense that it is necessary to integrate out dependencies on the inferred causes of sensory information. This enables the value of sensory trajectories to be optimized through action. Furthermore, we show that acting to optimize value and perception are two aspects of exactly the same principle; namely the minimization of a quantity (free energy) that bounds the probability of sensations, given a particular agent or phenotype. This principle can be derived, in a straightforward way, from the very existence of biological agents, by considering the probabilistic behaviour of an ensemble of agents belonging to the same class. Put simply, we sample the world to maximise the evidence for our existence.

2. “Objects are always imagined as being present in the field of vision as would have to be there in order to produce the same impression on the nervous mechanism” - Hermann Ludwig Ferdinand von Helmholtz Geoffrey Hinton From the Helmholtz machine and the Bayesian Brain to Action and self-organization Richard Feynman Thomas Bayes Hermann Haken

3. Overview Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong and categorization Simulated lesions Action Active inference Reaching Policies Control and attractors The mountain-car problem

4. pH transport falling temperature Self-organization that minimises an ensemble density to ensure a limited repertoire of states are occupied (i.e., ensuring states have a random attracting set). Particle density contours showing Kelvin-Helmholtz instability, forming beautiful breaking waves. In the self-sustained state of Kelvin-Helmholtz turbulence the particles are transported away from the mid-plane at the same rate as they fall, but the particle density is nevertheless very clumpy because of a clumping instability that is caused by the dependence of the particle velocity on the local solids-to-gas ratio (Johansen, Henning, & Klahr 2006)

5. How can an active agent minimise its equilibrium entropy? This entropy is bounded by the entropy of sensory signals (under simplifying assumptions) Crucially, because the density on sensory signals is at equilibrium, it can be interpreted as the proportion of time each agent entertains them (the sojourn time). This ergodic argument means that entropy is the path integral of surprise experienced by a particular agent: This means agents minimise surprise at all times. But there is one small problem…Agents cannot access surprise; however, they can evaluate a free-energy bound on surprise, which is induced with a recognition density q :

6. Overview Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

7. The free-energy principle Sensations Action Internal states of the agent (m) External states in the world Action to minimise a bound on surprise Perception to optimise the bound

8. The generative model The free-energy rests on expected Gibb’s energy and can be evaluated, given a generative model comprising a likelihood and prior: So what models might the brain use?

9. Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem lateral Backward (nonlinear) Forward (linear) Processing hierarchy

10. Hierarchical (deep) dynamic models

11. Prediction errors Hierarchal form Gibb’s energy: a simple function of prediction error Likelihood and empirical priors Dynamical priors Structural priors

12. The recognition density and its sufficient statistics Mean-field approximation: Laplace approximation: Perception and inference Learning and memory Activity-dependent plasticity Synaptic activity Synaptic efficacy Functional specialization Attentional gain Enabling of plasticity Synaptic gain Attention and salience

13. David Mumford Perception and message-passing Forward prediction error Backward predictions Synaptic plasticity Synaptic gain

14. Horace Barlow The free-energy principle and infomax The infomax principle requires the mutual information between sensory data and their conditional representation to be maximal, under prior constraints on the representations If the recognition density is a point mass In short, the infomax principle is a special case of the free-energy principle that obtains when we discount uncertainty and represent sensory data with point estimates of their causes. Alternatively, the free-energy is a generalization of the infomax principle that covers probability densities on the unknown causes of data.

15. Overview Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong and categorization Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

16. Synthetic song-birds Vocal centre Syrinx Sonogram Frequency 0.5 1 1.5 Time (sec)

17. prediction and error 20 15 10 5 0 -5 10 20 30 40 50 60 time Causal states 20 15 stimulus 10 5000 5 4500 0 4000 -5 3500 -10 10 20 30 40 50 60 3000 time (bins) 2500 2000 0.2 0.4 0.6 0.8 time (seconds) Recognition and message passing Backward predictions Forward prediction error hidden states 20 15 10 5 0 -5 10 20 30 40 50 60 time

18. Perceptual categorization 5000 5000 5000 Song A Song B Song C 4000 4000 4000 Frequency (Hz) 3000 3000 3000 2000 2000 2000 0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8 time (seconds)

19. Generative models of birdsong: sequences of sequences Neuronal hierarchy Syrinx sonogram Frequency (KHz) 0.5 1 1.5 Time (sec) Kiebel et al

20. Simulated lesion studies: a model for false inference in psychopathology? percept LFP 60 5000 4500 40 4000 20 Frequency (Hz) LFP (micro-volts) 3500 0 3000 -20 2500 2000 -40 1 1.5 0 500 1000 1500 2000 no structural priors LFP 60 5000 40 4500 20 4000 Frequency (Hz) LFP (micro-volts) 3500 0 3000 -20 2500 -40 2000 -60 1 1.5 0 500 1000 1500 2000 no dynamical priors LFP 60 5000 40 4500 4000 20 Frequency (Hz) LFP (micro-volts) 3500 0 3000 -20 2500 -40 2000 -60 0.5 1 1.5 0 500 1000 1500 2000 time (seconds) peristimulus time (ms)

21. Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

22. From reflexes to action prediction True dynamics Generative model dorsal root action ventral horn

23. Movement trajectory visual input Descending sensory prediction error From reflexes to action Jointed arm proprioceptive input

24. Overview Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

25. Cost-functions, priors and policies with attractors Adriaan Fokker Max Planck At equilibrium we have: This means maxima of the equilibrium density must have negative divergence. We can exploit this to ensure maxima lie in A, where cost increases dissipation

26. The mountain car problem The environment The cost-function 0.7 0.6 0.5 0.4 height 0.3 0.2 0.1 0 -2 -1 0 1 2 position happiness position True equations of motion Policy (expected equations of motion)

27. a Exploring & exploiting the environment With cost (i.e., exploratory dynamics)

28. Adaptive policies and trajectories Using just the free-energy principle and a simple gradient ascent scheme, we have solved a benchmark problem in optimal control theory using a handful of learning trials. Note that we use reinforcement learning or dynamic programming.

29. Computational motor control Minimisation of sensory prediction error Attention and biased competition Optimisation of synaptic gain representing the precision (salience) of predictions Exploration and exploitation Policies as prior expectations on motions Predictive coding and hierarchical inference Minimisation of prediction error with recurrent message passing Associative plasticity Optimisation of synaptic efficacy Optimal control and value learning Optimisation of a free-energy bound on surprise or value The Bayesian brain hypothesis Minimising the difference between a recognition density and the conditional density on sensory causes Perceptual leaning and memory Optimisation of synaptic efficacy to represent causal structure in the sensorium The free-energy principle Minimisation of the free-energy of sensations and the representation of their causes Infomax and the redundancy minimisation principle Maximisation of the mutual information between sensations and representations Probabilistic neuronal coding Encoding a recognition density in terms of conditional expectations and uncertainty Model selection and evolution Optimising the agent’s model and priors through neurodevelopment and natural selection

30. time-scale process Perception and Action: The optimisation of neuronal and neuromuscular activity to suppress prediction errors (or free-energy) based on generative models of sensory data. Learning and attention: The optimisation of synaptic gain and efficacy over seconds to hours, to encode the precisions of prediction errors and causal structure in the sensorium. This entails suppression of free-energy over time. Neurodevelopment: Model optimisation through activity-dependent pruning and maintenance of neuronal connections that are specified epigenetically Evolution: Optimisation of the average free-energy (free-fitness) over time and individuals of a given class (e.g., conspecifics) by selective pressure on the epigenetic specification of their generative models.

31. Thank you And thanks to collaborators: Jean Daunizeau Lee Harrison Stefan Kiebel James Kilner Klaas Stephan And colleagues: Peter Dayan Jörn Diedrichsen Paul Verschure Florentin Wörgötter