Basic Models in Neuroscience

1. Basic Models in Neuroscience Oren Shriki 2010 Associative Memory

2. Associative Memory in Neural Networks • Original work by John Hopfield (1982). • The model is based on a recurrent network with stable attractors.

3. The Basic Idea • Memory patterns are stored as stable attractors of a recurrent network. • Each memory pattern has a basin of attraction in the phase space of the network.

5. Information Storage • The information is stored in the pattern of synaptic interactions.

6. Energy Function In some models the dynamics are governed by an energy function The dynamics lead to one of the local minima of the energy function, which are the stored memories.

40. Important properties of the model • Content Addressable Memory (CAM) - Access to memory is based on the content and not an address. • Error correction – The network “corrects” the neurons which are inconsistent with the memory pattern.

41. The Mathematical Model

42. Binary Networks • We will use binary neurons: (-1) means ‘inactive’ and (+1) means ‘active’. • The dynamics are given by: External input Input from within the network

43. Stability Condition for a Neuron • The condition for a neuron to remain with the same activity is that its current activity and its current input have the same sign:

44. Energy Function • If the external inputs are constant the network may reach a stable state, but this is not guaranteed (the attractors may be limit cycles and the network may even be chaotic). • When the recurrent connections are symmetric and there is no self coupling we can write an energy function, such that at each time step the energy decreases or does not change. • Under these conditions, the attractors of the network are stable fixed points, which are the local minima of the energy function.

45. Energy Function • Mathematically, the conditions are: • The energy is given by: • And one can prove that:

46. Setting the Connections • Our goal is to embed in the network stable stead-states which will form the memory patterns. • To ensure the existence of such states, we will choose symmetric connections, that guarantee the existence of an energy function.

47. Setting the Connections • We will denote the P memory patterns by: • For instance, for a network with 4 neurons and 3 memory patterns, the patterns can be:

48. Setting the Connections • Hopfield proposed the following rule: The correlation among neurons across memory patterns A normalization factor

49. Choosing the Patterns to Store • To enhance the capacity of the network we will choose patterns that are not similar to one another. • In the Hopfield model, (-1) and (+1) are chosen with equal probabilities. In addition, there are no correlations among the neurons within a pattern and there are no correlations among patterns.

50. Memory Capacity • Storing more and more patterns adds more constraints to the pattern of connections. • There is a limit on the number of stable patterns that can be stored. • In practice, a some point a new pattern will not be stable even if we set the network to this pattern.