1 / 22

Neural Networks Chapter 2 - PowerPoint PPT Presentation

Neural Networks Chapter 2. Joost N. Kok Universiteit Leiden. Hopfield Networks. Network of McCulloch-Pitts neurons Output is 1 iff and is -1 otherwise . Hopfield Networks. Hopfield Networks. Hopfield Networks. Hopfield Networks.

Related searches for Neural Networks Chapter 2

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Neural NetworksChapter 2

Joost N. Kok

Universiteit Leiden

• Network of McCulloch-Pitts neurons

• Output is 1 iff and is -1 otherwise

• Associative Memory Problem:Store a set of patterns in such a way that when presented with a new pattern, the network responds by producing whichever of the stored patterns most closely resembles the new pattern.

• Resembles = Hamming distance

• Configuration space = all possible states of the network

• Stored patterns should be attractors

• Basins of attractors

• N neurons

• Two states: -1 (silent) and 1 (firing)

• Fully connected

• Symmetric Weights

• Thresholds

w13

w16

w57

-1

+1

• State:

• Weights:

• Dynamics:

• Hebb’s learning rule:

• Make connection stronger if neurons have the same state

• Make connection weaker if the neurons have a different state

neuron 1

synapse

neuron 2

• Weight betweenneuron i and neuron j is given by

• Opposite patterns give the same weights

• This implies that they are also stable points of the network

• Capacity of Hopfield Networks is limited: 0.14 N

• Hopfield defines the energy of a network:

E = - ½ ijSiSjwij + i Siqi

• If we pick unit i and the firing rule does not change its Si, it will not change E.

• If we pick unit i and the firing rule does change its Si, it will decrease E.

• Energy function:

• Alternative Form:

• Extension: use stochastic fire rule

• Si := +1 with probability g(hi)

• Si := -1 with probability 1-g(hi)

g(x) =

1 + e – xb

Hopfield Networks

• Nonlinear function:

b

g(x)

b 0

x

• Replace the binary threshold units by binary stochastic units.

• Defineb = 1/T

• Use “temperature” T to make it easier to cross energy barriers.

• Start at high temperature where its easy to cross energy barriers.

• Reduce slowly to low temperature where good states are much more probable than bad ones.

A B C

• Kick the network our of spurious local minima

• Equilibrium: becomes time independent