Neural networks chapter 2
Download
1 / 22

Neural Networks Chapter 2 - PowerPoint PPT Presentation


  • 340 Views
  • Updated On :

Neural Networks Chapter 2. Joost N. Kok Universiteit Leiden. Hopfield Networks. Network of McCulloch-Pitts neurons Output is 1 iff and is -1 otherwise . Hopfield Networks. Hopfield Networks. Hopfield Networks. Hopfield Networks.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Neural Networks Chapter 2' - Faraday


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Neural networks chapter 2

Neural NetworksChapter 2

Joost N. Kok

Universiteit Leiden


Hopfield networks
Hopfield Networks

  • Network of McCulloch-Pitts neurons

  • Output is 1 iff and is -1 otherwise





Hopfield networks4
Hopfield Networks

  • Associative Memory Problem:Store a set of patterns in such a way that when presented with a new pattern, the network responds by producing whichever of the stored patterns most closely resembles the new pattern.


Hopfield networks5
Hopfield Networks

  • Resembles = Hamming distance

  • Configuration space = all possible states of the network

  • Stored patterns should be attractors

  • Basins of attractors


Hopfield networks6
Hopfield Networks

  • N neurons

  • Two states: -1 (silent) and 1 (firing)

  • Fully connected

  • Symmetric Weights

  • Thresholds


Hopfield networks7
Hopfield Networks

w13

w16

w57

-1

+1


Hopfield networks8
Hopfield Networks

  • State:

  • Weights:

  • Dynamics:


Hopfield networks9
Hopfield Networks

  • Hebb’s learning rule:

    • Make connection stronger if neurons have the same state

    • Make connection weaker if the neurons have a different state


Hopfield networks10
Hopfield Networks

neuron 1

synapse

neuron 2


Hopfield networks11
Hopfield Networks

  • Weight betweenneuron i and neuron j is given by


Hopfield networks12
Hopfield Networks

  • Opposite patterns give the same weights

  • This implies that they are also stable points of the network

  • Capacity of Hopfield Networks is limited: 0.14 N


Hopfield networks13
Hopfield Networks

  • Hopfield defines the energy of a network:

    E = - ½ ijSiSjwij + i Siqi

  • If we pick unit i and the firing rule does not change its Si, it will not change E.

  • If we pick unit i and the firing rule does change its Si, it will decrease E.


Hopfield networks14
Hopfield Networks

  • Energy function:

  • Alternative Form:

  • Updates:



Hopfield networks16
Hopfield Networks

  • Extension: use stochastic fire rule

    • Si := +1 with probability g(hi)

    • Si := -1 with probability 1-g(hi)


Hopfield networks17

1

g(x) =

1 + e – xb

Hopfield Networks

  • Nonlinear function:

b

g(x)

b 0

x


Hopfield networks18
Hopfield Networks

  • Replace the binary threshold units by binary stochastic units.

  • Defineb = 1/T

  • Use “temperature” T to make it easier to cross energy barriers.

    • Start at high temperature where its easy to cross energy barriers.

    • Reduce slowly to low temperature where good states are much more probable than bad ones.

A B C


Hopfield networks19
Hopfield Networks

  • Kick the network our of spurious local minima

  • Equilibrium: becomes time independent


ad