1 / 61

Understanding Hebb's Rule: Foundations of Learning in Artificial Neural Networks

This article explores Hebb's Rule, a pivotal learning principle in artificial neural networks (ANNs), introduced by Donald Hebb in 1949. The rule postulates that synaptic strength adjusts based on the correlational activity between neurons, leading to learning. This document discusses the formalization of Hebb's Rule, its implications on synaptic modifications, and how normalization enhances its effectiveness without external guidance. By analyzing both traditional and normalized Hebbian learning, we uncover its role in unsupervised learning algorithms and the self-organization of neural architectures.

toyah
Download Presentation

Understanding Hebb's Rule: Foundations of Learning in Artificial Neural Networks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. COMP305. Part I. Artificial neural networks.

  2. Topic 3. Learning Rules of the Artificial Neural Networks.

  3. Hebb’s rule (1949). • Hebb conjectured that a particular type of use-dependent modificationof the connection strength of synapses might underlie learning in the nervous system.

  4. Hebb’s rule (1949). • Hebb introduced a neurophysiological postulate: “…When an axon of cell A is near enough to excite a cell B and repeatedly and persistently tales part in firing it, some growth process or metabolic change takes place in one or both cells, such that A’s efficiency as one of the cells firing B, is increased.”

  5. Hebb’s rule (1949). The simplest formalisation of Hebb’s rule is to increase weight of connection at every next instant in the way: (1) where (2)

  6. Hebb’s rule (1949). (1) where (2) here wjik is the weight of connection at instant k, wjik+1 is the weight of connection at the following instant k+1, Dwjikis increment by which the weight of connection is enlarged, Cis positive coefficient which determines learning rate, aikis input value from the presynaptic neuron at instant k, Xjkis output of the postsynaptic neuron at the same instant k.

  7. Hebb’s rule (1949). (1) where (2) • Thus, the weight of connection changes at the next instant only if both preceding input via this connection and the resulting output simultaneously are not equal to 0.

  8. Hebb’s rule (1949). (1) where (2) • Equation (2) emphasises the correlation nature of a Hebbian synapse. It is sometimes referred to as the activity product rule.

  9. Hebb’s rule (1949). (1) where (2) • Hebb’s original learning rule (2) referred exclusively to excitatory synapses, and has the unfortunate property that it can only increase synaptic weights, thus washing out the distinctive performance of different neurons in a network, as the connections drive into saturation..

  10. Hebb’s rule (1949). (1) where (2) • However, when the Hebbian rule is augmented by a normalisationrule, e.g. keeping constant the total strength of synapses upon a given neuron, it tends to “sharpen” a neuron’s predisposition “without a teacher”, causing its firing to become better and better correlated with a cluster of stimulus patterns.

  11. Normalised Hebb’s rule. (1) where (2) normalisation: (3) • Hebb’s rule plays an important role in studies of ANN algorithms much “younger” than the rule itself, such as unsupervised learning or self-organisation.

  12. Normalised Hebb in practice. a Input unit No 1 2 3 4 w 1 1 w a 2 2 q X a w 3 3 w 4 a 4

  13. Normalised Hebb in practice. t=0C=1 Input unit No 1 2 3 4 1 1 q =1 X 1 1

  14. Normalised Hebb in practice. t=0C=1 Input unit No 1 2 3 4 1 1 q =1 X 1 1

  15. Normalised Hebb in practice. t=0C=1 Input unit No 1 2 3 4 1 1 q =1 X 1 1

  16. Normalised Hebb in practice. t=0C=1 Input unit No 1 2 3 4 0.5 0.5 q =1 X 0.5 0.5

  17. Normalised Hebb in practice. t=0C=1 Input unit No 1 2 3 4 1 0.5 0.5 0 q =1 X 0.5 1 0.5 0

  18. Normalised Hebb in practice. t=0C=1 Input unit No 1 2 3 4 1 0.5 0.5 0 q =1 X 0.5 1 0.5 0

  19. Normalised Hebb in practice. t=0C=1 Input unit No 1 2 3 4 1 0.5 0.5 0 q =1 1 0.5 1 0.5 0

  20. Normalised Hebb in practice. t=0C=1 Input unit No 1 2 3 4 1 0.5 0.5 0 q =1 1 0.5 1 0.5 0

  21. Normalised Hebb in practice. t=1C=1 Input unit No 1 2 3 4 1.5 0.5 q =1 X 1.5 0.5

  22. Normalised Hebb in practice. t=1C=1 Input unit No 1 2 3 4 1.5 0.5 q =1 X 1.5 0.5

  23. Normalised Hebb in practice. t=1C=1 Input unit No 1 2 3 4 1.5 0.5 q =1 X 1.5 0.5

  24. Normalised Hebb in practice. t=1C=1 Input unit No 1 2 3 4 0.67 0.22 q =1 X 0.67 0.22

  25. Normalised Hebb in practice. t=1C=1 Input unit No 1 2 3 4 0.67 0.22 q =1 X 0.67 0.22

  26. Normalised Hebb in practice. t=1C=1 Input unit No 1 2 3 4 0.67 0.22 q =1 X 0.67 Continue… 0.22

  27. Normalised Hebb in practice. t=1C=1 Input unit No 1 2 3 4 1 0.67 0.22 0 q =1 X 0.67 1 0.22 0

  28. Normalised Hebb in practice. t=1C=1 Input unit No 1 2 3 4 1 0.67 0.22 0 q =1 1 0.67 1 0.22 0

  29. Normalised Hebb in practice. t=1C=1 Input unit No 1 2 3 4 1 0.67 0.22 0 q =1 1 0.67 1 0.22 0

  30. Normalised Hebb in practice. t=2C=1 Input unit No 1 2 3 4 1.67 0.22 q =1 X 1.67 0.22

  31. Normalised Hebb in practice. t=2C=1 Input unit No 1 2 3 4 1.67 0.22 q =1 X 1.67 0.22

  32. Normalised Hebb in practice. t=2C=1 Input unit No 1 2 3 4 1.67 0.22 q =1 X 1.67 0.22

  33. Normalised Hebb in practice. t=2C=1 Input unit No 1 2 3 4 0.70 0.09 q =1 X 0.70 0.09

  34. Normalised Hebb in practice. t=2C=1 Input unit No 1 2 3 4 0.70 0.09 q =1 X 0.70 0.09

  35. Normalised Hebb in practice. t=2C=1 Input unit No 1 2 3 4 0.70 0.09 q =1 X 0.70 0.09 Continue…

  36. Normalised Hebb in practice. t=2C=1 Input unit No 1 2 3 4 1 0.70 0.09 0 q =1 X 0.70 1 0.09 0

  37. Normalised Hebb in practice. t=2C=1 Input unit No 1 2 3 4 1 0.70 0.09 0 q =1 1 0.70 1 0.09 0

  38. Normalised Hebb in practice. t=2C=1 Input unit No 1 2 3 4 1 0.70 0.09 0 q =1 1 0.70 1 0.09 0

  39. Normalised Hebb in practice. t=3C=1 Input unit No 1 2 3 4 1.70 0.09 q =1 X 1.70 0.09

  40. Normalised Hebb in practice. t=3C=1 Input unit No 1 2 3 4 1.70 0.09 q =1 X 1.70 0.09

  41. Normalised Hebb in practice. t=3C=1 Input unit No 1 2 3 4 1.70 0.09 q =1 X 1.70 0.09

  42. Normalised Hebb in practice. t=3C=1 Input unit No 1 2 3 4 0.71 0.04 q =1 X 0.71 0.04

  43. Normalised Hebb in practice. t=3C=1 Input unit No 1 2 3 4 0.71 0.04 q =1 X 0.71 0.04 Continue…

  44. Normalised Hebb in practice. t=3C=1 Input unit No 1 2 3 4 1 0.71 0.04 0 q =1 X 0.71 1 0.04 0

  45. Normalised Hebb in practice. t=3C=1 Input unit No 1 2 3 4 1 0.71 0.04 0 q =1 1 0.71 1 0.04 0

  46. Normalised Hebb in practice. t=3C=1 Input unit No 1 2 3 4 1 0.71 0.04 0 q =1 1 0.71 1 0.04 0

  47. Normalised Hebb in practice. t=3C=1 Input unit No 1 2 3 4 1 1.71 0.04 0 q =1 1 1.71 1 0.04 0

  48. Normalised Hebb in practice. t=4C=1 Input unit No 1 2 3 4 1.71 0.04 q =1 X 1.71 0.04

  49. Normalised Hebb in practice. t=4C=1 Input unit No 1 2 3 4 1.71 0.04 q =1 X 1.71 0.04

  50. Normalised Hebb in practice. t=4C=1 Input unit No 1 2 3 4 0.71 0.02 q =1 X 0.71 0.02

More Related