1 / 21

Comp 3503 / 5013 Dynamic Neural Networks

Comp 3503 / 5013 Dynamic Neural Networks. Daniel L. Silver March, 2014. Outline. Hopfield Networks Boltzman Machines Mean Field Theory Restricted Boltzman Machines (RBM). Dynamic Neural Networks. See handout for image of spider, beer and dog

lacey
Download Presentation

Comp 3503 / 5013 Dynamic Neural Networks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Comp 3503 / 5013Dynamic Neural Networks Daniel L. Silver March, 2014

  2. Outline • Hopfield Networks • Boltzman Machines • Mean Field Theory • Restricted BoltzmanMachines (RBM)

  3. Dynamic Neural Networks • See handout for image of spider, beer and dog • The search for a model or hypothesis can be considered the relaxation of a dynamic system into a state of equilibrium • This is the nature of most physical systems • Pool of water • Air in a room • Mathematics is that of thermal-dynamics • Quote from John Von Neumann

  4. Hopfield Networks • See hand out

  5. Hopfield Networks • Hopfield Network video intro • http://www.youtube.com/watch?v=gfPUWwBkXZY • http://faculty.etsu.edu/knisleyj/neural/ • Try these Applets: • http://lcn.epfl.ch/tutorial/english/hopfield/html/index.html • http://www.cbu.edu/~pong/ai/hopfield/hopfieldapplet.html

  6. Hopfield Networks Basics with Geoff Hinton: • Introduction to Hopfield Nets • http://www.youtube.com/watch?v=YB3-Hn-inHI • Storage capacity of Hopfield Nets • http://www.youtube.com/watch?v=O1rPQlKQBLQ

  7. Hopfield Networks Advanced concepts with Geoff Hinton: • Hopfield nets with hidden units • http://www.youtube.com/watch?v=bOpddsa4BPI • Necker Cube • http://www.cs.cf.ac.uk/Dave/JAVA/boltzman/Necker.html • Adding noise to improve search • http://www.youtube.com/watch?v=kVgT2Eaa6KA

  8. Boltzman Machine • See Handout • http://www.scholarpedia.org/article/Boltzmann_machine Basics with Geoff Hinton • Modeling binary data • http://www.youtube.com/watch?v=MKdvJst8a6k • BM Learning Algorithm • http://www.youtube.com/watch?v=QgrFsnHFeig

  9. Limitations of BMs • BM Learning does not scale well • This is due to several factors, the most important being: • The time the machine must be run in order to collect equilibrium statistics grows exponentially with the machine's size = number of nodes • For each example – sample nodes, sample states • Connection strengths are more plastic when the units have activation probabilities intermediate between zero and one. Noise causes the weights to follow a random walk until the activities saturate (variance trap).

  10. Potential Solutions • Use a momentum term as in BP: • Add a penalty term to create sparse coding (encourage shorter encodings for different inputs) • Use implementation tricks to do more in memory – batches of examples • Restrict number of iterations in + and – phases • Restrict connectivity of network wij(t+1)=wij(t) +ηΔwij+αΔwij(t-1)

  11. Restricted Boltzman Machine hjpj=1/(1-e-Σj) Recall = Relaxation Σj=wijvi j wij voorho Σi=wijhj i vipi=1/(1-e-Σi) Oscar Winner SF/Fantasy • Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/

  12. Restricted Boltzman Machine hjpj=1/(1-e-Σj) Recall = Relaxation Σj=wijvi j wij voorho Σi=wijhj i vipi=1/(1-e-Σi) Oscar Winner SF/Fantasy • Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/

  13. Restricted Boltzman Machine hjpj=1/(1-e-Σj) Recall = Relaxation SF/Fantasy Oscar Winner Σj=wijvi j wij voorho Σi=wijhj i vipi=1/(1-e-Σi) Oscar Winner SF/Fantasy • Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/

  14. Restricted Boltzman Machine hjpj=1/(1-e-Σj) Recall = Relaxation SF/Fantasy Oscar Winner Σj=wijvi j wij voorho Σi=wijhj i vipi=1/(1-e-Σi) Oscar Winner SF/Fantasy • Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/

  15. Restricted Boltzman Machine hjpj=1/(1-e-Σj) Learning = ~ Gradient Descent = Constrastive Divergence Update hidden units Σj=wijvi j P=P+vihj voorho i Σi=wijhj vipi=1/(1-e-Σi) Oscar Winner SF/Fantasy • Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/

  16. Restricted Boltzman Machine hjpj=1/(1-e-Σj) Learning = ~ Gradient Descent = Constrastive Divergence Reconstruct visible units Σj=wijvi j voorho i Σi=wijhj vipi=1/(1-e-Σi) Oscar Winner SF/Fantasy • Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/

  17. Restricted Boltzman Machine hjpj=1/(1-e-Σj) Learning = ~ Gradient Descent = Constrastive Divergence Reupdatehidden units Σj=wijvi j N=N+vihj voorho i Σi=wijhj vipi=1/(1-e-Σi) Oscar Winner SF/Fantasy • Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/

  18. Restricted Boltzman Machine hjpj=1/(1-e-Σj) Learning = ~ Gradient Descent = Constrastive Divergence Update weights Σj=wijvi j wij=wij+ηΔwij voorho Δwij=<P>-<N> Σi=wijhj i vipi=1/(1-e-Σi) Oscar Winner SF/Fantasy • Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/

  19. Restricted Boltzman Machine • RBM Overview: • http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ • Wikipedia on DLA and RBM: • http://en.wikipedia.org/wiki/Deep_learning • RBM Details and Code: • http://www.deeplearning.net/tutorial/rbm.html

  20. Restricted Boltzman Machine Geoff Hinton on RBMs: • RBMs and Constrastive Divergence Algorithm • http://www.youtube.com/watch?v=fJjkHAuW0Yk • An example of RBM Learning • http://www.youtube.com/watch?v=Ivj7jymShN0 • RBMs applied to Collaborative Filtering • http://www.youtube.com/watch?v=laVC6WFIXjg

  21. Additional References • Courseracourse – Neural Networks fro Machine Learning: • https://class.coursera.org/neuralnets-2012-001/lecture • ML: Hottest Tech Trend in next 3-5 Years • http://www.youtube.com/watch?v=b4zr9Zx5WiE

More Related