1 / 50

Neural Networks for Optimization

Neural Networks for Optimization. Bill Wolfe California State University Channel Islands. Neural Models. Simple processing units Lots of them Highly interconnected Exchange excitatory and inhibitory signals Variety of connection architectures/strengths

verity
Download Presentation

Neural Networks for Optimization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Neural Networks for Optimization Bill Wolfe California State University Channel Islands

  2. Neural Models • Simple processing units • Lots of them • Highly interconnected • Exchange excitatory and inhibitory signals • Variety of connection architectures/strengths • “Learning”: changes in connection strengths • “Knowledge”: connection architecture • No central processor: distributed processing

  3. Simple Neural Model • aiActivation • ei External input • wij Connection Strength Assume: wij = wji (“symmetric” network) W = (wij) is a symmetric matrix

  4. Net Input Vector Format:

  5. Dynamics • Basic idea:

  6. Energy

  7. Lower Energy • da/dt = net = -grad(E)  seeks lower energy

  8. Problem: Divergence

  9. A Fix: Saturation

  10. Keeps the activation vector inside the hypercube boundaries Encourages convergence to corners

  11. Summary: The Neural Model aiActivation eiExternal Input wijConnection Strength W (wij = wji) Symmetric

  12. Example: Inhibitory Networks • Completely inhibitory • wij = -1 for all i,j • k-winner • Inhibitory Grid • neighborhood inhibition

  13. Traveling Salesman Problem • Classic combinatorial optimization problem • Find the shortest “tour” through n cities • n!/2n distinct tours

  14. TSP solution for 15,000 cities in Germany

  15. TSP50 City Example

  16. Random

  17. Nearest-City

  18. 2-OPT

  19. An Effective Heuristic for the Traveling Salesman Problem S. Lin and B. W. Kernighan Operations Research, 1973 http://www.jstor.org/view/0030364x/ap010105/01a00060/0

  20. Centroid

  21. Monotonic

  22. Neural Network Approach neuron

  23. Tours – Permutation Matrices tour: CDBA permutation matrices correspond to the “feasible” states.

  24. Not Allowed

  25. Only one city per time stopOnly one time stop per cityInhibitory rows and columns inhibitory

  26. Distance Connections: Inhibit the neighboring cities in proportion to their distances.

  27. putting it all together:

  28. Research Questions • Which architecture is best? • Does the network produce: • feasible solutions? • high quality solutions? • optimal solutions? • How do the initial activations affect network performance? • Is the network similar to “nearest city” or any other traditional heuristic? • How does the particular city configuration affect network performance? • Is there a better way to understand the nonlinear dynamics?

  29. typical state of the network before convergence

  30. “Fuzzy Readout”

  31. Initial Phase Fuzzy Tour Neural Activations

  32. Monotonic Phase Fuzzy Tour Neural Activations

  33. Nearest-City Phase Fuzzy Tour Neural Activations

  34. Fuzzy Tour Lengths tour length iteration

  35. Average Results for n=10 to n=70 cities (50 random runs per n) # cities

  36. DEMO 2 Applet by Darrell Long http://hawk.cs.csuci.edu/william.wolfe/TSP001/TSP1.html

  37. Conclusions • Neurons stimulate intriguing computational models. • The models are complex, nonlinear, and difficult to analyze. • The interaction of many simple processing units is difficult to visualize. • The Neural Model for the TSP mimics some of the properties of the nearest-city heuristic. • Much work to be done to understand these models.

  38. EXTRA SLIDES

  39. E = -1/2 { ∑i ∑x ∑j ∑y aix ajy wixjy } = -1/2 { ∑i ∑x ∑y (- d(x,y)) aix ( ai+1y + ai-1y) + ∑i ∑x ∑j (-1/n) aix ajx + ∑i ∑x ∑y (-1/n) aix aiy + ∑i ∑x ∑j ∑y (1/n2) aix ajy }

  40. wixjy = 1/n2 - 1/n y = x, j ≠ i; (row) 1/n2 - 1/n y ≠ x, j = i; (column) 1/n2 - 2/n y = x, j = i; (self) 1/n2 - d(x, y) y ≠ x, j = i +1, or j = i - 1. (distance ) 1/n2 j ≠ i-1, i, i+1, and y ≠ x; (global )

  41. Brain • Approximately 1010 neurons • Neurons are relatively simple • Approximately 104 fan out • No central processor • Neurons communicate via excitatory and inhibitory signals • Learning is associated with modifications of connection strengths between neurons

  42. Fuzzy Tour Lengths tour length iteration

  43. Average Results for n=10 to n=70 cities (50 random runs per n) tour length # cities

  44. with external input e = 1/2

  45. Perfect K-winner Performance: e = k-1/2

More Related