Architecture and Equilibria 结构和平衡

1. 神经网络与模糊系统 Architecture and Equilibria结构和平衡 Chapter 6 学生： 李 琦 导师：高新波

2. 6.1 Neutral Network As Stochastic Gradient system Classify Neutral network model By their synaptic connection topologies and by how learning modifies their connection topologies synaptic connection topologies how learning modifies their connection topologies

3. 6.1 Neutral Network As Stochastic Gradient system

4. 6.2 Global Equilibria: convergence and stability Three dynamical systems in neural network: synaptic dynamical system neuronal dynamical system joint neuronal-synaptic dynamical system Historically,Neural engineers study the first or second neural network.They usually study learning infeedforward neural networks and neural stabilityinnonadaptive feedbackneural networks. RABAM and ART network depend on joint equilibration of the synaptic and neuronal dynamical systems.

5. 6.2 Global Equilibria: convergence and stability Equilibrium is steady state (for fixed-point attractors) Convergence is synaptic equilibrium. Stability is neuronal equilibrium. More generally neural signals reach steady state even though the activations still change. We denote steady state in the neuronal field : Global stability: Stability - Equilibrium dilemma: Neurons fluctuate faster than synapses fluctuate. Convergence undermines stability.

6. 6.3 Synaptic convergence to centroids: AVQ Algorithms Competitive learning adaptively quantizes the input pattern space . Probability density function characterizes the continuous distributions of patterns in . We shall prove that competitive AVQ synaptic vector converge exponentially quickly to pattern-class centroids and, more generally, at equilibrium they vibrate about the centroids in a Browmian motion.

7. 6.3 Synaptic convergence to centroids: AVQ Algorithms Competitive AVQ Stochastic Differential Equations: The Random Indicator function Supervised learning algorithms depend explicitly on the indicator functions.Unsupervised learning algorithms don’t require this pattern-class information. Centriod of :

8. 6.3 Synaptic convergence to centroids: AVQ Algorithms The Stochastic unsupervised competitive learning law: We want to show that at equilibrium As discussed in Chapter 4: The linear stochastic competitive learning law: The linear supervised competitive learning law:

9. 6.3 Synaptic convergence to centroids: AVQ Algorithms The linear differential competitive learning law: In practice:

10. 6.3 Synaptic convergence to centroids: AVQ Algorithms Competitive AVQ Algorithms 1. Initialize synaptic vectors: 2.For random sample , find the closest (“winning”) synaptic vector : gives the squared Euclidean norm of x 3.Update the wining synaptic vectors by the UCL ,SCL,or DCL learning algorithm.

11. 6.3 Synaptic convergence to centroids: AVQ Algorithms Unsupervised Competitive Learning (UCL) defines a slowly decreasing sequence of learning coefficient Supervised Competitive Learning (SCL)

12. 6.3 Synaptic convergence to centroids: AVQ Algorithms Differential Competitive Learning (DCL) denotes the time change of the jth neuron’s competitive signal In practice we often use only the sign of the signal difference or , the sign of the activation difference.

14. 基于UCL的AVQ算法 T=10 T=20 T=30 T=40 T=100

15. 6.3 Synaptic convergence to centroids: AVQ Algorithms Stochastic Equilibrium and Convergence Competitive synaptic vector converge to decision-class centroids. The centroids may correspond to local maxima of the sampled but unknown probability density function

16. 6.3 Synaptic convergence to centroids: AVQ Algorithms AVQ centroid theorem: If a competitive AVQ system converges, it converges to the centroid of the sampled decision class. Proof. Suppose the jth neuron in wins the competition. Suppose the jth synaptic vector codes for decision class So Suppose the synaptic vector has reached equilibrium:

17. 6.3 Synaptic convergence to centroids: AVQ Algorithms In general the AVQ centroid theorem concludes that at equilibrium:

18. 6.3 Synaptic convergence to centroids: AVQ Algorithms • Arguments: • The AVQ centriod theorem applies to the stochastic SCL and DCL law. • The spatial and temporal integrals are approximate equal. • The AVQ centriod theorem assumes that stochastic convergence occurs.

19. 6.4 AVQ Convergence Theorem AVQ Convergence Theorem: Competitive synaptic vectors converge exponentially quickly to pattern-class centroids. Proof.Consider the random quadratic form L: Note: The pattern vectors x do not change in time.

20. 6.4 AVQ Convergence Theorem L equals a random variable at every time t. E[L] equals a deterministic number at every t. So we use the average E[L] as Lyapunov function for the stochastic competitive dynamical system.

21. 6.4 AVQ Convergence Theorem Assume: sufficient smoothness to interchange the time derivative and the probabilistic integral—to bring the time derivative “inside” the integral. So, the competitive AVQ system is asymptotically stable, and in general converges exponentially quickly to a locally equilibrium. Suppose .Then every synaptic vector has reached equilibrium and is constant (with probability one) if holds.

22. 6.4 AVQ Convergence Theorem Since p(x) is a nonnegative weight function, the weighted integral of the learning differences must equal zero : So, with probability one, equilibrium synaptic vector equal centroids. More generally, averageequilibrium synaptic vector are centroids:

23. 6.4 AVQ Convergence Theorem Arguments: The vector integral in equals the gradient of with respect to . So the AVQ convergence theorem implies that the class centroids-and, asymptotically ,competitive synaptic vector-minimize the mean-squared error of vector quantization.

24. 6.5 Global stability of feedback neural networks • Global stability is jointly neuronal-synaptic steady state. • Global stability theorems are powerful but limited. • Their power: • their dimension independence • nonlinear generality • their exponentially fast convergence to fixed points. • Their limitation: • not tell us where the equilibria occur in the state space.

25. 6.5 Global stability of feedback neural networks Stability-Convergence Dilemma Stability-Convergence Dilemma arises from the asymmetry in neuronal and synaptic fluctuation rates. Neurons change faster than synapses change. Neurons fluctuate at the millisecond level. Synapses fluctuate at the second or even minute level. The fast-changing neurons must balance the slow-changing synapses.

26. 6.5 Global stability of feedback neural networks Stability-Convergence Dilemma 1.Asymmetry:Neurons in and fluctuate faster than the synapses in M. 2.stability: (pattern formation). 3.Learning: 4.Undoing: The ABAM theorem offers a general solution to stability-convergence dilemma.

27. 6.6 The ABAM Theorem Hebbian ABAM model: Competitive ABAM model (CABAM): Differential Hebbian ABAM model: Differential competitive ABAM model:

28. 6.6 The ABAM Theorem The ABAM Theorem: The Hebbian ABAM and competitive ABAM models are globally stable. We define the dynamical systems as above. If the positivity assumptions hold, then the models are asymptotically stable, and the squared activation and synaptic velocities decrease exponentially quickly to their equilibrium values:

29. 6.6 The ABAM Theorem Proof.The proof uses the bounded lyapunov functionL: This proves global stability for signal Hebbian ABAMs.

30. 6.6 The ABAM Theorem for the competitive learning law: We assume that behaves approximately as a zero-one threshold. This proves global stability for the competitive ABAM system.

31. 6.6 The ABAM Theorem Also for signal Hebbian learning: along trajectories for any nonzero change in any neuronal activation or any synapse. This proves asymptotic global stability. (Higher-Order ABAMs, Adaptive Resonance ABAMs, Differential Hebbian ABAMs)

32. 6.7 structural stability of unsupervised learning and RABAM • Structural stability is insensitivity to small perturbations. • Structural stability allows us to perturb globally stable feedback systems without changing their qualitative equilibrium behavior. • Structural stability differs from the global stability, or convergence to fixed points. • Structural stability ignores many small perturbations. Such perturbations preserve qualitative properties.

33. 6.7 structural stability of unsupervised learning and RABAM Random Adaptive Bidirectional Associative Memories RABAM Brownian diffusions perturb RABAM models. Suppose denote Brownian-motion (independent Gaussian increment) processes that perturb state changes in the ith neuron in ,the jth neuron in ,and the synapse ,respectively. The signal Hebbiandiffusion RABAM model:

34. 6.7 structural stability of unsupervised learning and RABAM With the stochastic competitives law: (Differential Hebbian, differential competitive diffusion laws) The signal-Hebbian noise RABAM model:

35. 6.7 structural stability of unsupervised learning and RABAM The RABAM theorem ensures stochastic stability. In effect, RABAM equilibria are ABAM equilibria that randomly vibrate. The noise variances control the range of vibration. Average RABAM behavior equals ABAM behavior. RABAM Theorem. The RABAM model above is global stable. If signal functions are strictly increasing and amplification functions and are strictly positive, the RABAM model is asymptotically stable.

36. 6.7 structural stability of unsupervised learning and RABAM Proof. The ABAM lyapunov function L : defines a random process. At each time t, L(t) is a random variable. The expected ABAM lyapunov function E(L) is a lyapunov function for the RABAM system.

37. 6.7 structural stability of unsupervised learning and RABAM

38. 6.7 structural stability of unsupervised learning and RABAM Noise-Saturation Dilemma: How neurons can have an effective infinite dynamical range when they operate between upper and lower bounds and yet not treat small input signals as noise: If the are sensitive to large inputs, then why do not small inputs get lost in internal system noise? If the are sensitive to small inputs, then why do they not all saturate at their maximum values in response to large inputs?

39. 6.7 structural stability of unsupervised learning and RABAM RABAM Noise Suppression Theorem: As the above RABAM dynamical systems converge exponentially quickly, the mean-squared velocities of neuronal activations and synapses decrease to their lower bounds exponentially quickly: Guarantee: no noise processes can destabilize a RABAM if the noise processes have finite instantaneous variances (and zero mean). (Unbiasedness Corollary, RABAM Annealing Theorem)

40. Thank you!