Evolution and Insights of Artificial Neural Networks

1. Ups and Downs of NN (= ANN + DNN) Study Ch1: Introduction • 1958: Perceptron (linear model) • 1969: Perceptron has limitation • 1980s: Multi-layer perceptron • Do not have significant difference from DNN today • 1986: Backpropagation • Usually more than 3 hidden layers is not helpful • 1989: 1 hidden layer is “good enough”, why deep? • 2006: RBM initialization (breakthrough) • 2009: GPU • 2011: Start to be popular in speech recognition • 2012: win ILSVRC image competition

2. 1.1 Introduction to ANN Technology Human brain is the most complex computing device that we have ever known. • Conventional computers are good in scientific and mathematical computations, creation and manipulation of databases, control functions, graphics, word processing, • How do computers learn, analyze, organize, adapt, comprehend, associate, recognize, plan, decide

3. 。 Many problems are not suitable to be solved by sequential machines and algorithms Parallel-processing architectures may be good tools for solving difficult-to-solve, or unsolved problems, e.g., perceptual-organizing problem visual pattern recognition subjective inference

4. Example 1: Perceptual-Organization Problem

6. Perceptual organization is well processed in a parallel manner.

7. Example 2: Visual Pattern Recognition

8. Example 3:Subjective inference (reasoning)

9. 1.2 Neurophysiology ◎ Three major components constructing the human nervous system: brain, spinal cord, periphery

10. Periphery 10

11. Spinal Nerves

12. Brain Cerebral cortex Size: 1 Thick: 2-4 mm Layers: 6

13. ◎ Single-Neurons Physiology Three types of neurons: 1. unipolar 2. bipolar 3. multipolar Terminal 末梢神經 感覺器官 Pathway 連絡神經 脊髓 Central 中樞神經 腦

14. ◎ Structural Components The input impulses can be excitatory or inhibitory and are summed at the axon hillock. If the summed potential is sufficient, an action potential is generated.

15. ◎ Synaptic Function Neurotransmitters are held in vesicles and are released near presynaptic membrane into synaptic cleft and absorbed by postsynaptic membrane.

16. Signal : frequency of pulses Learning: adjusting synaptic gaps Memory : strength of synaptic connections Knowledge is acquired through a learning process. Acquired knowledge is stored in interneuron links (i.e., synaptic connections) in terms of strengths. Comparison between nervous and computer systems

17. 1.3 Artificial Neural Networks • A collection of parallel processing units (neurons) connected together • Various neural systems with different functions and characteristics result from i) Functions of neurons ii) Ways of connections iii) Flows of information

18. Characteristics: nonlinearity, non-locality, non-algorithm, dynamics, adaptivity, fault-tolerance, input-output mapping, evidential response, self-organization, • Functions: learn, analyze, organize, comprehend, associate, recognize, plan, decide

19. ◎Neuron Model Bias term b: feedback, error, gain, adjustment

20. Types of activation function : 1. Threshold function (Heaviside function) 2. Sigmoid function Logistic function Hyperbolic function

21. 3. Softsign function 4. Softplus function 5. ReLU function 21

22. The following NM, i.e., perceptron, can differentiate the patterns that are linearly separable. : inputs, where : weights. Separating hyperplane:

23. XOR problem This problem cannot be solved by a pereceptron. Reason: Let which is a line in the plane.

24. One solution: The hidden layer provides two lines that can separate the plane into three regions. The two regions containing (0,0) and (1,1) are associated with a network output of 0. The central region is associated with a network output of 1.