Introduction to Artificial Neural Networks

1. Introduction to ArtificialNeural Networks 主講人: 虞台文

2. Content • Fundamental Concepts of ANNs. • Basic Models and Learning Rules • Neuron Models • ANN structures • Learning • Distributed Representations • Conclusions

3. Introduction to ArtificialNeural Networks Fundamental Concepts of ANNs

4. What is ANN? Why ANN? • ANN  Artificial Neural Networks • To simulate human brain behavior • A new generation of information processing system.

5. Applications • Pattern Matching • Pattern Recognition • Associate Memory (Content Addressable Memory) • Function Approximation • Learning • Optimization • Vector Quantization • Data Clustering • . . .

6. Traditional Computers are inefficient at these tasks although their computation speed is faster. Applications • Pattern Matching • Pattern Recognition • Associate Memory (Content Addressable Memory) • Function Approximation • Learning • Optimization • Vector Quantization • Data Clustering • . . .

7. The Configuration of ANNs • An ANN consists of a large number of interconnected processing elements called neurons. • A human brain consists of ~1011 neurons of many different types. • How ANN works? • Collective behavior.

8. The Biologic Neuron

9. The Biologic Neuron 二神經原之神經絲接合部分 軸突 樹狀突

10. The Biologic Neuron Excitatory or Inhibitory

11. x1 wi1 x2  yi i wi2 f (.) a (.) wim xm The Artificial Neuron

12. x1 wi1 x2  yi i wi2 f (.) a (.) wim xm The Artificial Neuron

13. x1 wi1 x2  yi i wi2 f (.) a (.) wim xm wij positive  excitatory negative  inhibitory zero  no connection The Artificial Neuron

14. x1 wi1 x2  yi i wi2 f (.) a (.) wim xm The Artificial Neuron Proposed by McCulloch and Pitts [1943] M-P neurons

15. y x2 w1 x1 + w2 x2 =   x1 w1 w2 x1 x2 What can be done by M-Pneurons? • A hard limiter. • A binary threshold unit. • Hyperspace separation. 0 1

16. What ANNs will be? • ANN  A neurally inspired mathematical model. • Consists a large number of highly interconnected PEs. • Its connections (weights) holds knowledge. • The response of PE depends only on local information. • Its collective behavior demonstrates the computation power. • With learning, recalling and, generalization capability.

17. Three Basic Entities of ANN Models • Models of Neurons or PEs. • Models of synaptic interconnections and structures. • Training or learning rules.

18. Introduction to ArtificialNeural Networks Basic Models and Learning Rules Neuron Models ANN structures Learning

19.  i f (.) a (.) Processing Elements Extensions of M-P neurons What integration functions we may have? What activation functions we may have?

20. M-P neuron  Quadratic Function i Spherical Function f (.) a (.) Polynomial Function Integration Functions

21.  i a f (.) a (.) 1 f Activation Functions M-P neuron: (Step function)

22.  i a f (.) a (.) 1 f 1 Activation Functions Hard Limiter (Threshold function)

23.  i f (.) a (.) a 1 f 1 Activation Functions Ramp function:

24.  i f (.) a (.) Activation Functions Unipolar sigmoid function:

25.  i f (.) a (.) Activation Functions Bipolar sigmoid function:

26. y L1 L3 L1 L2 L3 L2 y x x Example: Activation Surfaces

27. y L1 1=1 3= 4 2=1 L3 L1 L2 L3 1 0 1 1 L2 1 0 y x x Example: Activation Surfaces x1=0 xy+4=0 y1=0

28. y L1 L3 L1 L2 L3 L2 y x x Example: Activation Surfaces 010 Region Code 011 110 111 001 101 100

29. z y L1 L4 L3 L1 L2 L3 L2 y x x Example: Activation Surfaces z=0 z=1

30. z y L1 4=2.5 L4 L3 1 1 1 L1 L2 L3 L2 y x x Example: Activation Surfaces z=0 z=1

31. z L4 L1 L2 L3 y x Example: Activation Surfaces M-P neuron: (Step function)

32. =2 z =3 L4 L1 L2 L3 =5 =10 y x Unipolar sigmoid function: Example: Activation Surfaces

33. Introduction to ArtificialNeural Networks Basic Models and Learning Rules Neuron Models ANN structures Learning

34. ANN Structure (Connections)

35. y1 y2 yn . . . w1m w2m wn1 w21 w22 wn2 w11 wnm w12 x1 x2 xm Single-Layer Feedforward Networks

36. y1 y2 yn . . . Output Layer . . . . . . . . . Input Layer x1 x2 xm Multilayer Feedforward Networks Hidden Layer

37. Pattern Recognition Multilayer Feedforward Networks Where the knowledge from? Classification Output Analysis Learning Input

38. Single Node with Feedback to Itself Feedback Loop

39. y1 y2 yn . . . x1 x2 xm Single-Layer Recurrent Networks

40. y1 y2 y3 . . . . . . x1 x2 x3 Multilayer Recurrent Networks

41. Introduction to ArtificialNeural Networks Basic Models and Learning Rules Neuron Models ANN structures Learning

42. Learning • Consider an ANN with n neurons and each with m adaptive weights. • Weight matrix:

43. How? Learning To “Learn” the weight matrix. • Consider an ANN with n neurons and each with m adaptive weights. • Weight matrix:

44. Learning Rules • Supervised learning • Reinforcement learning • Unsupervised learning

45. Supervised Learning • Learning with a teacher • Learning by examples  Training set

46. y x ANN W Supervised Learning d Error signal Generator

47. Reinforcement Learning • Learning with a critic • Learning by comments

48. y x ANN W Reinforcement Learning Reinforcement Signal Critic signal Generator

49. Unsupervised Learning • Self-organizing • Clustering • Form proper clusters by discovering the similarities and dissimilarities among objects.

50. y x ANN W Unsupervised Learning