Visual Illusions and Neural Networks

1. Visual Illusions Visual Illusions demonstrate how we perceive an “interpreted version” of the incoming light pattern rather that the exact pattern itself. Neural Networks Lecture 3: Models of Neurons and Neural Networks

2. Visual Illusions He we see that the squares A and B from the previous image actually have the same luminance (but in their visual context are interpreted differently). Neural Networks Lecture 3: Models of Neurons and Neural Networks

3. How do NNs and ANNs work? • NNs are able to learn by adapting their connectivity patterns so that the organism improves its behavior in terms of reaching certain (evolutionary) goals. • The strength of a connection, or whether it is excitatory or inhibitory, depends on the state of a receiving neuron’s synapses. • The NN achieves learning by appropriately adapting the states of its synapses. Neural Networks Lecture 3: Models of Neurons and Neural Networks

4. An Artificial Neuron synapses neuron i • x1 x2 Wi,1 Wi,2 … xi … Wi,n xn net input signal output Neural Networks Lecture 3: Models of Neurons and Neural Networks

5. fi(neti(t)) 1 0 neti(t)  The Activation Function • One possible choice is a threshold function: The graph of this function looks like this: Neural Networks Lecture 3: Models of Neurons and Neural Networks

6. Binary Analogy: Threshold Logic Units (TLUs) Example: w1 = 1 • x1 w2 = 1 x2  = 1.5 x1 x2 x3 w3 = -1 x3 TLUs in technical systems are similar to the threshold neuron model, except that TLUs only accept binary inputs (0 or 1). Neural Networks Lecture 3: Models of Neurons and Neural Networks

7. XOR Binary Analogy: Threshold Logic Units (TLUs) Yet another example: • x1 w1 =  = x1  x2 w2 = x2 Impossible! TLUs can only realize linearly separable functions. Neural Networks Lecture 3: Models of Neurons and Neural Networks

8. 1 1 1 0 1 1 x2 x2 0 1 0 1 0 0 0 1 0 1 x1 x1 Linear Separability • A function f:{0, 1}n  {0, 1} is linearly separable if the space of input vectors yielding 1 can be separated from those yielding 0 by a linear surface (hyperplane) in n dimensions. • Examples (two dimensions): linearly separable linearly inseparable Neural Networks Lecture 3: Models of Neurons and Neural Networks

9. Linear Separability • To explain linear separability, let us consider the function f:Rn  {0, 1} with where x1, x2, …, xn represent real numbers. This is the exactly the function that our threshold neurons use to compute their output from their inputs. Neural Networks Lecture 3: Models of Neurons and Neural Networks

10. 3 3 3 x2 x2 x2 2 2 2 1 1 1 -3 -3 -3 -2 -2 -2 -1 -1 -1 1 1 1 2 2 2 3 3 3 x1 x1 x1 -1 -1 -1 -2 -2 -2 -3 -3 -3 Linear Separability Input space in the two-dimensional case (n = 2): 1 1 1 0 0 0 w1 = 1, w2 = 2, = 2 w1 = -2, w2 = 1, = 2 w1 = -2, w2 = 1, = 1 Neural Networks Lecture 3: Models of Neurons and Neural Networks

11. Linear Separability • So by varying the weights and the threshold, we can realize any linear separation of the input space into a region that yields output 1, and another region that yields output 0. • As we have seen, a two-dimensional input space can be divided by any straight line. • A three-dimensional input space can be divided by any two-dimensional plane. • In general, an n-dimensional input space can be divided by an (n-1)-dimensional plane or hyperplane. • Of course, for n > 3 this is hard to visualize. Neural Networks Lecture 3: Models of Neurons and Neural Networks

12. x2 1 0 1 0 1 0 0 1 x1 Linear Separability • Of course, the same applies to our original function f of the TLU using binary input values. • The only difference is the restriction in the input values. • Obviously, we cannot find a straight line to realize the XOR function: In order to realize XOR with TLUs, we need to combine multiple TLUs into a network. Neural Networks Lecture 3: Models of Neurons and Neural Networks

13. Multi-Layered XOR Network 1 • x1 0.5 -1 x2 1 x1  x2 0.5 1 -1 x1 0.5 1 x2 Neural Networks Lecture 3: Models of Neurons and Neural Networks

14. x1 Wi,1 xi Wi,2 x2 Capabilities of Threshold Neurons • What can threshold neurons do for us? • To keep things simple, let us consider such a neuron with two inputs: The computation of this neuron can be described as the inner product of the two-dimensional vectorsx and wi, followed by a threshold operation. Neural Networks Lecture 3: Models of Neurons and Neural Networks

15. The Net Input Signal • The net input signal is the sum of all inputs after passing the synapses: This can be viewed as computing the inner product of the vectors wi and x: where  is the angle between the two vectors. Neural Networks Lecture 3: Models of Neurons and Neural Networks

16. second vector component x wi first vector component Capabilities of Threshold Neurons • Let us assume that the threshold  = 0 and illustrate the function computed by the neuron for sample vectors wiand x: Since the inner product is positive for -90    90, in this example the neuron’s output is 1 for any input vector x to the right of or on the dotted line, and 0 for any other input vector. Neural Networks Lecture 3: Models of Neurons and Neural Networks

17. Capabilities of Threshold Neurons • By choosing appropriate weights wi and threshold  we can place the line dividing the input space into regions of output 0 and output 1in any position and orientation. • Therefore, our threshold neuron can realize any linearly separable function Rn  {0, 1}. • Although we only looked at two-dimensional input, our findings apply to any dimensionality n. • For example, for n = 3, our neuron can realize any function that divides the three-dimensional input space along a two-dimension plane. Neural Networks Lecture 3: Models of Neurons and Neural Networks