- 112 Views
- Uploaded on
- Presentation posted in: General

Derivation of a Learning Rule for Perceptrons

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Neural Networks

Single Layer Perceptrons

x1

wk1

x2

wk2

.

.

.

wkm

xm

Adaline

(Adaptive Linear Element)

Widrow [1962]

Goal:

Neural Networks

Single Layer Perceptrons

- The following cost function (error function) should be minimized:

i: index of data set, the ith data set

j : index of input, the jth input

Neural Networks

Single Layer Perceptrons

- With

then

- As already obtained before,

Weight Modification Rule

- Defining

we can write

Neural Networks

Single Layer Perceptrons

- Batch Learning Mode

- Incremental Learning Mode

Neural Networks

Single Layer Perceptrons

x1

wk1

x2

wk2

.

.

.

wkm

xm

Goal:

Neural Networks

Single Layer Perceptrons

x1

wk1

x2

wk2

.

.

.

wkm

xm

Goal:

Neural Networks

Single Layer Perceptrons

- For arbitrary activation function,

Neural Networks

Single Layer Perceptrons

Depends on the activation function used

Neural Networks

Single Layer Perceptrons

Linear function

Tangent sigmoid

function

Logarithmic sigmoid

function

Neural Networks

Single Layer Perceptrons

Neural Networks

Single Layer Perceptrons

x1

w11

x2

w12

Given a neuron with linear activation function (a=0.5), write an m-file that will calculate the weights w11 and w12 so that the input [x1;x2] can match output y1 the best.

- Use initial values w11=1 and w12=1.5, and η= 0.01.
- Determine the required number of iterations.
- Note: Submit the m-file in hardcopy and softcopy.

[x1;x2]=[2;3]

[x1;x2]=[[2 1];[3 1]]

Case 2

Case 1

[y1]=[5 2]

[y1]=[5]

- Odd-numbered Student ID

- Even-numbered Student ID

Neural Networks

Single Layer Perceptrons

x1

w11

x2

w12

Given a neuron with a certain activation function, write an m-file that will calculate the weights w11 and w12 so that the input [x1;x2] can match output y1 the best.

- Use initial values w11=0.5 and w12=–0.5, and η= 0.01.
- Determine the required number of iterations.
- Note: Submit the m-file in hardcopy and softcopy.

[x1]=[0.2 0.5 0.4]

[x2]=[0.5 0.8 0.3]

[y1]=[0.1 0.7 0.9]

?

- Even Student ID:Linear function

- Odd Student ID:Logarithmic sigmoid function

Neural Networks

Multi Layer Perceptrons

x1

x2

x3

wlk

wji

wkj

Hidden layers

Input

layer

Output

layer

y1

Outputs

Inputs

y2

- Possessessigmoid activation functionsin the neurons to enable modeling of nonlinearity.
- Contains one or more “hidden layers”.
- Trained using the “Backpropagation” algorithm.

Neural Networks

Multi Layer Perceptrons

- What activation functions should be used?
- How many inputs does the network need?
- How many hidden layers does the network need?
- How many hidden neurons per hidden layer?
- How many outputs should the network have?

- There is no standard methodology to determine these values. Even there is some heuristic points, final values are determinate by a trial and error procedure.

Neural Networks

Multi Layer Perceptrons

x1

x2

x3

wlk

wji

wkj

- MLP with one hidden layer is a universal approximator.
- MLP can approximate any function within any preset accuracy
- The conditions: the weights and the biases are appropriately assigned through the use of adequate learning algorithm.

- MLP can be applied directly in identification and control of dynamic system with nonlinear relationship between input and output.
- MLP delivers the best compromise between number of parameters, structure complexity, and calculation cost.

Neural Networks

Multi Layer Perceptrons

f(.)

f(.)

f(.)

Function signal

Error signal

- Computations at each neuron j:
- Neuron output, yj
- Vector of error gradient, ¶E/¶wji

Forward propagation

“Backpropagation

Learning Algorithm”

Backward propagation

Neural Networks

Multi Layer Perceptrons

If node j is an output node,

dj(n)

yj(n)

netj(n)

wji(n)

ej(n)

yi(n)

-1

f(.)

Neural Networks

Multi Layer Perceptrons

If node j is a hidden node,

dk(n)

netk(n)

yj(n)

yk(n)

netj(n)

wji(n)

wkj(n)

yi(n)

ek(n)

f(.)

f(.)

-1

Neural Networks

Multi Layer Perceptrons

k

j

i

Right

Left

k

j

i

Right

Left

- Forward Pass
- Fix wji(n)
- Compute yj(n)

- Backward Pass
- Calculate dj(n)
- Update weights wji(n+1)