A note about gradient descent:
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

A note about gradient descent: Consider the function f(x)=(x-x 0 ) 2 Its derivative is: PowerPoint PPT Presentation


  • 79 Views
  • Uploaded on
  • Presentation posted in: General

A note about gradient descent: Consider the function f(x)=(x-x 0 ) 2 Its derivative is: By gradient descent. + -. x 0. Solving the differential equation:. or in the general form:. What is the solution of this type of equation:. Try:. THE PERCEPTRON: (Classification).

Download Presentation

A note about gradient descent: Consider the function f(x)=(x-x 0 ) 2 Its derivative is:

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


A note about gradient descent consider the function f x x x 0 2 its derivative is

A note about gradient descent:

Consider the function f(x)=(x-x0)2

Its derivative is:

By gradient descent .

+ -

x0


A note about gradient descent consider the function f x x x 0 2 its derivative is

Solving the differential equation:

or in the general form:

What is the solution of this type of equation:

Try:


A note about gradient descent consider the function f x x x 0 2 its derivative is

THE PERCEPTRON:

(Classification)

Threshold unit:

where is the output for input pattern ,

are the synaptic weights and is the desired output

AND

w1 w2 w3 w4 w5


A note about gradient descent consider the function f x x x 0 2 its derivative is

1

0 1

-1.5

1 1

AND

Linearly seprable


A note about gradient descent consider the function f x x x 0 2 its derivative is

-0.5

1 1

OR

1

0 1

Linearly separable


A note about gradient descent consider the function f x x x 0 2 its derivative is

Perceptron learning rule:

Convergence proof:

Hertz, Krough, Palmer (HKP)

- did you receive the email?

Assignment 3a:

program in matlab a preceptron

with a perceptron learning rule

and solve the OR, AND and XOR problems. (Due before Feb 27)

w1 w2 w3 w4 w5

Show Demo


A note about gradient descent consider the function f x x x 0 2 its derivative is

Summary – what can perceptrons do and how?


A note about gradient descent consider the function f x x x 0 2 its derivative is

Linear single layer network:

( approximation, curve fitting)

*

or

Linear unit:

where is the output for input pattern ,

are the synaptic weights and is the desired output

Minimize mean square error:

w1 w2 w3 w4 w5


A note about gradient descent consider the function f x x x 0 2 its derivative is

Linear single layer network:

( approximation, curve fitting)

Linear unit:

where is the output for input pattern ,

are the synaptic weights and is the desired output

Minimize mean square error:

w1 w2 w3 w4 w5


A note about gradient descent consider the function f x x x 0 2 its derivative is

The best solution is obtained when E is minimal.

For linear neurons there is an exact solution for this called the pseudo-inverse (see HKP).

Looking for a solution by gradient descent:

-gradient

E

w

Chain rule


A note about gradient descent consider the function f x x x 0 2 its derivative is

and

Since:

Error:

Therefore:

Which types of problems can a linear network solve?


A note about gradient descent consider the function f x x x 0 2 its derivative is

Sigmoidal neurons:

for

example:

Which types of problems can a sigmoidal networks solve?

Assignment 3b – Implement a one layer linear and sigmoidal network, fit a 1D a linear, a sigmoid and a quadratic function, for both networks.


A note about gradient descent consider the function f x x x 0 2 its derivative is

Multi layer networks:

Output layer

  • Can solve non linearly separable classification problems.

  • Can approximate any arbitrary function, given ‘enough’ units in the hidden layer.

Hidden layer

Input layer


A note about gradient descent consider the function f x x x 0 2 its derivative is

Note: is not a vector but a matrix


A note about gradient descent consider the function f x x x 0 2 its derivative is

Solving linearly inseparable problems

XOR

Hint: XOR = or and not and


A note about gradient descent consider the function f x x x 0 2 its derivative is

XOR

-.5

1 0.5

.5

0

0.5 -0.5 1 -1

How do we learn a multi-layer network

The credit assignment problem !


A note about gradient descent consider the function f x x x 0 2 its derivative is

Gradient descent/ Back Propagation, the solution to the credit assignment problem:

Where:

{

From hidden layer to output weights:


A note about gradient descent consider the function f x x x 0 2 its derivative is

For input to hidden layer:

{

Where:

and

and


A note about gradient descent consider the function f x x x 0 2 its derivative is

and

For input to hidden layer:

Assignment 3c: Program a 2 layer network in matlab, solve the XOR problem. Fit the curve: x(x-1) between 0 and 1, how many hidden units did you need?


A note about gradient descent consider the function f x x x 0 2 its derivative is

  • Formal neural networks can accomplish many tasks, for example:

  • Perform complex classification

  • Learn arbitrary functions

  • Account for associative memory

  • Some applications: Robotics, Character recognition, Speech recognition,

  • Medical diagnostics.

  • This is not Neuroscience, but is motivated loosely by neuroscience and carries important information for neuroscience as well.

  • For example: Memory, learning and some aspects of development are assumed to be based on synaptic plasticity.


A note about gradient descent consider the function f x x x 0 2 its derivative is

What did we learn today?

Is BackProp biologically realistic?


  • Login