- By
**kin** - Follow User

- 105 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about ' Introduction to Neural Networks' - kin

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

### Introduction to Neural Networks

BRNN

Gianluca Pollastri, Head of Lab

School of Computer Science and Informatics and

Complex and Adaptive Systems Labs

University College Dublin

Credits

- Geoffrey Hinton, University of Toronto.
- borrowed some of his slides for “Neural Networks” and “Computation in Neural Networks” courses.
- Paolo Frasconi, University of Florence.
- This guy taught me Neural Networks in the first place (*and* I borrowed some of his slides too!).

Recurrent Neural Networks (RNN)

- One of the earliest versions: Jeffrey Elman, 1990, Cognitive Science.
- Problem: it isn’t easy to represent time with Feedforward Neural Nets: usually time is represented with space.
- Attempt to design networks with memory.

RNNs

- The idea is having discrete time steps, and considering the hidden layer at time t-1 as an input at time t.
- This effectively removes cycles: we can model the network using an FFNN, and model memory explicitly.

BPTT

- BackPropagation Through Time.
- If Ot is the output at time t, It the input at time t, and Xt the memory (hidden) at time t, we can model the dependencies as follows:

BPTT

- We can model both f() and g() with (possibly multilayered) networks.
- We can transform the recurrent network by unrolling it in time.
- Backpropagation works on any DAG. An RNN becomes one once it’s unrolled.

gradient in BPTT

- GRADIENT(I,O,T) {
- # I=inputs, O=outputs, T=targets
- T := size(O);
- X0 := 0;
- for t := 1..T
- Xt := f( Xt-1 , It );
- for t := 1..T {
- Ot := g( Xt , It );
- g.gradient( Ot - Tt );
- δt = g.deltas( Ot - Tt );
- }
- for t := T..1
- f.gradient(δt );
- δt-1 += f.deltas(δt );
- }

What I will talk about

- Neurons
- Multi-Layered Neural Networks:
- Basic learning algorithm
- Expressive power
- Classification
- How can we *actually* train Neural Networks:
- Speeding up training
- Learning just right (not too little, not too much)
- Figuring out you got it right
- Feed-back networks?
- Anecdotes on real feed-back networks (Hopfield Nets, Boltzmann Machines)
- Recurrent Neural Networks
- Bidirectional RNN
- 2D-RNN
- Concluding remarks

BRNN

Ft = ( Ft-1 , Ut )

Bt = ( Bt+1 , Ut )

Yt = ( Ft , Bt , Ut )

- () () ed () are realised with NN
- (), () and () are independent from t: stationary

BRNN

Ft = ( Ft-1 , Ut )

Bt = ( Bt+1 , Ut )

Yt = ( Ft , Bt , Ut )

- () () ed () are realised with NN
- (), () and () are independent from t: stationary

BRNN

Ft = ( Ft-1 , Ut )

Bt = ( Bt+1 , Ut )

Yt = ( Ft , Bt , Ut )

- () () ed () are realised with NN
- (), () and () are independent from t: stationary

Ft = ( Ft-1 , Ut )

Bt = ( Bt+1 , Ut )

Yt = ( Ft , Bt , Ut )

- () () ed () are realised with NN
- (), () and () are independent from t: stationary

Inference in BRNNs

- FORWARD(U) {
- T size(U);
- F0 BT+1 0;
- for t 1..T
- Ft = ( Ft-1 , Ut );
- for t T..1
- Bt = ( Bt+1 , Ut );
- for t 1..T
- Yt = ( Ft , Bt , Ut );
- return Y;
- }

GRADIENT(U,Y) {

T size(U);

F0 BT+1 0;

for t 1..T

Ft = ( Ft-1 , Ut );

for t T..1

Bt = ( Bt+1 , Ut );

for t 1..T {

Yt = ( Ft , Bt , Ut );

[δFt, δBt] = .backprop&gradient( Yt - Yt );

}

for t T..1

δFt-1 += .backprop&gradient(δFt );

for t 1..T

δBt+1 += .backprop&gradient(δBt );

}

Learning in BRNNsWhat I will talk about

- Neurons
- Multi-Layered Neural Networks:
- Basic learning algorithm
- Expressive power
- Classification
- How can we *actually* train Neural Networks:
- Speeding up training
- Learning just right (not too little, not too much)
- Figuring out you got it right
- Feed-back networks?
- Anecdotes on real feed-back networks (Hopfield Nets, Boltzmann Machines)
- Recurrent Neural Networks
- Bidirectional RNN
- 2D-RNN
- Concluding remarks

Download Presentation

Connecting to Server..