neural networks
Download
Skip this Video
Download Presentation
Neural Networks

Loading in 2 Seconds...

play fullscreen
1 / 23

Neural Networks - PowerPoint PPT Presentation


  • 239 Views
  • Uploaded on

Neural Networks. Marcel Jiřina. Introduction. Neural networks and their use to classification and other tasks ICS AS CR Theoretical computer science Neural networks , genetic alg. and n onlinear methods Numeric algorithms .. 1 mil. eq. Fuzzy sets, approximate reasoning, possibility th.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Neural Networks' - ashlyn


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
neural networks

Neural Networks

Marcel Jiřina

Institute of Computer Science, Prague

introduction
Introduction
  • Neural networks and their use to classification and other tasks
  • ICS AS CR
      • Theoretical computer science
      • Neural networks, genetic alg. and nonlinear methods
      • Numeric algorithms ..1 mil. eq.
      • Fuzzy sets, approximate reasoning, possibility th.
      • Applications: Nuclear science, Ecology, Meteorology, Reliability in machinery, Medical informatics …

Institute of Computer Science, Prague

structure of talk
Structure of talk
  • NN classification
  • Some theory
  • Interesting paradigms
  • NN and statistics
  • NN and optimization and genetic algorithms
  • About application of NN
  • Conlusions

Institute of Computer Science, Prague

nn classification

Approximators

Associative memories

General

Predictors

Auto-associative

Hetero-associative

Classifiers

Teacher

MLP-BP

RBF

GMDH

NNSU

Marks

Klán

Hopfield

Perceptron(*)

Hamming

No teacher

Kohonen

CarpentierGrossberg

(SOM)

NE

Kohonen

(NE)

Signals

Continuous, real-valued

Binary, multi-valued (continuous)

NN classification

NE – not existing. Associated response can be arbitrary and then must be given - by teacher

Feed-forward, recurrent

Fixed structure - growing

Institute of Computer Science, Prague

some theory
Some theory

Kolmogorov theorem

Kůrková – Theorem

Sigmoid transfer function 

Institute of Computer Science, Prague

mlp bp
MLP - BP

Three layer - Single hidden layer MLP – 4 layer – 2 hidden

Other paradigms have its own theory – another

Institute of Computer Science, Prague

interesting paradigms
Interesting paradigms

Paradigm – general notion on structure, functions and algorithms of NN

  • MLP - BP
  • RBF
  • GMDH
  • NNSU

All: approximators

Approximator + thresholding = Classifier

Institute of Computer Science, Prague

mlp bp1
MLP - BP

MLP – error Back Propagation

coefficients , (0,1)

- Lavenberg-Marquart

- Optimization tools

MLP with jump transfer function

- Optimization

Feed – forward (in recall)

Matlab, NeuralWorks, …

Good when default is sufficient or when network is well tuned: Layers, neurons, , 

Institute of Computer Science, Prague

slide9
RBF
  • Structure same as in MLP
  • Bell-shaped transfer function (Gauss)
    • Number and positions of centers: random – cluster analysis
    • “broadness” of that bell
    • Size of individual bells
    • Learning methods
  • Theory similar to MLP
  • Matlab, NeuralWorks, …

Good when default is sufficient or when network is well tuned : Layers mostly one hidden, # neurons, transfer function, proper cluster analysis (fixed No. of clusters, variable? Near – Far metric or criteria)

Institute of Computer Science, Prague

gmdh 1 5
GMDH 1 (…5)

Group Method Data Handling

          • Group – initially a pair of signals only
  • “per partes” or successive polynomial approximator
  • Growing network
  • “parameterless” – parameter-barren
          • No. of new neurons in each layer only (processing time)
          • (output limits, stopping rule parameters)
  • Overtraining – learning set is split to
          • Adjusting set
          • Evaluation set

GMDH 2-5: neuron, growing network, learning strategy, variants

Institute of Computer Science, Prague

gmdh 2 neuron
GMDH 2 – neuron
  • Two inputs x1, x2 only
          • True inputs
          • Outputs from neurons of the preceding layer
  • Full second order polynomial

y = a x12 + b x1 x2 + c x22 + d x1 + e x2 + f

y = neuron’s output

  • n inputs => n(n-1)/2 neurons in the first layer
  • Number of neurons grows exponentially
  • Order of resulting polynomial grows exponentially: 2, 4, 8, 16, 32, …
  • Ivakhnenko polynomials … some elements are missing

Institute of Computer Science, Prague

gmdh 3 learning a neuron
GMDH 3 – learning a neuron
  • Matrix of data: inputs and desired value

u1, u2 , u3, …, un,y sample 1

u1, u2 , u3, …, un,y sample 1

…. sample m

  • A pair of two u’s are neuron’s inputs x1, x2
  • m approximating equations, one for each sample

a x12 + b x1 x2 + c x22 + d x1 + e x2 + f = y

  • Matrix X = Y= (a, b, c, d, e, f)t
      • Each row of X is x12+x1x2+x22+x1+x2+1
  • LMS solution  = (XtX)-1XtY
  • If XtX is singular, we omit this neuron

Institute of Computer Science, Prague

gmdh 4 growing network
GMDH 4 - growing network

x1, x2 y = desired output

Institute of Computer Science, Prague

gmdh 5 learn strategy
GMDH 5 learn. strategy

Problem: Number of neurons grows exponentially

NN=n(n-1)2

  • Let the first layer of neurons grow unlimited
  • In next rows:
    • [learning set split to adjusting set and evaluating set]
    • Compute parameters a,…f using adjusting set
    • Evaluate error using evaluating set and sort
    • Select some n best neurons and delete the others
    • Build the next layer OR
    • Stop learning if stopping condition is met.

Institute of Computer Science, Prague

gmdh 6 learn strategy 2
GMDH 6 learn. Strategy 2

Select some n best neurons and delete the others

Control parameter of GMDH network

Institute of Computer Science, Prague

gmdh 7 variants
GMDH 7 - variants
  • Basic – full quadratic polynomial – Ivakh. poly
  • Cubic, Fourth order simplified …
    • Reach higher order in less layers and less params
  • Different stopping rules
  • Different ratio of sizes of adjusting set and evaluating set

Institute of Computer Science, Prague

nnsu ga
NNSU GA

Neural Network with Switching Units

learned by the use of Genetic Algorithm

  • Approximator by lot of local hyper-planes; today also by local more general hyper-surfaces
  • Feed-forward network
  • Originally derived from MLP for optical implementation
  • Structure looks like columns above individual inputs
  • More … František

Institute of Computer Science, Prague

learning and testing set
Learning and testing set
  • Learning set
    • Adjusting (tuning) set
    • Evaluation set
  • Testing set

One data set – the splitting influences results

  • Fair evaluation problem

Institute of Computer Science, Prague

nn and statistics
NN and statistics
  • MLP-BP mean squared error minimization
    • Sum of errors squared … MSE criterion
    • Hamming distance for (pure) classifiers
  • No other statistical criteria or tests are in NN:
    • NN transforms data, generates mapping
    • statistical criteria or tests are outside NN (2, K-S, C-vM,…)

Is NN good for K-S test? … is y=sin(x) good for 2 test?

  • Bayes classifiers, k-th nearest neighbor, kernel methods …

Institute of Computer Science, Prague

nn and optimization and genetic algorithms
NN and optimization and genetic algorithms

Learning is an optimization procedure

  • Specific to given NN
  • General optimization systems or methods
  • Whole NN
  • Parts – GMDH and NNSU - linear regression
  • Genetic algorithm
    • Not only parameters, the structure, too
    • May be faster than iterations

Institute of Computer Science, Prague

about application of nn
About application of NN
  • Soft problems
    • Nonlinear
    • Lot of noise
    • Problematic variables
    • Mutual dependence of variables
  • Application areas
    • Economy
    • Pattern recognition
    • Robotics
    • Particle physics

Institute of Computer Science, Prague

strategy when using nn
Strategy when using NN
  • For “soft problems” only
  • NOT for
      • Exact function generation
      • periodic signals etc.
  • First subtract all “systematics”
      • Nearly noise remains
      • Approximate this nearly noise
      • Add back all systematics
  • Understand your paradigm
      • Tune it patiently or
      • Use “parameterless” paradigm

Institute of Computer Science, Prague

conlusions
Conlusions
  • Powerfull tool
    • Good when well used
    • Simple paradigm, complex behavior
  • Special tool
    • Approximator
    • Classifier
  • Universal tool
    • Very different problems
    • Soft problems

Institute of Computer Science, Prague

ad