Least-squares-based Multilayer perceptron training with weighted adaptation -- Software simulation project. EE 690 Design of Embodied Intelligence. Outline. Multilayer Perceptron Least-squares based Learning Algorithm Weighted Adaptation in training
Design of Embodied Intelligence
BP has two phases:
Forward pass phase: feedforward propagation of input signals through network
Backward pass phase: propagates the error backwards through network
hidden layerMultilayer Perceptron Learning
We want to know how to modify weights in order to decrease E.
Optimized signalsLeast-squares based Learning Algorithm
z1Least-squares based Learning Algorithm (I)
(1). y2=f -1(z2), scale y1 to (-1, 1).
(2). Based on W2,b2:W2.z1=y2-b2.
(3). y1=f-1(z1), scale y1 to (-1, 1).
(4). Optimize W1, b1 to satisfy W1.x-b1=y1.
(5). Evaluate z1, y1 using the new W1 and bias b1.
(6). Optimize W2, b2 to satisfy W2.z1+b2=y2.
(7). Evaluate z2, y2 using the new W2 and bias b2.
(8). Evaluate the MSE
z1Least-squares based Learning Algorithm (III)
(1). Evaluate z1, y1 using the initial W1 and bias b1.
(2). y2=f -1(d).
(3). Optimize W2, b2 to satisfy W2.z1+b2=y2.
(4). Based on W2,b2, optimize z1 to satisfy
(6). Optimize W1, b1 to satisfy W1.x+b1=y1.
(7). Evaluate y1, z1, y2, z2 using the new W1,W2
and bias b1,b2.
(8). Evaluate the MSE
The location of x on the transfer function determines how much the signal can be changed
Number of layers
affect the training accuracy, determined by users: critical
approximation error component + noise component
Should not be learned
Should be reduced
signal energy/noise energy
Learning should stop – ?
If there is useful signal left unlearned
If noise dominates in the error signal
Noise dominates in the error signal,
Little information left unlearned,
Learning should stop
SNRFe< threshold SNRFWGN
Applied in optimizing number of iterations in back-propagation training to avoid overfitting (overtraining)
1 x N vector: “Values”Software simulation project
Number of training samples: 500
Number of testing samples: 232
z1Software simulation project
J_train (num_layer, num_neuron)
J_test (num_layer, num_neuron)
SNRF (num_layer, num_neuron)
“IEEE Neural Networks Council Standards Committee Working Group on Data modeling Benchmarks”
“Neural Network Databases and Learning Data”
“UCI Machine Learning Repository”