Download
1 / 26
280 likes | 450 Views
This document details the fundamental process of backpropagation within a feedforward neural network. It explains the forward pass calculation using sigmoid activation functions, where nodes compute their outputs based on weighted sums of input signals. The document illustrates the backpropagation error calculation for each output neuron and highlights how to update weights connecting hidden layers to outputs and across layers. Key formulas, such as the derivative of the sigmoid and updates for weights during learning, are thoroughly explained, aiding readers in grasping the mechanics of neural network training.
E N D
Notes on Backpropagation Alex Churchill
Feed Forward • Node C = sigmoid(A * weightca + B * weightcb) C D
Feed Forward • Node C = sigmoid(0.1 * 0.1+ 0.7*0.5) C D
Feed Forward • Node C = sigmoid(0.01+0.35) = 0.59 C 0.59 D
Feed Forward • Node D = sigmoid(A * weightda + B * weightdb) C 0.59 D
Feed Forward • Node D = sigmoid(0.1 * 0.3+ 0.7*0.2) C 0.59 D
Feed Forward • Node D = sigmoid(0.03+0.14) = 0.54 C 0.59 E D 0.54
Feed Forward • Node E = sigmoid(C * weightec + D * weighted) C 0.59 E D 0.54
Feed Forward • Node E = sigmoid(0.59*0.2 + 0.54*0.1)=0.542 C 0.59 E D 0.54
Feed Forward • Node E = sigmoid(0.59*0.2 + 0.54*0.1)=0.542 C 0.59 E 0.542 0.542 D 0.54
Backpropagation • Calculate error for each output neuron at the output layer (L) • For each hidden layer (L-1 to L – n) pass the error backwards from the layer above • Update the weights connecting the last hidden layer (L-1) to the output layer (L) • Update the weights connecting each lower layer
Backpropagation • Calculate error (δk)for each output neuron (k) at the output layer (L) This is calculated using: δk=(yk-tk)*g'(xk) Where g’ is the first derivative of the sigmoid and xk is the pre-sigmoided output
Backpropagation δk=(yk-tk)*g'(xk) δk=(0.542-1)*g'(xk)=(0.542-1)*(0.542)*(1-0.542) = -0.114 Target = 1 C Learning rate = 1 0.59 E 0.542 0.542 D 0.54
Backpropagation δk=(yk-tk)*g'(xk) δk=(0.542-1)*g'(xk)=(0.542-1)*(0.542)*(1-0.542) = -0.114 Target = 1 C Learning rate = 1 δk=-0.114 0.59 E 0.542 0.542 D 0.54
Backpropagation 2. For each hidden layer (L-1 to L – n) pass the error backwards from the layer above This is calculated using: Where j is the hidden neuron and k is the output neuron
Backpropagation δc=(wecδk)*g'(xc) = 0.2 * -0.114 * 0.59*(1-0.59)=-0.0055 δd=(wedδk)*g'(xd) = 0.1 * -0.114 * 0.54*(1-0.54)=-0.0028 Target = 1 C Learning rate = 1 δk=-0.114 0.59 E 0.542 0.542 D 0.54
Backpropagation δc=-0.0055 δc=(wecδk)*g'(xc) = 0.2 * -0.114 * 0.59*(1-0.59)=-0.0055 δd=(wedδk)*g'(xd) = 0.1 * -0.114 * 0.54*(1-0.54)=-0.0028 Target = 1 C Learning rate = 1 δk=-0.114 0.59 E δd=-0.0028 0.542 0.542 D 0.54
Backpropagation 3. Update the weights connecting the last hidden layer (L-1) to the output layer (L) This is calculated using: Where j is the hidden neuron and k is the output neuron. aj is the sigmoided output of the hidden neuron
Backpropagation δc=-0.0055 Wec=wec-ηδeC = 0.2 - 1* -0.114*0.59=0.267 Wed=wed -ηδed = 0.1 - 1* -0.114*0.54=0.162 Target = 1 C Learning rate = 1 δe=-0.114 0.59 E δd=-0.0028 0.542 0.542 D 0.54
Backpropagation δc=-0.0055 Wec=wec-ηδeC = 0.2 - 1* -0.114*0.59=0.267 Wed=wed -ηδed = 0.1 - 1* -0.114*0.54=0.162 Target = 1 C Learning rate = 1 0.267 0.59 E δd=-0.0028 0.542 0.542 D 0.162 0.54
Backpropagation 4. Update the weights connecting each lower layer. This is calculated using: Where j is the hidden neuron (or input neuron) in the layer below and k is the hidden neuron in the layer above.
Backpropagation δc=-0.0055 Wca=wca-ηδcA = 0.1 - 1* -0.0055*0.1=0.1005 Wcd=wcd-ηδdA = 0.3 - 1* -0.0028*0.1=0.3003 Target = 1 C Learning rate = 1 0.267 0.59 E δd=-0.0028 0.542 0.542 D 0.162 0.54
Backpropagation δc=-0.0055 Wca=wca-ηδcA = 0.1 - 1* -0.0055*0.1=0.1005 Wcd=wcd-ηδdA = 0.3 - 1* -0.0028*0.1=0.3003 Target = 1 C 0.1005 Learning rate = 1 0.267 0.59 0.3003 E δd=-0.0028 0.542 0.542 D 0.162 0.54
