Some slide material taken from: SAS Education

2. Objectives Understand the structure of a Neural Network Understand how the objective function leads to overgeneralization Understand how overgeneralization is counteracted by taking into account the validation data

3. Neural network models (multi-layer perceptrons) Often regarded as a mysterious and powerful predictive modeling technique. The most typical form of the model is, in fact, a natural extension of a regression model: A generalized linear model on a set of derived inputs These derived inputs are themselves a generalized linear model on the original inputs The usual link for the derived input�s model is inverse hyperbolic tangent, a shift and rescaling of the logit function Ability to approximate virtually any continuous association between the inputs and the target You simply need to specify the correct number of derived inputs

4. Neural Network Model

5. Input layer, hidden layer, output layer Multi-layer perceptron models were originally inspired by neurophysiology and the interconnections between neurons. The basic model form arranges neurons in layers. The input layer connects to a layer of neurons called a hidden layer, which, in turn, connects to a final layer called the target, or output, layer. The structure of a multi-layer perceptron lends itself to a graphical representation called a network diagram.

6. Neural Network Diagram

7. NNs as a Universal Approximator

8. An example

9. Objective Function

10. Neural Network Training

11. Neural Network Training

12. Convergence Training concludes when small changes in the parameter values no longer decrease the value of the objective function. The network is said to have reached a local minimum in the objective.

13. Neural Network Training Convergence

14. Overgeneralization A small value for the objective function, when calculated on training data, need not imply a small value for the function on validation data. Typically, improvement on the objective function is observed on both the training and the validation data over the first few iterations of the training process. At convergence, however, the model is likely to be highly overgeneralized and the values of the objective function computed on training and validation data may be quite different.

15. Training Overgeneralization

16. Final Model To compensate for overgeneralization, the overall average profit, computed on validation data, is examined. The final parameter estimates for the model are taken from the training iteration with the maximum validation profit.

17. Neural Network Final Model