# Neural Networks and Classical Linear Regression - PowerPoint PPT Presentation

1 / 26

Neural Networks and Classical Linear Regression. Szu Hua Huang Jianjun Luo Texas Tech University 6/10/2014. Contents. Neural Networks and Classical Linear Regression Introduction Neural Networks Neural Networks VS Classical Linear Regression Case study. Neural Network - Neuron.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Neural Networks and Classical Linear Regression

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

## Neural Networks and Classical Linear Regression

Szu Hua Huang Jianjun Luo

Texas Tech University

6/10/2014

### Contents

• Neural Networks and Classical Linear Regression

• Introduction Neural Networks

• Neural Networks VS Classical Linear Regression

• Case study

### STRUCTURE OF A NEURAL NETWORK

Multilayer feedforward networks

### A simple MLP (multilayer perceptron)

Hidden Layer

Output Layer

Input Layer

Weights(Regression Coefficients)

Back-propagation Model

Updating (Learning)

3

W13

1

W36

Node

W14

W15

6

W46

Output

Input

4

W23

W56

W24

2

Data from True Function

W25

5

Errors

Transfer Function, g(s)

### Neural Network VS Classical Regression Model

Neural Network

Classical Regression

Linear

Normality

Constant variability of error terms

Independence Assumption

• Non-linear

### Neural Network VS Classical Regression Model

Neural Network

Classical Regression

Regression coefficient shows the effect

• Weight estimates(regression coefficient) do not tell you the effect

• No guarantee the best linear combination of parameter estimates

### Neural Network VS Classical Regression Model

Multi-collinearity

Outlier or Influential

No effect to Neural Network ?

It does hurt the Classical Regression

• No effect to Neural Network ?

• It does hurt the Classical Regression

### Overfitting

Neural Network

Classical Regression

Not very sensitive to the given data.

• Sensitive to the given data, too much flexibility to the underlying distribution of data.

• Big Sample size can help to solve the problem of overfitting.

### Assessment

Neural Network

Classical Regression

R square

• Optimization plot based on the updated weight estimates at each iteration of the iterative grip search routine.

• Using valid data

Optimization plot

• 1. When the new weights are only incrementally different from those of the preceding iteration

• 2. When the misclassification rate reaches a required threshold

• 3. When the limit on the number of runs is reached

ASE

training

validation

0

5

10

15

20

Iteration

### Supplement-Other Optimization Algorithms:

• Newton

• Quasi-Newton method

• Levenberg-Marquardt

• Gauss-Newton Method

• etc

Stanford Open Course – Machine Learning

B: Case Study

• Dataset

• The School Children Data Set from Lewis & Taylor “Introduction to Experimental Ecology” (1967)

• Includes 126 male records

• Variables:

• Age (months)

• height (inches)

• weight (pounds)

• Purpose

• Predicting the weight of male school children based on their age and height.

• Comparing neural networks with OLS

• Exploration of the dataset

### Classical Linear Regression Model

procreg data=men;

model weight=height age;

output out=regout p=pred r=resid;

run;

### Output of OLS

Neural Network Model

INPUT

OUTPUT

HIDDEN

COMBINATION

w1+w2S_Height+w3S_Age = H11

TRANSFORMATION

tanh(H11)) =A

Standardization

H11

Height

S_Height

Weight

COMBINATION

w7+w8A+w9B=Weight

S_Age

Age

Standardization

TRANSFORMATION

tanh(H12) =B

COMBINATION

w4+w5S_Height+w6S_Age = H12

H12

18

### Neural Network in SAS

• The SAS neural network procedure

• PROC NEURAL

• SAS Enterprise Miner

• A visual programming with a GUI interface

• ### Neural Network Modeling using SAS Enterprise Miner

• To save time, I recorded the following video to show how to build the Neural Network Model with SAS Enterprise Miner.

• In case you are interested, I uploaded this video to YouTube:

### Comparing Neural Network and Classical Linear Regression predicted values

Output: Observed and Predicted Values of Male's Weight against Age

References

Eric Roberts. Neural networks. Available online at: http://www-cs-faculty.stanford.edu/~eroberts/courses/soco/projects/neural-networks/

Jim Georges, 2009. Applied analytics using SAS Enterprise Miner 6.1 Course Notes. SAS Institute Inc.

Lewis, T. and Taylor, L.R. 1967. Introduction to Experimental Ecology, Academic Press, Inc.

Randall Matignon, 2005. Neural Network Modeling using SAS Enterprise Miner. AuthorHouse

SAS Institute, 1999. SAS/STAT User’s Guide Version 8. Available online at: http://ciser.cornell.edu/sasdoc/saspdf/common/mainpdf.htm

Sue Walsh, 2002. Applying Data Mining Techniques Using Enterprise Miner Course Notes. SAS Institute Inc.

Wikipedia. Neural network. Available online at: http://en.wikipedia.org/wiki/Neural_network

Thank You!