Chapter 7 Artificial Neural Networks

1. Chapter 7Artificial Neural Networks

2. [Artificial] Neural Networks • A class of powerful methods readily applied to: • Prediction • Classification • Clustering • Biological Neural Net (human brain) is the most powerful – we can generalize/learn from experience • Computers are best at following pre-determined instructions • Computerized Neural Nets attempt to bridge the gap, but do not try to mimic the brain • Predicting time-series in financial world • Diagnosing medical conditions • Identifying clusters of valuable customers • Fraud detection • Etc…

3. Neural Networks • When applied in well-defined domains, their ability to generalize and learn from data “mimics” a human’s ability to learn from experience. • Very useful in Data Mining…better results are the hope • Drawback – training a neural network results in internal weights distributed throughout the network making it difficult to understand why a solution is valid

4. Neural Network History • 1940's McCulloch and Pitts introduced the first neural network computing model to explain function of biological neurons • 1950's, Rosenblatt introduced perceptron, which is simple type of ANN. Nice property: convergence • Because perceptrons have limited expressive power (classes must be separable by hyperplane), the interest in ANNs waned. • Later people realized that the lack of convergence isn't such a big deal so multilayer perceptrons were used • referred to as universal approximators • Great interest in 1980's in ANNs

5. Neural Network History • 1980s: • Back propagation – better way of training a neural net • Computing power became available • Researchers became more comfortable with n-nets • Relevant operational data more accessible • Useful applications (expert systems) emerged

6. ANN’s vs. Human Brain • Human Brain has Billions of neurons and lots of interconnection • ANNS have few neurons and few connections • Human Brian is massively parallel while ANNs that we deal with are not • Some tasks can exploit massive parallelism while some cannot. Guess which? • But people are working on massively parallel ANNs (Blue Brain project).

7. Loan Prospector – HNC/Fair Isaac • A Neural Network (Expert System) is like a black box that knows how to process inputs to create a useful output. • The calculation(s) are quite complex and difficult to understand

8. Neural Net Limitations • Neural Nets are good for prediction and estimation when: • Inputs are well understood • Output is well understood • Experience is available for examples to use to “train” the neural net application (expert system) • Neural Nets are only as good as the training set used to generate it. The resulting model is static and must be updated with more recent examples and retraining for it to stay relevant • These points are true of all prediction methods!

9. Feed-Forward Neural Net Examples • One-way flow through the network from the inputs to the outputs

10. The Unit of a Neural Network • The unit of a neural network is modeled on the biological neuron • The unit combines its inputs into a single value, which it then transforms to produce the output; together these are called the activation function This is cool stuff!

11. ANN Computations • Most common combination function is the weighted sum • The transfer function is often: • Sigmoid (logistic) • Linear • Other • ANN with weighted sum and linear transfer function is doing linear regression • ANN gets power from non-linear transfer function

12. Loan Appraiser - revisited • Illustrates that a neural network (feed-forward in this case) is filled with seemingly meaningless weights • The appraised value of this property is $176,228 (not a bad deal for San Diego!)

13. Neural Network Training • Training is the process of setting the best weights on the edges connecting all the units in the network • The goal is to use the training set to calculate weights where the output of the network is as close to the desired output as possible • Back propagation has been used since the 1980s to adjust the weights (other methods are now available): • Calculates the error by taking the difference between the calculated result and the actual result • The error is fed back through the network and the weights are adjusted to minimize the error • Momentum: keep weights changing in the same direction • Learning rate: allow bigger changes at start then slow down

14. ANN Process • Identify input and outputs • Transform the inputs into narrow range (-1 to 1) • Set up a network with appropriate topology • Train the network on the training examples • Will take many epochs • Use the validation set to choose the right set of weights to avoid overfitting the training data • Evaluate the ANN on the test data

15. ANN Architectures • Many different ways to configure ANNS • Most common is feedforward • Connections only go forward, no loops • Loops are good for time series prediction where may want to remember what happened recently (recurrent neural networks) • How many hidden layers and how many nodes in each layer • No easy answer. Too many levels/nodes leads to overfitting. One hidden layer often sufficient.

16. ANN as a Data Mining Method • Rate of learning: pretty slow, many epochs (runs through the data) • Rate of classifying new examples: fast • Interpretability: Very bad • Expressive Power: very good • So how compares to Decision Trees? • Some differences: • Handles numeric outputs, not just classification • Good with numeric inputs • No data fragmentation problem