machine learning n.
Skip this Video
Loading SlideShow in 5 Seconds..
Machine Learning PowerPoint Presentation
Download Presentation
Machine Learning

Loading in 2 Seconds...

play fullscreen
1 / 21

Machine Learning - PowerPoint PPT Presentation

  • Updated on

Machine Learning. Part I: Classification and Bayesian Learning. Ref: E. Alpaydin, Intro to Machine Learning, MIT 2004. Machine Learning. Machine Leaning is programming computers to optimize a perf criteria using example data or past experience Inference from samples

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Machine Learning

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
machine learning

Machine Learning

Part I:

Classification and Bayesian Learning

Ref: E. Alpaydin, Intro to Machine Learning, MIT 2004

machine learning1
Machine Learning
  • Machine Leaning is programming computers to optimize a perf criteria using example data or past experience
    • Inference from samples
  • There is a process that explains the data we observe. But we don’t know the details about how the data are generated.
    • Internet requests, failure events, etc
  • It’s hard to identify (model) the process completely, we could construct a good and useful approximation that detect certain patterns. Such patterns would help us to understand the process and make predictions about the future.
types of machine learning
Types of Machine Learning
  • Supervised learning is to create a function from training data. The training data consist of pairs of input objects (typically vectors), and desired outputs.
    • Classification: Given an input, the output is Boolean (yes/no) to predict a class label of the input object;
    • Regression: If the label is a numerical value, learn the function f(x) that best explain the input instance;
  • Unsupervised learning: manual labels of inputs are not used.
    • Clustering: partition a data set into subsets (clusters), so that the data in each subset share some common trait
  • Semi-supervised learning: make use of both labeled and unlabeled data for training
  • Reinforcement Learning
    • Learning a policy: A sequence of outputs; No supervised output but delayed reward
    • Examples: game playing, robot navigation
supervised learning
Supervised Learning
  • Use of Supervised Learning
  • Classification
  • Regression
  • Evaluation Methodology
  • Bayesian Learning for Classification
why supervised learning
Why Supervised Learning?
  • Prediction of future cases:Use the rule to predict the output for future inputs
  • Knowledge extraction:The rule is easy to understand
  • Compression:The rule is simpler than the data it explains
  • Outlier detection: Exceptions that are not covered by the rule, e.g., fraud
E.g: Credit scoring

Differentiating betweenlow-riskand high-risk customers from their income and savings

Rule-based prediction


Discriminant: IF income > θ1 AND savings > θ2

THEN low-riskELSE high-risk

learning a class from examples
Learning a Class from Examples
  • Given a set of examples of cars, with a label of “family car” or not according to a survey, class learning is to find a description that is shared by all positive examples.
  • Use of the class info
    • Prediction: Is car x a family car?
    • Knowledge extraction: What do people expect from a family car?
training set x
Training set X

Input representation

Attributes: price & engine power

Label of each instance

hypothesis class c
Hypothesis Class: C

Most specific hypothesis, S

Most general hypothesis, G

Learning is to find a particular

hypothesis h to approximate C

model selection generalization
Model Selection & Generalization
  • Learning is an ill-posed problem: data is not sufficient to find a unique solution
    • Limited number of sample data
    • Some data might be noise due to imprecision in recording, labeling, or hidden (latent, unobservable) attributes that affect the label of instances
  • The need for inductive bias: assumptions about class structureH
    • Why rectangle, not circle or irregular shape?
    • What’s degree of tightness of fitting?
  • Generalization:How well a model performs on new data
noise and model complexity
Noise and Model Complexity

Simple model is preferred

  • Easy to use (check)

(lower time complexity)

  • Easy to train (lower

space complexity)

  • Easyto explain

(more interpretable)

  • Easy to generalize (lower

variance )

Noise: any anomaly in the data

which leads it infeasible to reach

a zero-error classification

with a simple hypothesis class

probably approximately correct pac learning
Probably Approximately Correct (PAC) Learning
  • How many training examples N should we have, suchthat with probability at least 1 ‒ δ, h has error at most ε ?
  • Each strip is at most ε/4
  • Pr that we miss a strip 1‒ ε/4
  • Pr that N instances miss a strip (1 ‒ ε/4)N
  • Pr that N instances miss 4 strips 4(1 ‒ ε/4)N
  • 4(1 ‒ ε/4)N ≤ δ and (1 ‒ x)≤exp( ‒ x)
  • 4exp(‒ εN/4) ≤ δ and N ≥ (4/ε)log(4/δ)
2 class vs k class
2-Class vs K-Class

K-class problem be

viewed as K 2-class problem:

Train hypotheses

hi(x), i =1,...,K:

  • Examples
    • Price of a used car
    • Speed of Top500
  • x : car attributes

y : price

y = g (x | θ)

g ( ) model,

θ parameters

Linear regression

  • y = wx+w0
basic concepts
Basic Concepts
  • Interpolation
    • Find a function that best fits a training set with no presence of noise
    • r = f(x)
  • Extrapolation
    • Predict the output for any x, if x is NOT in the training set
  • Regression
    • Noise factor must be considered
    • r = f(x) +  OR there’re hidden variables we couldn’t observe: r = f(x, z)

For a given test set, find g() that minimizes the empirical error

underfitting vs overfitting
Underfitting vs Overfitting
  • Underfitting: Hypothesis (H) less complex than actual model (C)
    • Using a line to fit data sampled from a 3rd order polynomial
    • Accuracy increases with more sample data; may not enough if the hypothesis is too complex
  • Overfitting: H more complex than C
    • Having more training data helps but only up to a certain point
triple trade off
Triple Trade-Off

Trade-off between three factors :

    • Complexity of the hypothesisH, c (H): capacity of the hypothesis class
    • Training set size, N,
    • Generalization error, E, on new examples
  • As N­, E¯
  • As c (H)­, first E¯ and then E­(The error of an over-complex hypothesis can be kept in check by increasing the amount of training data, but only up to a point)
cross validation
  • To estimate generalization error, we need data unseen during training.
  • Three types of data in cross-validation:
    • Training set (50%)
    • Validation set (25%)
    • Test (publication) set (25%)
  • Resampling when there is few data
dimensions of a supervised learner summary
Dimensions of a Supervised Learner: Summary
  • Model g() and parameter 
  • Loss function L(): diff between desired output and approximation
  • Optimization procedure:

return the argument that minimizes