evaluating hypotheses reading coursepack learning from examples section 4 pp 16 21 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Evaluating Hypotheses Reading: Coursepack : Learning From Examples, Section 4 (pp. 16-21) PowerPoint Presentation
Download Presentation
Evaluating Hypotheses Reading: Coursepack : Learning From Examples, Section 4 (pp. 16-21)

Loading in 2 Seconds...

play fullscreen
1 / 17

Evaluating Hypotheses Reading: Coursepack : Learning From Examples, Section 4 (pp. 16-21) - PowerPoint PPT Presentation


  • 66 Views
  • Uploaded on

Evaluating Hypotheses Reading: Coursepack : Learning From Examples, Section 4 (pp. 16-21). Evaluating Hypotheses . What we want: hypothesis that best predicts unseen data Assumption: Data is “ iid ” (independently and identically distributed). Accuracy and Error.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Evaluating Hypotheses Reading: Coursepack : Learning From Examples, Section 4 (pp. 16-21)' - libba


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
evaluating hypotheses
Evaluating Hypotheses
  • What we want: hypothesis that best predicts unseen data
  • Assumption: Data is “iid” (independently and identically distributed)
accuracy and error
Accuracy and Error
  • Accuracy = fraction of correct classifications on unseen data (test set)
  • Error rate = 1 − Accuracy
how to use available data to best measure accuracy1
How to use available data to best measure accuracy?

Split data into training and test sets.

how to use available data to best measure accuracy2
How to use available data to best measure accuracy?

Split data into training and test sets.

But how to split?

how to use available data to best measure accuracy3
How to use available data to best measure accuracy?

Split data into training and test sets.

But how to split?

Too little training data:

Too little test data:

how to use available data to best measure accuracy4
How to use available data to best measure accuracy?

Split data into training and test sets.

But how to split?

Too little training data: Don’t get optimal classifier

Too little test data: Measured accuracy is not correct

one solution k fold cross validation
One solution: “k-fold cross validation”
  • Each example is used both as a training instance and as a test instance.
  • Split data into k disjoint parts: S1, S2, ..., Sk.
  • For i = 1 to k

Select Sito be the test set. Train on the remaining data, test on Si, to obtain accuracy Ai .

  • Report as the final accuracy.
avoid peeking at test data when training
Avoid “peeking” at test data when training

Example from readings:

Split data into training and test sets.

Train model with one learning parameter

(e.g., “gain” vs “gain ratio”)

Test on test set.

Repeat with other learning parameter.

Test on test set.

Return accuracy of model with best performance.

What’s wrong with this procedure?

avoid peeking at test data when training1
Avoid “peeking” at test data when training

Example from readings:

Split data into training and test sets.

Train model with one learning parameter

(e.g., “gain” vs “gain ratio”)

Test on test set.

Repeat with other learning parameter.

Test on test set.

Return accuracy of model with best performance.

Problem: You used the test set to select the best model – but is part of the

learning process! Risk of overfitting to a particular test set.

Need to evaluate final learned model on previously unseen data.

slide12

Can also solve this problem by using k-fold cross-validation to select model parameters, and then evaluate the resulting model on unseen test data that has been set aside previous to training.

evaluating classification algorithms
Evaluating classification algorithms

“Confusion matrix” for a given class c

Predicted (or “classified”)

True False

Actu al (in class c) (not in class c)

True (in class c) TruePositiveFalseNegative

False (not in class c) FalsePositiveTrueNegative

evaluating classification algorithms1
Evaluating classification algorithms

“Confusion matrix” for a given class c

Predicted (or “classified”)

True False

Actu al (in class c) (not in class c)

True (in class c) TruePositiveFalseNegative

False (not in class c) FalsePositiveTrueNegative

Type 2 error

Type 1 error

slide15

Precision: Fraction of true positives out of all predicted positives:

  • Recall: Fraction of true positives out of all actual positives:
error vs loss
Error vs. Loss
  • Loss functions