Ling 570 day 7 classifiers
This presentation is the property of its rightful owner.
Sponsored Links
1 / 86

Ling 570 Day 7: Classifiers PowerPoint PPT Presentation


  • 116 Views
  • Uploaded on
  • Presentation posted in: General

Ling 570 Day 7: Classifiers. Outline. Open questions One last bit on POS taggers: Evaluation Classifiers!. Evaluating Taggers. Evaluation. How can we evaluate a POS tagger?. Evaluation. How can we evaluate a POS tagger? Overall error rate w.r.t . gold-standard test set.

Download Presentation

Ling 570 Day 7: Classifiers

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Ling 570 day 7 classifiers

Ling 570 Day 7:Classifiers


Outline

Outline

  • Open questions

  • One last bit on POS taggers: Evaluation

  • Classifiers!


Evaluating taggers

Evaluating Taggers


Evaluation

Evaluation

  • How can we evaluate a POS tagger?


Evaluation1

Evaluation

  • How can we evaluate a POS tagger?

    • Overall error rate w.r.t. gold-standard test set.


Evaluation metric

Evaluation metric

  • A common single metric is accuracy

  • Defined as

  • Why not precision and recall?


Evaluation metric1

Evaluation metric

  • Precision

  • Recall


Tagging scenarios

Tagging Scenarios

  • But tagging doesn’t have to result in one tag

    • It might be possible that we don’t want to resolve tag ambiguities at tag time

      • E.g., save them for some other process

  • Or there may not necessarily be one answer for any given word/sequence (e.g., the gold standard contains multiple answers)

  • In either case eval is not so cut and dried


Evaluation2

Evaluation

  • How can we evaluate the POS tagger?

    • Overall error rate w.r.t. gold-standard test set.

    • Error rates on particular tags

    • Error rates on particular words

    • Tag confusions...


Error analysis

Error Analysis

  • Confusion matrix (contingency table)

  • Identify primary contributors to error rate

    • Noun (NN) vsProperNoun (NNP) vsAdj (JJ)

    • Preterite (VBD) vs Participle (VBN) vs Adjective (JJ)


Evaluation3

Evaluation

  • Result is compared to manually coded “Gold Standard”

    • Typically accuracy reaches 96-97%

    • Compare with result for a baseline tagger (no context).


Evaluation4

Evaluation

  • Result is compared to manually coded “Gold Standard”

    • Typically accuracy reaches 96-97%

    • Compare with result for a baseline tagger (no context).

      • E.g. Most common class

        • Assign most frequent POS for each words


Evaluation5

Evaluation

  • Result is compared to manually coded “Gold Standard”

    • Typically accuracy reaches 96-97%

    • Compare with result for a baseline tagger (no context).

      • E.g. Most common class

        • Assign most frequent POS for each words

  • Important:

    • 100% is impossible even for human annotators.


Text classification and machine learning

Text Classification andMachine Learning


Example sentiment detection

Example: Sentiment detection

  • Input: Customer reviews

  • Output: Did the customer like the product?

  • Setup:

    • Get a bunch of labeled data

    • SOMEBODY HAS TO DO THIS BY HAND!

    • Want to predict the sentiment given new reviews

  • Features:

    • Information about the reviews

    • Words: “love”, “happy”, “warm”; “waste”, “joke”, “mistake”


Example sentiment detection1

Example: Sentiment detection

  • Input: Customer reviews

  • Output: Did the customer like the product?

  • Setup:

    • Get a bunch of labeled data

    • SOMEBODY HAS TO DO THIS BY HAND!

    • Want to predict the sentiment given new reviews

  • Features:

    • Information about the reviews

    • Words: “love”, “happy”, “warm”; “waste”, “joke”, “mistake”


Example sentiment detection2

Example: Sentiment detection

Don't waste your money. This company is a joke. They were and have been backordered for quite some time, but they just kept advertising on TV all through the holiday season.I agree with another reviewer that it's too easy to make a mistake when ordering on the company webite. I thought I was doing a mock order to see the final price before actually placing the order, but no. It placed the order immedia

  • Input: Customer reviews

  • Output: Did the customer like the product?

  • Setup:

    • Get a bunch of labeled data

    • SOMEBODY HAS TO DO THIS BY HAND!

    • Want to predict the sentiment given new reviews

  • Features:

    • Information about the reviews

    • Words: “love”, “happy”, “warm”; “waste”, “joke”, “mistake”


Example sentiment detection3

Example: Sentiment detection

Don't waste your money. This company is a joke. They were and have been backordered for quite some time, but they just kept advertising on TV all through the holiday season.I agree with another reviewer that it's too easy to make a mistake when ordering on the company webite. I thought I was doing a mock order to see the final price before actually placing the order, but no. It placed the order immedia

  • Input: Customer reviews

  • Output: Did the customer like the product?

  • Setup:

    • Get a bunch of labeled data

    • SOMEBODY HAS TO DO THIS BY HAND!

    • Want to predict the sentiment given new reviews

  • Features:

    • Information about the reviews

    • Words: “love”, “happy”, “warm”; “waste”, “joke”, “mistake”

I love my snuggie! It is warm. I wear it with the opening to the back.The length is long but keeps your feet warm when you set down. If I need to walk around I put the opening to the front. It fits great either way and keeps me very warm. Try it before you judge. You will be very happy and warm. I have gave some to family and friends and the all loved them after they tryed them.


Example sentiment detection4

Example: Sentiment detection

Don't waste your money. This company is a joke. They were and have been backordered for quite some time, but they just kept advertising on TV all through the holiday season.I agree with another reviewer that it's too easy to make a mistake when ordering on the company webite. I thought I was doing a mock order to see the final price before actually placing the order, but no. It placed the order immedia

  • Input: Customer reviews

  • Output: Did the customer like the product?

  • Setup:

    • Get a bunch of labeled data

    • SOMEBODY HAS TO DO THIS BY HAND!

    • Want to predict the sentiment given new reviews

  • Features:

    • Information about the reviews

    • Words: “love”, “happy”, “warm”; “waste”, “joke”, “mistake”

I love my snuggie! It is warm. I wear it with the opening to the back.The length is long but keeps your feet warm when you set down. If I need to walk around I put the opening to the front. It fits great either way and keeps me very warm. Try it before you judge. You will be very happy and warm. I have gave some to family and friends and the all loved them after they tryed them.


Example sentiment analysis

Example: Sentiment Analysis

  • Task:

    • Given a review, predict if they liked it

  • Outcomes:

    • Positive / negative, or 1-5 scale

    • Split by aspect (liked the comfort, hated the price)

  • What information would help us predict this?


Formalizing the task

Formalizing the task

  • Task:

    • inputs, e.g. review

    • outcomes; could be

  • Data set input/output pairs

  • Split into training, dev, test

    • Experimentation cycle

      • Learn a classifier on train

      • Tune free parameters on dev data

      • Evaluate on test

    • Need to keep these separate!

Test

data

Training

data

Development

data


Classification examples

Classification Examples

  • Spam filtering

  • Call routing

  • Text classification


Pos tagging

POS Tagging

  • Task: Given a sentence, predict tag of each word

  • Is this a classification problem?


Pos tagging1

POS Tagging

  • Task: Given a sentence, predict tag of each word

  • Is this a classification problem?

  • Categories: N, V, Adj,…

  • What information is useful?


Pos tagging2

POS Tagging

  • Task: Given a sentence, predict tag of each word

  • Is this a classification problem?

  • Categories: N, V, Adj,…

  • What information is useful?

  • How do POS tagging, text classification differ?


Pos tagging3

POS Tagging

  • Task: Given a sentence, predict tag of each word

  • Is this a classification problem?

  • Categories: N, V, Adj,…

  • What information is useful?

  • How do POS tagging, text classification differ?

    • Sequence labeling problem


Word segmentation

Word Segmentation

  • Task: Given a string, break into words

  • Categories:


Word segmentation1

Word Segmentation

  • Task: Given a string, break into words

  • Categories:

    • B(reak), NB (no break)

    • B(eginning), I(nside), E(nd)

  • e.g. c1 c2 || c3 c4 c5


Word segmentation2

Word Segmentation

  • Task: Given a string, break into words

  • Categories:

    • B(reak), NB (no break)

    • B(eginning), I(nside), E(nd)

  • e.g. c1 c2 || c3 c4 c5

    • c1/NB c2/B c3/NB c4/NB c5/B

    • c1/B c2/E c3/B c4/I c5/E

  • What type of task?


Word segmentation3

Word Segmentation

  • Task: Given a string, break into words

  • Categories:

    • B(reak), NB (no break)

    • B(eginning), I(nside), E(nd)

  • e.g. c1 c2 || c3 c4 c5

    • c1/NB c2/B c3/NB c4/NB c5/B

    • c1/B c2/E c3/B c4/I c5/E

  • What type of task?

    • Also sequence labeling


The structure of a classification problem

The Structure of a Classification Problem


Classification problem steps

Classification Problem Steps

  • Input processing:

    • Split data into training/dev/test

    • Convert data into an Attribute-Value Matrix

      • Identify candidate features

      • Perform feature selection

      • Create AVM representation

  • Training

  • Testing

  • Evaluation


Classifiers in practice

Classifiers in practice

Training data

Model learning

Model


Classifiers in practice1

Classifiers in practice

Training data

Model learning

Model

Predictions

Test data

Modeltesting


Classifiers in practice2

Classifiers in practice

Training data

Model learning

Model

Preprocessing

Post-processing

Predictions

Test data

Modeltesting


Classifiers in practice3

Classifiers in practice

Training data

Model learning

Model

Preprocessing

Post-processing

Predictions

Test data

Modeltesting

Evaluation


Representing input

Representing Input

  • Potentially infinite values to represent


Representing input1

Representing Input

  • Potentially infinite values to represent

  • Represent input as feature vector

    • x=<v1,v2,v3,…,vn>

    • x=<f1=v1,f2=v2,…,fn=vn>


Representing input2

Representing Input

  • Potentially infinite values to represent

  • Represent input as feature vector

    • x=<v1,v2,v3,…,vn>

    • x=<f1=v1,f2=v2,…,fn=vn>

  • What are good features?


Example i

Example I

  • Spam Tagging

    • Classes: Spam/Not Spam

    • Input:

      • Email messages


Ling 570 day 7 classifiers

Doc1

Western Union Money Transfer [email protected] Bishops Square Akpakpa E1 6AO, CotonouBenin RepublicWebsite: http://www.westernunion.com/ info/selectCountry.asPPhone: +229 99388639Attention Beneficiary,This to inform you that the federal ministry of finance Benin Republic has started releasing scam victim compensation fund mandated by United Nation Organization through our office.I am contacting you because our agent have sent you the first payment of $5,000 for your compensation funds total amount of $500 000 USD (Five hundred thousand united state dollar)We need your urgent response so that we shall release your payment information to you.You can call our office hot line for urgent attention(+22999388639)


Ling 570 day 7 classifiers

Doc2

  • Hello! my dear. How are you today and your family? I hope all is good,kindly pay Attention and understand my aim of communicating you todaythrough this Letter, My names is Saif al-Islam  al-Gaddafi the Son offormer  Libyan President. i was born on 1972 in Tripoli Libya,By Gaddafi’ssecond wive.I want you to help me clear this fund in your name which i deposited inEurope please i would like this money to be transferred into your accountbefore they find it.the amount is 20.300,000 million GBP British Pounds sterling through a


Ling 570 day 7 classifiers

Doc3

  • from: [email protected]

  • Apply for loan at 3% interest Rate..Contact us for details.


Ling 570 day 7 classifiers

Doc4

  • from: [email protected]

  • REMINDER:If you have not received a PIN number to vote in the elections and have not already contacted us, please contact either DragoRadev ([email protected]) or Priscilla Rasmussen ([email protected]) right away.Everyone who has not received a pin but who has contacted us already will get a new pin over the weekend.Anyone who still wants to join for 2011 needs to do this by Monday (November 7th) in order to be eligible to vote.And, if you do have your PIN number and have not voted yet, remember every vote counts!


What are good features

What are good features?


Possible features

Possible Features

  • Words!


Possible features1

Possible Features

  • Words!

    • Feature for each word


Possible features2

Possible Features

  • Words!

    • Feature for each word

      • Binary: presence/absence

      • Integer: occurrence count

      • Particular word types: money/sex/: [Vv].*gr.*


Possible features3

Possible Features

  • Words!

    • Feature for each word

      • Binary: presence/absence

      • Integer: occurrence count

      • Particular word types: money/sex/: [Vv].*gr.*

  • Errors:

    • Spelling, grammar


Possible features4

Possible Features

  • Words!

    • Feature for each word

      • Binary: presence/absence

      • Integer: occurrence count

      • Particular word types: money/sex/: [Vv].*gr.*

  • Errors:

    • Spelling, grammar

  • Images


Possible features5

Possible Features

  • Words!

    • Feature for each word

      • Binary: presence/absence

      • Integer: occurrence count

      • Particular word types: money/sex/: [Vv].*gr.*

  • Errors:

    • Spelling, grammar

  • Images

  • Header info


Representing input attribute value matrix

Representing Input:Attribute-Value Matrix


Representing input attribute value matrix1

Representing Input:Attribute-Value Matrix


Representing input attribute value matrix2

Representing Input:Attribute-Value Matrix


Representing input attribute value matrix3

Representing Input:Attribute-Value Matrix


Representing input attribute value matrix4

Representing Input:Attribute-Value Matrix


Classifier

Classifier

  • Result of training on input data

    • With or without class labels


Classifier1

Classifier

  • Result of training on input data

    • With or without class labels

  • Formal perspective:

    • f(x) =y: x is input; y in C


Classifier2

Classifier

  • Result of training on input data

    • With or without class labels

  • Formal perspective:

    • f(x) =y: x is input; y in C

    • More generally:

      • f(x)={(ci,scorei)}, where

        • x is input,

        • ci in C,

        • scoreiis score for category assignment


Testing

Testing

  • Input:

    • Test data:

      • e.g. AVM

    • Classifier

  • Output:


Testing1

Testing

  • Input:

    • Test data:

      • e.g. AVM

    • Classifier

  • Output:

    • Decision matrix

    • Can assign highest scoring class to each input


Testing2

Testing

  • Input:

    • Test data:

      • e.g. AVM

    • Classifier

  • Output:

    • Decision matrix

    • Can assign highest scoring class to each input


Testing3

Testing

  • Input:

    • Test data:

      • e.g. AVM

    • Classifier

  • Output:

    • Decision matrix

    • Can assign highest scoring class to each input


Evaluation6

Evaluation

  • Confusion matrix:

  • Precision: TP/(TP+FP)


Evaluation7

Evaluation

  • Confusion matrix:

  • Precision: TP/(TP+FP)

  • Recall: TP/(TP+FN)


Evaluation8

Evaluation

  • Confusion matrix:

  • Precision: TP/(TP+FP)

  • Recall: TP/(TP+FN)

  • F-score: 2PR/(P+R)


Evaluation9

Evaluation

  • Confusion matrix:

  • Precision: TP/(TP+FP)

  • Recall: TP/(TP+FN)

  • F-score: 2PR/(P+R)

  • Accuracy = (TP+TN)/(TP+TN+FP+TN)


Evaluation10

Evaluation

  • Confusion matrix:

  • Precision: TP/(TP+FP)

  • Recall: TP/(TP+FN)

  • F-score: 2PR/(P+R)

  • Accuracy = (TP+TN)/(TP+TN+FP+TN)

  • Why F-score? Accuracy?


Evaluation example

Evaluation Example

  • Confusion matrix:

  • Precision: 1/(1+4)=1/5

  • Recall: TP/(TP+FN)=1/6

  • F-score: 2PR/(P+R)=2*1/5*1/6/(1/5+1/6)=2/11

  • Accuracy = 91%


Evaluation example1

Evaluation Example

  • Confusion matrix:

  • Precision:


Evaluation example2

Evaluation Example

  • Confusion matrix:

  • Precision: 1/(1+4)=1/5

  • Recall: TP/(TP+FN)


Evaluation example3

Evaluation Example

  • Confusion matrix:

  • Precision: 1/(1+4)=1/5

  • Recall: TP/(TP+FN)=1/6

  • F-score: 2PR/(P+R)=


Evaluation example4

Evaluation Example

  • Confusion matrix:

  • Precision: 1/(1+4)=1/5

  • Recall: TP/(TP+FN)=1/6

  • F-score: 2PR/(P+R)=2*1/5*1/6/(1/5+1/6)=2/11

  • Accuracy = 91%


Classification problem steps1

Classification Problem Steps

  • Input processing:

    • Split data into training/dev/test


Classification problem steps2

Classification Problem Steps

  • Input processing:

    • Split data into training/dev/test

    • Convert data into an Attribute-Value Matrix

      • Identify candidate features

      • Perform feature selection

      • Create AVM representation


Classification problem steps3

Classification Problem Steps

  • Input processing:

    • Split data into training/dev/test

    • Convert data into an Attribute-Value Matrix

      • Identify candidate features

      • Perform feature selection

      • Create AVM representation

  • Training


Classification problem steps4

Classification Problem Steps

  • Input processing:

    • Split data into training/dev/test

    • Convert data into an Attribute-Value Matrix

      • Identify candidate features

      • Perform feature selection

      • Create AVM representation

  • Training

  • Testing

  • Evaluation


Classification algorithms

Classification Algorithms

Will be covered in detail in 572

  • Nearest Neighbor

  • Naïve Bayes

  • Decision Trees

  • Neural Networks

  • Maximum Entropy


Feature design representation

Feature Design & Representation

  • What sorts of information do we want to encode?


Feature design representation1

Feature Design & Representation

  • What sorts of information do we want to encode?

    • words, frequencies, ngrams, morphology, sentence length, etc

  • Issue


Feature design representation2

Feature Design & Representation

  • What sorts of information do we want to encode?

    • words, frequencies, ngrams, morphology, sentence length, etc

  • Issue: Learning algorithms work on numbers

    • Many work only on binary values (0/1)


Feature design representation3

Feature Design & Representation

  • What sorts of information do we want to encode?

    • words, frequencies, ngrams, morphology, sentence length, etc

  • Issue: Learning algorithms work on numbers

    • Many work only on binary values (0/1)

    • Others work on any real-valued input

  • How can we represent different information

    • Numerically?

    • Binary?


Representation

Representation

  • Words/tags/ngrams/etc


Representation1

Representation

  • Words/tags/ngrams/etc

    • One feature per item:


Representation2

Representation

  • Words/tags/ngrams/etc

    • One feature per item:

      • Binary: presence/absence

      • Real: counts

  • Binarizing numeric features:


Representation3

Representation

  • Words/tags/ngrams/etc

    • One feature per item:

      • Binary: presence/absence

      • Real: counts

  • Binarizing numeric features:

    • Single threshold

    • Multiple thresholds

    • Binning: 1 binary feature/bin


  • Login