Stemming Algorithms

1 / 30

# Stemming Algorithms - PowerPoint PPT Presentation

Stemming Algorithms. 資訊擷取與推薦技術：期中報告 指導教授：黃三益 老師 學生： 9142608 黃 哲修 9142609 張家豪. Outline. Introduction Types of stemming algorithms Experimental evaluations of stemming Stemming to compress inverted files Summary Appendix. Introduction.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about 'Stemming Algorithms' - eli

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### Stemming Algorithms

9142609 張家豪

Outline
• Introduction
• Types of stemming algorithms
• Experimental evaluations of stemming
• Stemming to compress inverted files
• Summary
• Appendix
Introduction
• Stemming is one technique to provide ways of finding morphological variants of search terms.
• Used to improve retrieval effectiveness and to reduce the size of indexing files.
• Taxonomy for stemming algorithms
Introduction (con’t)
• Criteria for judging stemmers
• Correctness
• Overstemming: too much of a term is removed.
• Understemming: too little of a term is removed.
• Retrieval effectiveness

measured with recall and precision, and on their speed, size, and so on

• compression performance
Type of stemming algorithms
• Table lookup approach
• Successor Variety
• n-gram stemmers
• Affix Removal Stemmers
Table lookup approach
• Store a table of all index terms and their stems, so terms from queries and indexes could be stemmed very fast.
• Problems
• There is no such data for English. Or some terms are domain dependent.
• The storage overhead for such a table, though trading size for time is sometimes warranted.
Successor Variety approach
• Determine word and morpheme boundaries based on the distribution of phonemes in a large body of utterances.
• The successor variety of a string is the number of different characters that follow it in words in some body of text.
• The successor variety of substrings of a term will decrease as more characters are added until a segment boundary is reached.
Successor Variety approach (con’t)

Successor Variety approach (con’t)
• cutoff method
• some cutoff value is selected and a boundary is identified whenever the cutoff value is reached
• peak and plateau method
• segment break is made after a character whose successor variety exceeds that of the characters immediately preceding and following it
• complete method
Successor Variety approach (con’t)
• entropy method
• : the number of words in a text body beginning with the i length sequence of letters 
• : the number of words in with the successor j
• The probability that a member of number of words in

has the successor j is given by

• The entropy of is
Successor Variety approach (con’t)
• Two criteria used to evaluate various segmentation methods
• the number of correct segment cuts divided by the total number of cuts
• the number of correct segment cuts divided by the total number of true boundaries
• After segmenting, if the first segment occurs in more than 12 words in the corpus, it is probably a prefix.
Successor Variety approach (con’t)
• The successor variety stemming process has three parts
• determine the successor varieties for a word
• segment the word using one of the methods
• select one of the segments as the stem
n-gram stemmers
• Association measures are calculated between pairs of terms based on shared unique digrams.

statistics => st ta at ti is st ti ic cs

unique digrams = at cs ic is st ta ti

statistical => st ta at ti is st ti ic ca al

unique digrams = al at ca ic is st ta ti

• Dice’s coefficient (similarity)

A and B are the numbers of unique digrams in the first and the second words. C is the number of unique digrams shared by A and B.

n-gram stemmers (con’t)
• Similarity measures are determined for all pairs of terms in the database, forming a similarity matrix
• Once such a similarity matrix is available, terms are clustered using a single link clustering method (as described in Ch.16)
Affix Removal Stemmers
• Affix removal algorithms remove suffixes and/or prefixes from terms leaving a stem
• If a word ends in “ies” but not ”eies” or ”aies ” (Harman 1991)

Then “ies” -> “y”

• If a word ends in “es” but not ”aes” , or ”ees ” or “oes”

Then “es” -> “e”

• If a word ends in “s” but not ”us” or ”ss ”

Then “s” -> “NULL”

The Porter algorithm
• The Porter algorithm consists of a set of condition/action rules.
• The condition fall into three classes
• Conditions on the stem
• Conditions on the suffix
• Conditions on rules
Conditions on the stem

1.The measure , denoted m ,of a stem is based on its alternate vowel-consonant sequences.

Conditions on the stem (con’t)

2.*<X> ---the stem ends with a given letter X

3.*v*---the stem contains a vowel

4.*d ---the stem ends in double consonant

5.*o ---the stem ends with a consonant-vowel-consonant,sequence ,where the final consonant is not w, x or y

Suffix conditions take the form: (current_suffix == pattern)

Conditions on the rules
• The rules are divided into steps. The rules in a step are examined in sequence , and only one rule from a step can apply

{ step1a(word);

step1b(stem);

if (the second or third rule of step 1b was used) step1b1(stem);

step1c(stem);

step2(stem);

step3(stem);

step4(stem);

step5a(stem);

step5b(stem);

}

Stemming Studies : Conclusion
• The majority of stemming’s affection on retrieval performance have been positive
• Stemming is as effective as manual conflation
• The effect of stemming is dependent on the nature of vocabulary used
• There appears to be little difference between the retrieval effectiveness of different full stemmers
Stemming to compress inverted files

Lennon et al. report the following compression percentages for various stemmers and databases. It is obvious that the savings in storage can be substantial.

Compression rates also increase for affix removal stemmers as the number of suffixes increases.

Summary
• Stemmers are used to conflate terms to improve retrieval effectiveness and /or to reduce the size of indexing file.
• Stemming will increase recall at the cost of decreased precision.
• Stemming can have marked effect on the size of indexing files ,sometimes decreasing the size of file as much as 50 percent .