introduction to information retrieval ir l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Introduction to Information Retrieval (IR) PowerPoint Presentation
Download Presentation
Introduction to Information Retrieval (IR)

Loading in 2 Seconds...

play fullscreen
1 / 22

Introduction to Information Retrieval (IR) - PowerPoint PPT Presentation


  • 217 Views
  • Uploaded on

Introduction to Information Retrieval (IR) Mark Craven craven@cs.wisc.edu craven@biostat.wisc.edu 5730 Medical Sciences Center Documents and Corpora document: a passage of free text or hypertext Usenet posting Web page newswire story MEDLINE abstract journal article

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Introduction to Information Retrieval (IR)' - Albert_Lan


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
introduction to information retrieval ir

Introduction to Information Retrieval (IR)

Mark Craven

craven@cs.wisc.edu

craven@biostat.wisc.edu

5730 Medical Sciences Center

documents and corpora
Documents and Corpora
  • document: a passage of free text or hypertext
    • Usenet posting
    • Web page
    • newswire story
    • MEDLINE abstract
    • journal article
  • corpus (pl. corpora): a collection of documents
    • MEDLINE
    • Reuters stories from 1999
    • the Web
the ad hoc retrieval problem
The Ad-Hoc Retrieval Problem
  • given:
    • a document collection (corpus)
    • an arbitrary query
  • do:
    • return a list of relevant documents
  • this is the problem addressed by Web search engines
typical ir system

query

processor

spider

document

processor

Typical IR System

inverted index

the index and inverse index
The Index and Inverse Index
  • index: a relation mapping each document to the set of keywords it is about
  • inverse index
  • where do these come from?
inverted index

aardvark

anteater

hungry

zebra

Inverted Index

index

corpus

a simple boolean query

aardvark

anteater

hungry

zebra

A Simple Boolean Query
  • to answer query “hungry AND zebra”, get intersection of documents pointed to by “hungry” and documents pointed to by “zebra”
other things to consider
Other Things to Consider
  • How wan we search on phrases?
  • Should we treat these queries differently?
    • “a hungry zebra”
    • “the hungry zebra”
    • “hungry as a zebra”
  • If we query on “laugh zebra” should we return documents containing the following?
    • “laughing zebra”
    • “laughable zebra”
  • Boolean queries are too coarse - return too many or too few relevant documents.
handling phrases

aardvark

anteater

hungry

zebra

Handling Phrases

95

40

  • store position information in the inverted index
  • to answer query “hungry zebra”, look for documents having “hungry” at position i and “zebra” at position i + 1

25

38

26

handling phrases10
Handling Phrases
  • but this is a primitive notion of phrase
    • we might want “zebras that are hungry” to be considered a match to the phrase “hungry zebra”
    • this requires doing sentence analysis; determining parts of speech for words, etc.
stop words
Stop Words
  • Should we treat these queries differently?
    • “a hungry zebra”
    • “the hungry zebra”
    • “hungry as a zebra”
  • Some systems employ a list of stop words (a.k.a. function words) that are probably not informative for most searches.
    • a, an, the, that, this, of, by, with, to …
    • stop words in a query are ignored
    • but might be handled differently in phrases
stop words12
Stop Words

a

able

about

above

according

accordingly

across

actually

after

afterwards

again

against

all

allow

allows

almost

alone

along

already

also

although

always

am

among

amongst

an

and

another

any

anybody

anyhow

anyone

anything

anyway

anyways

anywhere

apart

appear

appreciate

appropriate

are

around

as

aside

ask

asking

associated

at

available

away

awfully

b

be

became

because

become

becomes

becoming

been

before

beforehand

behind

being

believe

below

beside

besides

best

better

between

beyond

both

brief

but

by

...

a special purpose stop list
A Special Purpose Stop List

Bos taurus

Botrytis cinerea

C. elegans

Chicken

Goat

Gorilla

Guinea pig

Hamster

Human

Mouse

Pig

Rat

Spinach

unknown gene

cDNA

DNA clone

BAC

PAC

cosmid

clone

genomic sequence

potentially degraded

stemming
Stemming
  • If we query on “laugh zebra” should we return documents containing the following?
    • “laughing zebra”
    • “laughable zebra”
  • Some systems perform stemming on words; truncating related words to a common stem.
    • laugh laugh-
    • laughs laugh-
    • laughing laugh-
    • laughed laugh-
stemming15
Stemming
  • the Lovins stemmer
    • 260 suffix patterns
    • iterative longest match procedure

(.*)SSES

$1SS

(.*[AEIOU].*)ED

$1

  • the Porter stemmer
    • about 60 patterns grouped into sets
    • apply patterns in each set before moving to next
stemming16
Stemming
  • May be helpful
    • reduces vocabulary 10-50%
    • may increase recall
  • May not be helpful
    • for some queries, the sense of a word is important
    • stemming algorithms are heuristic; may conflate semantically different words (e.g. gall and gallery)
  • As with stop words, might want to handle stemming differently in phrases
the vector space model

d1

d2

q

d3

The Vector Space Model
  • Boolean queries are too coarse - return too many or too few relevant documents.
  • Most IR systems are based on the vector space model
the vector space model18
The Vector Space Model
  • documents/queries represented by vectors in a high-dimensional space
  • each dimension corresponds to a word in the vocabulary
  • most relevant documents are those whose vectors are closest to query vector
vector similarity
Vector Similarity
  • one way to determine vector similarity is the cosine measure:
  • if the vectors are normalized, we can simply take their dot product
determining word weights
Determining Word Weights
  • lots of heuristics
  • one well established one is TFIDF (term frequency, inverse document frequency) weighting
    • numerator includes , number of occurrences of word in document
    • denominator includes , total number of occurrences of in corpus
tfidf one form
TFIDF: One Form

(N = total number of words in the corpus)

the probability ranking principle
The Probability Ranking Principle
  • most IR systems are based on the premise that ranking documents in order of decreasing probability is the right thing to do
  • assumes documents are independent
    • does wrong thing with duplicates
    • doesn’t promote diversity in returned documents