Measuring system performance
Download
1 / 44

Measuring system performance - PowerPoint PPT Presentation


  • 129 Views
  • Uploaded on

Measuring system performance. The library. A system view. Environment. U s e r s. Inputs. Outputs. Transformational process. energy money materials personnel information. products services. System performance measures. recall. precision. relevance.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Measuring system performance' - spike


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

The library
The library

A system view

Environment

U

s

e

r

s

Inputs

Outputs

Transformational

process

energy

money

materials

personnel

information

products

services


System performance measures
System performance measures

recall

precision

relevance


Robert taylor s four levels of question formation
Robert Taylor's four levels of question formation

The actual but unexpressed need for

information (the visceral need)

Q1

The conscious, within-brain description

of the need (the conscious need)

Q2

The formal statement of the need

(the formalized need)

Q3

The question as presented to the infor-

mation system (the compromised need)

Q4

Taylor, Robert S. 1968. Question-negotiation and information seeking in libraries. College & Research Libraries 29(3): 178-194 (May 1968).


System defined relevance
System-defined relevance

"My feet are killing me."

find health AND feet

The health of the lumber

90% industry in terms of cubic feet

of lumber produced


Information retrieval process
Information retrieval process

Question

formulation

Relevancy

determination

System: Which documents

are relevant to the query?

User: Are these documents

relevant to my needs?


Defining relevance
Defining relevance

System-defined

relevance

User-defined

relevance

vs.

Objective

Often topical.

Does it match the query?

Subjective.

Situational.

Is it useful?


User defined relevance
User-defined relevance

"My feet are killing me."

The effect of lysergic acid diethylamide

ingestion on toenail fungus in cloned mice

Soothing remedies for aching feet

Controlling the body by controlling the mind--

meditative techniques for dealing with pain


Determining topical relevance
Determining topical relevance

  • Analyze work as to what it is about

  • Assign to the document one or more terms from a finite list of topics

  • Users can then search on those topic indicators


Recall
Recall

No. of relevant documents retrieved

Total no. of relevant documents in the file

Recall =


Precision
Precision

No. of relevant documents retrieved

Total no. of documents retrieved from the file

Precision =


Precision vs recall
Precision vs. Recall

An inverse relationship

As the level of recall rises the level of precision generally declines and vice versa.

The Cranfield experiments (1957 & 1962)

Cyril Cleverdon, p.i.


Precision vs recall1
Precision vs. Recall

Subject: sexual dimorphism

Word stemming:

Recall

Precision

sex sexes sexual

sexy sexier sexiest

Field-specific searches:

Recall

Precision

DE,TI/sexual()dimorphism


User defined relevance1
User-defined relevance

"Relevance appears to be a subjective quality, unique between the individual and a given document supporting the assumption that relevance can only be judged by the information user."

Miranda Pao


Years later
Years later

"My feet are still killing me."

The effect of lysergic acid diethylamide

ingestion on toenail fungus in cloned mice

Soothing remedies for aching feet

Controlling the body by controlling the mind--

meditative techniques for dealing with pain


Factors affecting relevance 1
Factors affecting relevance (1)

  • Purpose of the information

  • Situation of the user

  • Level at which the information source is written

    • Journal of the Amer. Med. Assn.

    • Healthy times


Factors affecting relevance 2
Factors affecting relevance (2)

  • Subject knowledge of the user

    • Is the data new to the user?

    • Does the information relate to the user's prior knowledge?

  • Values - ethical, social, philosophical, political, religious, legal


User defined relevance2
User-defined relevance

Subjectivity and fluidity make it difficult to use as measuring tool for system performance


Incorporating user defined relevance into information retrieval systems 1
Incorporating user-defined relevance into information retrieval systems (1)

  • User performs search

  • System retrieves results

.

.

.


Incorporating user defined relevance into information retrieval systems 2
Incorporating user-defined relevance into information retrieval systems (2)

  • System asks user if he/she would like to retrieve similar documents

    • Search for other documents with similar word frequencies

    • Search for other documents with same subject descriptors


Search for other documents with same subject descriptors
Search for other documents with same subject descriptors retrieval systems (2)

Main Author:

Title:

Subject(s):

Gribbin, John R.

In search of Schrodinger's cat :

quantum physics and reality /

by John Gribbin.

Schrodinger, Erwin, 1887-1961.

Quantum theory History.

Reality.


Amazon com
Amazon.com retrieval systems (2)


Amazon com1
Amazon.com retrieval systems (2)


Amazon com2
Amazon.com retrieval systems (2)


Assisting users in determining relevancy
Assisting users in determining relevancy retrieval systems (2)

Title

Abstract

Indexing

terms

Citation

data

Source: Barry, Carol L. 1998. Document representations and clues to document relevance. Journal of the American Society for Information Science 49(14):1293-1303.


Document representation research

How relevant are these? retrieval systems (2)

Document representation research

Title: Getting good grades in graduate school

Title: How to impress your advisor in graduate school

Titles

Title: Writing a dissertation

Title: The well-written graduate paper

Getting good grades in graduate school

The best way to get good grades is to study hard…

How to impress your advisor in graduate school

Never show up late for a meeting with your advisor…

Full

text

How relevant are these?

The well-written graduate paper

Before finalizing your topic do a preliminary search on…

Writing a dissertation

The first thing to do is to pick a topic that truly interests you…


Document representation research1
Document representation research retrieval systems (2)

How relevant are these?

Titles

Citation

data

Indexing

terms

Abstracts

Full

text

Full

text

Full

text

Full

text

How relevant are these?


Utility studies indications that user found relevant materials
Utility studies - Indications that user found relevant materials

  • Citation & abstract databases

    • User requests citations be formatted for printing

    • User requests citations be sent by e-mail

    • User downloads citations

  • Full-text databases

    • Pull up the full text

    • Print the article

    • Download the article to their Blackberry


Utility studies indications that user found relevant materials1

If user stops may not have found a relevant article materials

Utility studies - Indications that user found relevant materials

Search

Short

list

chocolate


Utility studies indications that user found relevant materials2
Utility studies - Indications that user found relevant materials

Search

Short

list

Modifies

search

View full

citation

data for

article

View full

text of

article

Download

or print

article

Assume that user found article relevant


Characteristics of searches that produce relevant materials
Characteristics of searches that produce relevant materials materials

  • Subject searching

  • Utilization of Boolean operators

  • Search modification

  • Increased time in display activities

  • User of greater number of databases

Cooper, Michael Dr. and Hui-Min Chen. 2001. Predicting the relevance of a library catalog search. Journal of the American Society for Information Science and Technology 52 (10):813-827.


Importance of abstract 1
Importance of abstract (1) materials

  • Indication as to depth/scope of the article

  • Delineates methodology--indication of reliability and validity

  • Gives indication as to content novelty

Authors studied leg-hair count variations of Drosophila in Kawainui Marsh

Random sampling in 40 sectors during March, June, September & December

Greater variation in June


Importance of abstract 2
Importance of abstract (2) materials

  • Basis for research may indicate recency

  • Delineation of results indicates "tangibility" (important, useful data)

American housing market was selected because it is always robust.

Authors concluded that American teenagers listen to rock music.


Types of abstracts
Types of abstracts materials

  • Indicative

  • Informative

  • Critical (evaluative)

(Not common in library databases)


Indicative abstract
Indicative abstract materials

Indicates what the document is about but doesn't report findings

Title: A review of the current literature on relevance.

Abstract: The author reviews the current literature on relevance.


Informative abstract
Informative abstract materials

Acts as a substitute for the document

Title: The effects of library school on the mental health of library students

Abstract: The authors performed longitudinal studies on 32 graduate students in 8 library and information science programs and found a significant increase in aberrant psychological traits over time.

(fictitious title and abstracts)


Abstract creation
Abstract creation materials

  • Author-produced

  • Vendor-added

  • Automated abstracting


Automated abstracting
Automated abstracting materials

  • Word counts

  • Remove stop words

  • Weight remaining words according to frequency

  • Search for sentences with highest density of most frequently-occurring words


1 word count
1. Word count materials

Title: Seasonal variations in the feral cat population of Fargo

the 81

is 68

a 56

to 42

cats 61

number 45

season 27

winter 11

summer 11

spring 11

fall 11

monthly 10

temperature 61

variation 12

food 10

availability 10

average 9

concept 7

per 8

over 9

immediate 5

implement 3

mortality 8

survival 9


2 eliminate stop words
2. Eliminate stop words materials

Title: Seasonal variations in the feral cat population of Fargo

the 81

is 68

a 56

to 42

cats 61

number 45

season 27

winter 11

summer 11

spring 11

fall 11

monthly 10

temperature 61

variation 12

food 10

availability 10

average 9

concept 7

per 8

over 9

immediate 5

implement 3

mortality 8

survival 9


3 rank by frequency
3. Rank by frequency materials

Title: Seasonal variations in the feral cat population of Fargo

cats 61

temperature 61

number 45

seasonal 27

variation 12

winter 11

summer 11

spring 11

fall 11

monthly 10

food 10

availability 10

average 9

survival 9

mortality 8

concept 7

immediate 5

implement 3


4 search for sentences with highest density of high frequency words
4. Search for sentences with highest density of high frequency words

Title: Seasonal variations in the feral cat population of Fargo

We found a significant seasonalvariation in the number of cats.

The highest number of cats are found in the summer, the lowest number of cats in the winter.


Automated abstract
Automated abstract frequency words

... The Children's Internet Protection Act (CIPA) sets conditions on public libraries' receipt of federal financial assistance for Internet access. ... It would not have been possible for the broadcasting station to limit the use of federal funds to all non-editorializing activities. ... The instant Court distinguished Velazquez, restricting its holding to situations in which the grantee is "pit[ted] . . . against the Government. ... " Justice Stevens asserted that the filtering condition was unconstitutional because it distorted the normal usage of library Internet terminals as sources of a wide array of information. ... A condition mandating Internet filters distorts this mission by "deny[ing] patrons access to constitutionally protected speech that libraries would otherwise provide. ...


Relevance and information overload
Relevance and information overload frequency words

In this age of information overload, tools to aid the user in determining relevance are increasingly critical.


ad