Users and uses of bibliographic data the promise and paradox of bibliographic control
1 / 41

Users and Uses of Bibliographic Data: The Promise and Paradox of Bibliographic Control - PowerPoint PPT Presentation

  • Uploaded on

Users and Uses of Bibliographic Data: The Promise and Paradox of Bibliographic Control. NCSU Case study: Faceted Navigation. Andrew K. Pace Head, Information Technology NCSU Libraries. Agenda. NCSU’s Endeca-powered catalog Data Reality Check Relevance ranking in online catalogs

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Users and Uses of Bibliographic Data: The Promise and Paradox of Bibliographic Control' - ostinmannual

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Users and uses of bibliographic data the promise and paradox of bibliographic control

Users and Uses of Bibliographic Data: The Promise and Paradox of Bibliographic Control

NCSU Case study: Faceted Navigation

Andrew K. Pace

Head, Information Technology

NCSU Libraries


  • NCSU’s Endeca-powered catalog

  • Data Reality Check

  • Relevance ranking in online catalogs

  • Statistics: what are patrons doing?

  • The Metadata Paradox (in 3 parts)

  • A brief wish-list for the future of bibliographic control

Rumsfeld s law of bibliographic control
Rumsfeld’s Law of Bibliographic Control

You search the data you have, not the data you wish you had.

Existing catalogs are hard to use
Existing catalogs are hard to use

  • Known item searching works pretty well (sometimes), but …

  • Lots of topical searches and poor subject access

    • keyword gives too many or too few results – leads to general distrust among users

    • authority searching is under-utilized and misunderstood

  • Relevance = system sort order

  • Impossible to browse the collection

  • Unforgiving on spelling errors, stemming

  • Response time doesn’t meet expectations of web-savvy users

Valuable metadata is buried
Valuable metadata is buried

  • Subject headings are not leveraged in keyword searching

    • they should be browsed or linked from, not searched

  • Data from the item record is not leveraged

    • should be able to easily filter based on user’s changing requirements using item type, location, circulation status, popularity

In a nutshell
In a nutshell…

"Most integrated library systems, as they are currently configured and used, should be removed from public view."

- Roy Tennant, CDL

What s the big picture
What’s the big picture?

  • Improve the quality of the library catalog user experience

  • Exploit our existing authority infrastructure (aka make MARC data work harder)

  • Build a more flexible catalog tool that can be integrated with discovery tools of the future.

This gen search tools
“This-Gen” search tools

  • Proving that it’s possible to improve the search experience beyond the functionality that traditional online catalogs have supported.

What is endeca
What is Endeca?

  • Software company based in Cambridge, MA

  • Search and information access technology provider for a number of major e-commerce websites

  • Developers of the Endeca Information Access Platform

Why endeca
Why Endeca?

  • Customized relevance ranking of results

  • Better subject access by leveraging available metadata (including item level data!) through facets

  • Improved response time

  • Enhanced natural language searching through spell correction, etc.

  • True browse

Data reality check part i
Data Reality Check: Part I

  • Our Integrated Library System has ~80 MARC fields and subfields indexed in its keyword index

  • 33 additional indexed fields (29% !) are not publicly displayed

  • Displayed fields use 37 different labels

Users and uses of bibliographic data the promise and paradox of

Data Reality Check: Part II

  • A lot of those same fields are indexed in Endeca (just much more quickly and efficiently)

  • ~50 MARC fields are indexed

  • New catalog has ~37 Properties and 11 Dimensions derived from ~160 MARC fields and subfields

Data reality check part iii
Data Reality Check: Part III

  • Simple data is the best

    • MARC4J to convert MARC into flat files

    • Lots of stripping of punctuation…ugh

    • Perl to update files

Relevance ranking
Relevance Ranking

  • TF/IDF alone is inadequate for determining relevance order of bibliographic metadata

  • The Endeca MDEX Engine offers NCSU alternatives

    • Matching techniques: matchall, matchany, matchboolean, matchallpartial

  • A suite of relevance ranking options are applied to Boolean-type searches

Relevance ranking cont
Relevance Ranking (cont.)

  • Individual RR modules are combined and prioritized according to our specifications to form an overall RR strategy, or algorithm

  • Current strategy includes seven modules,

    • 5 of which rank results dynamically on things like: phrase, rank of the field in which term appears, weighted frequency

    • 2 final rules provide static ordering based on publication date and aggregate circulation totals

Relevance ranking challenge and promise
Relevance Ranking: Challenge and Promise


  • Uncharted territory required a best-guess approach

  • More experimentation required with “matchallpartial” to provide an interface intuitive enough for users to know what is happening


  • Having technology nimble enough to support experimentation with indexing and relevance strategies

Users and uses of bibliographic data the promise and paradox of


What patrons are doing

Users and uses of bibliographic data the promise and paradox of

July 06 – Jan 07



Most popular dimension values


New:New (92,037)

Format:Book (40,183)

Availability:Available (33,125)

Library: D.H. Hill (33,091)

Language: English (22,668)

Format: eBook (21,177)


LC Class: Q-Science (25,277)

Subject|Region: US (20,954)

Subject|Topic: History (20,861)

LC Class: T-Technology (16,951)

LC Class: H-Soc. Sci. (16,345)

Subject Topic: Bioethics (12,933)

Most Popular Dimension Values

July 06 – Jan 07

Out of 765,170 Navigation Requests

Plugging holes in the system
Plugging Holes in the System

  • Natural language problem

    • LCSH=United States—History—Revolution, 1775-1783

    • keyword=revolutionary war (834 hits)

    • Subject keyword=“United States—History—Revolution, 1775-1783” (3081 hits)

  • Facets taken out of the free-floating and hierarchical context of LCSH can be misleading

  • I’ve followed many a tag cloud, but assuming that browsing is still popular, how does one browse keywords?

Paradox 1
Paradox #1

We finally have interesting discovery tools that make use of bibliographic data in ways that show us that the data are not completely adequate for use with the new discovery tools.

Subject keywords
“Subject Keywords”

  • “[from Recommendations:]

    ….Abandon the attempt to do comprehensive subject analysis manually with LCSH in favor of subject keywords; urge LC to dismantle LCSH”

    -- Karen Calhoun, The Changing Nature of the Catalog and its Integration with Other Discovery Tools, report prepared for the Library of Congress, March 2006

Paradox 2
Paradox #2

“Subject keywords” should replace the controlled vocabulary from which the keywords themselves are most easily derived.

Let’s build bridges between the mountains of bibliographic description so that we can tear down the mountains.

If not lcsh then what
If not LCSH, then what?

(with some help from colleague Charley Pennell, Principle Cataloger for Metadata, NCSU)

  • Dissertation Abstracts Model

    • Constrained list of subject terms

    • Dissertations are at the edge of scholarship, so granular pre-existing thesaurus is inadequate

    • Vocabulary cannot be “thwarted” by authors

  • Social Tagging Model

    • Human intervention times n users

    • New difficulty of matching tags between creators

    • Contrary to the “literary warrant” that requires a hierarchical thesaurus to differentiate thousands of titles from one another

  • Full Text Model

    • Good for currency: hypothetically uses the most contemporary language

    • Harder for research: some sort of citation analysis required; but arguably could be solved computationally

    • The question of scale

Paradox 3
Paradox #3

Computational (e.g. non-human mediated) creation of subject-based facets will work perfectly once all the full text of every work is available in electronic format.

What does a search and retrieval system for 50 million books and 50 million articles look like?

G ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooogle

Bibliographic control wish list
Bibliographic Control Wish List

  • A classification or subject thesaurus system that enables faceted navigation

  • A work identifier for books and serials

  • Something other than LC Name Authority for “organizations”

  • Physical descriptions that help libraries send books to off-site shelving and to patron’s mailboxes

  • Something other than MARC in which to encode all of the above

  • Systems that can actually use the encoding

Paradox 4 the ultimate paradox
Paradox #4: The Ultimate Paradox

“You’re damned if you do and you’re damned if you don’t.”

- Bart Simpson


  • NCSU project site (includes these slides):


  • Andrew K. Pace

    • Head, Information Technology, NCSU Libraries