Forward in Reverse - PowerPoint PPT Presentation

Forward in reverse l.jpg
Download
1 / 73

  • 248 Views
  • Updated On :
  • Presentation posted in: Shopping

Forward in Reverse. A Gentle Overview Of Forward System Architecture Eric, Mike & Steve – WiLSWorld 2010. Outline. Intro to Forward with Demo Batch Processing (Backend) Web Application (Frontend) Challenges Q&As throughout. Intro & Demo. http://forward.library.wisconsin.edu.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Forward in Reverse

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Forward in reverse l.jpg

Forward in Reverse

A Gentle Overview Of Forward System Architecture

Eric, Mike & Steve – WiLSWorld 2010


Outline l.jpg

Outline

  • Intro to Forward with Demo

  • Batch Processing (Backend)

  • Web Application (Frontend)

  • Challenges

  • Q&As throughout


Intro demo l.jpg

Intro & Demo


Http forward library wisconsin edu l.jpg

http://forward.library.wisconsin.edu


Batch processing l.jpg

Batch Processing


Slide6 l.jpg

We have gobs & gobs of data.


Slide7 l.jpg

1) Extract it


Slide8 l.jpg

1a) ILS Data


Slide13 l.jpg

Sort, Deduplicate, Merge


Slide14 l.jpg

Antique Style Key

By Stars*Go*Blue

http://www.flickr.com/photos/artbydebora/1406682449/


Slide15 l.jpg

Common Identifier = OCLC Number


Slide16 l.jpg

  • Catalog Extract Processing Details

  • 14 Voyager Instances

  • 13M MARC bibliographic records extracted

  • Approximately 14 hours

  • Local C code

  • Sorted, deduplicated and merged output:

  • 8M records

  • 10GB Raw MARC data


Slide17 l.jpg

  • Why Merge?

  • URLs

  • Formats

  • Holdings


Slide18 l.jpg

1b) Digital Collection Data


Slide20 l.jpg

  • Fedora Extract Processing Details

  • 1 Fedora Repository

  • 13K “First Class” XML Objects extracted

  • Approximately 4 hours

  • Repository query language

  • XML output:

  • METS XML package

    • Structural XML

    • MODS Bibliographic XML

  • 41MB XML data


Slide21 l.jpg

2) Index it


Slide22 l.jpg

We take raw library data and process it with MARC/XML parsing tools and local parsing rulesin order to build a Solr search index.


Slide23 l.jpg

  • Raw data (MARC & METS XML)

  • Parsing libraries (Java code: marc4j, SAXParser)

  • Local code that defines parsing rules

  • Solr index


1 raw data l.jpg

1. Raw data


Slide25 l.jpg

LEADER 02000cam a22003734a 45 0

001 6939454

005 20051208125417.0

008 051104s2004 enka $b 001 0 eng

010 $a 2003045349

035 $a (OCoLC)ocm52165958

040 $aDLC $c DLC $d XMA $d BAKER $d UKM

015 $a GBA430162 $2 bnb

016 7$a 012906573 $2 Uk

020 $a 0754605175 (alk. paper)

024 $a 99811375970

042 $a pcc

049 $a GZMA

050 00$a B3376.W564 $b W55355 2004

082 00 $a 111/.85/092 $2 21

245 00 $a Wittgenstein, aesthetics, and philosophy / $c edited by Peter B. Lewis.

260 $a Aldershot, Hants, England ; $a Burlington, VT : $b Ashgate, $c c2004.

300 $a xii, 255 p. : $b ill. ; $c 24 cm.

440 0 $a Ashgate Wittgensteinian studies

505 0$a Wittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / Ilham Dilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. Hagberg

504 $a Includes bibliographical references (p. 235-247) and index.


Slide26 l.jpg

02000cam a22003734a 45 001000800000005001700008008004100025010001700066035002300083040003000106015001900136016001800155020002800173024001600201042000800217049000900225050002800234082002000262245007400282260006800356300003400424440003600458505081100494504006401305600005001369700002501419938007101444945001901515946003001534946001301564947002101577948001601598994001201614693945420051208125417.0051104s2004 enka b 001 0 eng a 2003045349 a(OCoLC)ocm52165958 aDLCcDLCdXMAdBAKERdUKM aGBA4301622bnb7 a0129065732Uk a0754605175 (alk. paper) a99811375970 apcc aGZMA00aB3376.W564bW55355 200400a111/.85/09222100aWittgenstein, aesthetics, and philosophy /cedited by Peter B. Lewis. aAldershot, Hants, England ;aBurlington, VT :bAshgate,cc2004. axii, 255 p. :bill. ;c24 cm. 0aAshgate Wittgensteinian studies0 aWittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / Ilham Dilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. Hagberg aIncludes bibliographical references (p. 235-247) and index.10aWittgenstein, Ludwig,d1889-1951xAesthetics.1 aLewis, Peter,d1947- aBaker & TaylorbBKTYc99.95d99.95i0754605175n0004227086sactive c1d89087961587 a714694b2005-11-23c81.86 c99.95d1 aHEUR 4801bm,stk aSCNd348032 a92bGZM


2 marc xml parsing libraries l.jpg

2. MARC/XML parsing libraries


Slide31 l.jpg

02000cam a22003734a 45 001000800000005001700008008004100025010001700066035002300083040003000106015001900136016001800155020002800173024001600201042000800217049000900225050002800234082002000262245007400282260006800356300003400424440003600458505081100494504006401305600005001369700002501419938007101444945001901515946003001534946001301564947002101577948001601598994001201614693945420051208125417.0051104s2004 enka b 001 0 eng a 2003045349 a(OCoLC)ocm52165958 aDLCcDLCdXMAdBAKERdUKM aGBA4301622bnb7 a0129065732Uk a0754605175 (alk. paper) a99811375970 apcc aGZMA00aB3376.W564bW55355 200400a111/.85/09222100aWittgenstein, aesthetics, and philosophy /cedited by Peter B. Lewis. aAldershot, Hants, England ;aBurlington, VT :bAshgate,cc2004. axii, 255 p. :bill. ;c24 cm. 0aAshgate Wittgensteinian studies0 aWittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / Ilham Dilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. Hagberg aIncludes bibliographical references (p. 235-247) and index.10aWittgenstein, Ludwig,d1889-1951xAesthetics.1 aLewis, Peter,d1947- aBaker & TaylorbBKTYc99.95d99.95i0754605175n0004227086sactive c1d89087961587 a714694b2005-11-23c81.86 c99.95d1 aHEUR 4801bm,stk aSCNd348032 a92bGZM


3 local code l.jpg

3. Local code


Slide38 l.jpg

4.

http://lucene.apache.org/solr/


What is solr l.jpg

What is Solr?

An XML API over a Lucene search index.


Slide42 l.jpg

Access to Raw Formats

Raw MARC stored for Merged record

Live calls made to Fedora

web services


Data refresh l.jpg

Data Refresh

Bibliographic: weekly

Circulation status: nightly


Slide45 l.jpg

For more information, see

http://sdg.library.wisc.edu/blog/2010/03/03/solr-marc-indexing-based-on-diffs/


Web application l.jpg

Web Application


Frontend l.jpg

Frontend?

  • (X)HTML

  • JavaScript

  • Cascading Style Sheets

  • Design

    • Information Architecture

    • User experience

    • Chrome (images, icons, pretty)


Forward colophon l.jpg

Forward Colophon

  • ActiveRecordBaseWithoutTable (Rails plugin)

  • Apache

  • Blacklight (Rails plugin)

  • Blueprint CSS

  • Bookreader (jQuery)

  • Capistrano

  • Crontab

  • Engines (Rails plugin)

  • Fedora

  • Freebase API

  • GeoIP (Ruby gem)

  • Google Books API

  • Haml (Rails plugin)

  • Happymapper (Ruby gem)

  • HathiTrust API

  • jQuery

  • Ken (Ruby gem)

  • LowPro (Prototype JS)

  • MARC4J

  • Passenger (modrails)

  • Prototype JS

  • PostgreSQL

  • Raphael

  • Ruby on Rails

  • Shibboleth

  • Subversion

  • Solr / Lucene

  • Summon (Ruby gem)

  • UW-Madison Libraries Staff Directory API

  • UWDC (Rails plugin)

  • Voyager API

  • Tender love and attention


Campus affiliation l.jpg

Campus Affiliation

Users localize to a school, allows us scope many features to their campus.

GeoIP RubyGem

Match IP addresses with physical locations.

Raphaël—JavaScript Library

“Small JavaScript library that should simplify your work with vector graphics on the web”.


Rapha l l.jpg

Raphaël

SVG elements, like the circles and squares in the Forward splash page, can be treated as XHTML elements allowing us to manipulate them with JavaScript and CSS.

http://raphaeljs.com/


Campus homepage l.jpg

Campus Homepage

Forward application stack:

Apache+Passenger (modrails)

Ruby on Rails

PostgreSQL

Apache Solr


Apache passenger l.jpg

Apache+Passenger

Phusion Passenger is an Apache module, which makes deploying Ruby and Ruby on Rails applications on Apache a breeze.

http://www.modrails.com/


Ruby on rails l.jpg

Ruby on Rails

“Ruby on Rails is an open-source web framework that’s optimized for programmer happiness and sustainable productivity.”

http://rubyonrails.org/


Postgresql l.jpg

PostgreSQL

“PostgreSQL is a powerful, open source object-relational database system. It has more than 15 years of active development and a proven architecture that has earned it a strong reputation for reliability, data integrity, and correctness.”

http://www.postgresql.org/


Apache solr l.jpg

Apache Solr

“Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling.”

http://lucene.apache.org/solr/


Results l.jpg

Results


Results three columns l.jpg

Results – Three columns


Results data sources l.jpg

Results – Data sources


Results facets solr l.jpg

Results – Facets – Solr


Results solr postgresql apis l.jpg

Results – Solr + PostgreSQL + APIs


Results context apis l.jpg

Results – Context – APIs


Results three main columns l.jpg

Results – Three main columns


Results css grid l.jpg

Results – CSS grid


Blueprint l.jpg

Blueprint

“Blueprint is a CSS framework, which aims to cut down on your development time. It gives you a solid foundation to build your project on top of, with an easy-to-use grid, sensible typography, useful plugins, and even a stylesheet for printing.”

http://blueprintcss.org/


Show book l.jpg

Show – Book


Show image l.jpg

Show – Image


Show full text book l.jpg

Show – Full Text Book


Show view full text book l.jpg

Show – View Full Text Book


Bookreader l.jpg

BookReader

“The Internet Archive BookReader is used to view books from the Internet Archive online and can also be used to view other books. ”

http://github.com/openlibrary/bookreader


Challenges l.jpg

Challenges


Challenges72 l.jpg

Challenges

  • Merging MARC, METS extracts

  • Batch processing time (Time/CPU constraints)

  • Page level indexing (Bookviewer - memory/disk constraints)

  • Voyager API

  • Organization challenges

    • big project, small shop

    • dealing with vendor silos

    • multiple cataloging standards

    • quality of services challenges


Thanks l.jpg

Thanks!


  • Login