Converting local portal data to myspace
This presentation is the property of its rightful owner.
Sponsored Links
1 / 26

Converting Local Portal Data to MySpace PowerPoint PPT Presentation


  • 76 Views
  • Uploaded on
  • Presentation posted in: General

Converting Local Portal Data to MySpace. Anne L. Highsmith, Consortia Systems Coordinator & Prabha Acharya, Programmer Analyst II Texas A&M University. Points to Cover. Background Data Conversion Issues Testing on a Cloned Database Conversion Process Results and Future Plans.

Download Presentation

Converting Local Portal Data to MySpace

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Converting local portal data to myspace

Converting Local Portal Data to MySpace

Anne L. Highsmith, Consortia Systems Coordinator &

Prabha Acharya, Programmer Analyst II

Texas A&M University


Points to cover

Points to Cover

  • Background

  • Data Conversion Issues

  • Testing on a Cloned Database

  • Conversion Process

  • Results and Future Plans

Converting Local Portal Data to MySpace ELUNA 2007


Background 1

Background (1)

  • TAMU had a local Electronic Resources Locater (ERL) function to list ejournals and databases.

  • Users searched ERL and clicked on links to access eresources

  • Users could store the links in their MyPortal record on the library’s website

Converting Local Portal Data to MySpace ELUNA 2007


Background 2

Background (2)

  • Metalib went into production at TAMU in January 2006 as Search Now.

  • TAMU had discussions with Ex Libris staff as to how MyPortal links could be converted to Metalib

    • First proposal, for Ex Libris staff to do a major conversion, not feasible

    • Decided to wait for Metalib 4.0 and X-Portal APIs to store data to MySpace

Converting Local Portal Data to MySpace ELUNA 2007


Rude awakening

Rude awakening!

  • Fall of 2006, Ex Libris staff informed TAMU there would be no API to store to MySpace

  • Alternatives

    • Margaritas?

    • Convert them ourselves?

Converting Local Portal Data to MySpace ELUNA 2007


Conversion options

Conversion options

  • Assumed conversion would have to be:

    • User-initiated

    • Title by title

  • Discovered a way do the conversion directly, without involving the end user

Converting Local Portal Data to MySpace ELUNA 2007


Data conversion issues

Data Conversion Issues

  • Patrons

  • Journals

  • Databases

Converting Local Portal Data to MySpace ELUNA 2007


Data conversion issues patrons

Data Conversion Issues - Patrons

  • How to match the MyPortal patron with the Metalib user?

    • MyPortal patrons identified by uin (University ID Number)

    • Our Metalib implementation uses uin as user login

    • Needed to match the uin with the z312_source_id

Converting Local Portal Data to MySpace ELUNA 2007


Data conversion issues journals

Data Conversion Issues - Journals

  • How to match MyPortal journals with Metalib?

    • Basic journal data stored in z11 table

      • Wrote Perl script (journals.pl) to extract journal identifier and issn from z11

    • Used MS Access to match MyPortal data, keyed by uin & issn, with Metalib data

    • User journal data stored in z312_jnl_001

      • Wrote Perl script (journals_by_patron.pl) to extract z312_source_id (uin) and journal identifier from Metalib user record in z312 table

Converting Local Portal Data to MySpace ELUNA 2007


Data conversion issues databases

Data Conversion Issues - Databases

  • How to match MyPortal databases with Metalib?

    • During creation of Metalib IRDS, stored the ERL key in the IRD cataloger note

      • Loaded IRD extract into MS Access

    • Matched Metalib user extract with IRD extract

    • Basic database data stored in z122 table

      • Wrote Perl script (databases_by_patron.pl) to extract z312_source_id (uin) and Metalib identifier (TEX #)

Converting Local Portal Data to MySpace ELUNA 2007


Data conversion issues final

Data Conversion Issues - Final

  • As a result of matching ERL data and Metalib data, created a file that contained 1 or 2 entries for each user whose data had to be converted:

    • Tab-separated record beginning with uin and list of database id numbers

    • Tab-separated record beginning with uin and list of journal id numbers

Converting Local Portal Data to MySpace ELUNA 2007


Testing on a cloned database

Testing on a Cloned Database

Converting Local Portal Data to MySpace ELUNA 2007


Testing cloning the database 1

Testing – Cloning the database (1)

  • TAMU has a test Metalib DB on separate server

  • Cloned production DB to test DB so could test in a realistic environment

    • mpsync - documented in Appendix D in the System Administrators Guide & Knowledge Base entry #1240.

    • exp_current_library & imp_current_library, documented in 7.8 in the SysAdmin Guide

  • Further doc – “Working with Test/Failover Installations of MetaLib”

Converting Local Portal Data to MySpace ELUNA 2007


Cloning export from production

Cloning – Export from Production

  • Created files on prod for export to test

    • Run mpsync to copy IRDs

      • TAMU has approx. 4500 IRDS

      • Elapsed time – 58 sec.

    • Run export current library for vir00

      • TAMU has 32,000+ user records

      • Elapsed time – 1 hr.

    • Run export current library for jnl01

      • TAMU has approx. 44,000 journal records

      • Elapsed time – 2:25 min.

Converting Local Portal Data to MySpace ELUNA 2007


Cloning import into test

Cloning – Import into Test

  • Import IRDS created by mpsync

    • Elapsed time – 3 hrs. 688 new records added

    • Matched existing keys. Appeared to create some duplicates

  • Import User Data

    • Elapsed time – 40 min.

    • Drops & re-adds tables

  • Import Journal Data

    • Elapsed time – 3 min.

    • Drops & re-adds tables

Converting Local Portal Data to MySpace ELUNA 2007


Conversion process

Conversion Process

  • Quality control

  • File preparation

  • Loading new patrons

  • Convert databases and ejournals

Converting Local Portal Data to MySpace ELUNA 2007


Conversion quality control

Conversion – Quality Control

  • Wrote a Perl script (pre_migration.pl) to collect stats and set up QC measures. For each user it reports:

    • Users who don’t have Metalib record

    • # of databases and journals already in Metalib

    • # of databases and journals to convert

    • # of users that have more than maximum number of databases or ejournals

Converting Local Portal Data to MySpace ELUNA 2007


Conversion file preparation

Conversion – File Preparation

  • Testing demonstrated that It takes about 3 seconds per resource to be migrated.

  • With about 13000 resources that would be about 10 hours

  • So, wrote Perl script (breakup_myportal_file.pl) to break the input file into several files of about 250 records each.

Converting Local Portal Data to MySpace ELUNA 2007


Conversion loading new patrons

Conversion – Loading new patrons

  • Since TAMU uses external authentication rather than patron load, some older MyPortal patrons not in Metalib database

  • Using file created during QC stage, create_xml_patron_load.pl formats user records for Metalib patron load utility

    • Set expiry date to 1 year in future

    • Based on a review of records created on that date, approx. 65 have since logged in.

Converting Local Portal Data to MySpace ELUNA 2007


Conversion convert databases and ejournals 1

Conversion - Convert databases and ejournals (1)

  • Run databases_by_patron.pl to get current lists from user records for comparison

Converting Local Portal Data to MySpace ELUNA 2007


Conversion convert databases and ejournals 2

Conversion - Convert databases and ejournals (2)

  • Using save_to_bookbag.cgi, for each user

    • Log in to Metalib

    • Send request to Metalib pds to get pds handle

    • Send request to Metalib, using just obtained pds handle, to get session id

    • Construct url of the form

      • http://metalibhost.edu/V/sessionid?pds_handle=pdshandle&func=find-db-add-res&resource=resourceid

Converting Local Portal Data to MySpace ELUNA 2007


Conversion convert databases and ejournals 3

Conversion - Convert databases and ejournals (3)

  • Set session id in cookie

  • For each database

    • Check to see if it duplicates db already stored

    • If not, store db to user’s clipboard

  • For each journal

    • Store to user’s record

Converting Local Portal Data to MySpace ELUNA 2007


Conversion convert databases and ejournals 4

Conversion - Convert databases and ejournals (4)

  • Do QC to make sure that all of the appropriate databases and ejournals are stored

  • Exit the session

  • Error logging revealed that for some users, 1 database or ejournal was not stored – these were re-run as an added input file

Converting Local Portal Data to MySpace ELUNA 2007


Final results

Final results

  • Processing all files took most of the week, off and on

  • Each file, limited to 250 records, took about 1 hour to process

  • Only post-production problem had nothing to do with conversion – ezproxy error.

  • Once users have logged into library portal, can now go directly to My Databases or My Ejournals list within Metalib

Converting Local Portal Data to MySpace ELUNA 2007


Future plans

Future plans

  • Conversion of Voyager My Bookbag to e-shelf

Converting Local Portal Data to MySpace ELUNA 2007


Contact download info

Contact & download info

Anne L. Highsmith

Consortia Systems Coordinator

Texas A&M University

[email protected]

URL to download perl code, cgi, etc.

http://libraryasp.tamu.edu/users/ahighsmi/presentations/save_to_bookbag.zip

Converting Local Portal Data to MySpace ELUNA 2007


  • Login