Metasearch: Selection Considerations

1. Metasearch: Selection Considerations NISO Metasearch Workshop October 30, 2003 Beth Forrest Warner Director, Digital Library Initiatives University of Kansas bwarner@ku.edu

2. First, some background on KU… • Carnegie Research 1 institution, ARL Library • Multi-campus: Lawrence, Medical Center, others • Approximately 28,000 students • Information Services division includes: • Library • IT units • Digital Library Initiatives • Approximately xxx databases, xxxx e-journals NISO Metasearch Workshop

3. KU’s metasearch environment • Metasearch implemented as part of full digital library system • ENCompass from Endeavor Information Systems • Production: November 2002 • 50+ databases (still adding) • Multi-protocol: http, Z39.50, XML (soon) • OpenURL linking (LinkFinderPlus) • Serials Solutions holdings NISO Metasearch Workshop

4. KU’s related systems environment • Integrated library management system: Voyager (Endeavor) • University portal: uPortal • Course management system: BlackBoard • Institutional repository system: DSpace • I/A/A: LDAP, local directory service, EZ-proxy NISO Metasearch Workshop

5. Selecting a metasearch system • Key points: • Define your audience, needs, & goals • Know your local environment • Assess integration requirements • Assess functionality requirements • Metasearch functions vs. broader portal functions • Understand the basic technology involved • Understand the range of players involved • Know your vendor(s) and their customers • Consider an RFI / RFP, take a test run NISO Metasearch Workshop

6. Define audience, needs, goals • Define your primary audience(s) • Students, faculty/staff, researchers, public, librarians • Local, remote • Define your needs & goals • Resource discovery / In-depth search • Core resources / Specialized resources • Commercial resources / Free resources • Local resources / Remote resources NISO Metasearch Workshop

7. Define local environment • Resources to search • Databases, vendors – who provides what • A&I databases, full-text, OPAC(s), websites, etc. • Licensing arrangements (unlimited, limited, pay per search) • Staff resources available • Position(s) / people available • Skill-sets • Authentication environment • ILMS vendor NISO Metasearch Workshop

8. Define integration needs • Stand-alone module vs. search system component • Related system(s) integration: • ILMS • Institutional portal • Specialized portals • Course management system • Institutional repository system • Harvesters NISO Metasearch Workshop

9. Define functionality requirements • General • Client • Searching & search results • Knowledge database • Authentication • Administration & vendor support Can be somewhat tricky to separate metasearch functions from application portal functions… NISO Metasearch Workshop

10. Metasearch application: Supports multiple standard and non-standard search protocols Enables access for novice and advanced users to heterogeneous targets with diverse metadata Supports standard metadata formats whenever possible Ensures reliable duplication of search results Links search results to full-text or other content delivery options whenever possible Unicode supported Provides use statistics Application portal: Helps users identify and select one or more targets most useful to their research Helps user understand choices for specifying search queries, manipulating and exporting search results, and navigating among portal functions Supports personalized user profiles (for session parameters, target selection, saved queries, etc.) Respects laws and institutional policies relating to user privacy Users can begin evaluating partial search results before complete search results are received Provides use statistics Functionality requirements: General NISO Metasearch Workshop

11. Functionality requirements: Client • Supports a web client in a browser-neutral environment on a variety of hardware platforms running various operating systems • Complies with Section 508 requirements, if applicable • Supports locally-customizable screen displays • Supports different user communities and “levels” of searchers (expert, skilled, novice) • Allows local branding, labeling, and search session defaults • Interface is modular; underlying system is backwardly compatible to allow for gradual implementation of new system functionality without losing local customization NISO Metasearch Workshop

12. Metasearch application Search results are viewable by target and as merged search results. Search results are clearly understandable. It is clear how search results returned from different targets are ordered, as well as how and when search results from different targets are de-duped Results (records) are parsed to allow for easy and consistent application manipulation and display Application portal Users can easily perform information-seeking tasks in an iterative manner Various sort options are available Users can easily save and export selected search result records from one or more queries without additional log-in requirements Functionality requirements: Searching & search results NISO Metasearch Workshop

13. Metasearch application Searching options include more than “keyword anywhere” support. Users can, for example, search keywords in specified fields or browse a “left-match” display of fields in an ordered list. Searchers can apply “search relationship identifiers” (such as Boolean operators, truncation, wildcards, and nesting) in a single advanced search query It is clear which search result records are linked to accessible digital content Application portal Between search sessions, users can save, modify, and delete both search queries and the targets associated with these queries Subject-based retrieval tools to enhance queries can be incorporated Functionality requirements: Searching & search results (2) NISO Metasearch Workshop

14. Functionality requirements:Knowledge Database Metasearch application • Vendor maintains basic descriptive metadata and configuration information; translators are maintained / updated frequently; changes to remote targets are monitored, incorporated quickly • Knowledge database is updateable online and through batch processes using update requests expressed in an open format • Contents of the knowledge database are exportable in an open format • Institutions can access vendor-supplied descriptive metadata and configuration information on an ongoing basis • Institutions can add local descriptive metadata and configuration information to knowledge database records; this information is retained when records are refreshed • The knowledge database can integrate metadata from multiple external sources NISO Metasearch Workshop

15. Metasearch application Different authentication methods to individual targets are supported (IP address, ID/password, specialized URLs, etc.) Application portal The architecture allows the use of different methods to authenticate users and recognizes various user roles or classes of users Proxy services are supported (ex. EZ-proxy) Allows administrators to configure the set of targets and portal functionalities displayed to an authenticated user based on information about the user and role/class Functionality requirements:Authentication NISO Metasearch Workshop

16. Metasearch application Provides access to Z39.50 target servers and one or more mechanisms for accessing non-Z39.50 target servers. No effective system limitations on the number of targets or resources within these targets No effective system limitations on the number of simultaneous active retrieval sessions. The system supports a substantial portion of institutional users Application portal Web-accessible administrative interfaces are secure, and private information (such as passwords and user profile data) is stored in an encrypted or secure environment The application interoperates with related library systems and applications the application supports secure user logins and enables institutions to determine what session-specific information is stored and purged The application can control access for different user roles/classes according to target-specific access control agreements Functionality requirements:Administration NISO Metasearch Workshop

17. Functionality requirements:System reliability, vendor support • The application provides for complete and robust recovery in case of operating system failure. Provides less than x% downtime. • Tools and comprehensive machine-readable documentation are available, with permission given to excerpt and adapt documentation for local use • Vendor can clearly define staff skill-sets required to implement / operate system • Vendor provides installation services and training, as requested • Vendor is available 24x7 for problem resolution • Vendor procedures and response mechanisms for handling bug reports and enhancement requests are clearly defined NISO Metasearch Workshop

18. Understand the basics • Portal vs. Metasearch functionality • Differences between search protocols • Whose search engine does what • Resource translators – who, what, when • The reality of de-duping results • Is relevance ranking really relevant? • Software vs. service • What you are really buying • Is a single-vendor solution critical? • Who handles what level of security NISO Metasearch Workshop

19. Understand the players: who’s on first…? authentication Search Portal: Federated Searching, etc. Resource Translators Searching, Authentication Z39.50, XML, http Resources Remote OPAC Local Commercial NISO Metasearch Workshop

20. Who provides search translators: In-house External partner Support staff Development staff Stability, financing Look beyond current functionality – consider development path, upgrade cycle Track record ILMS vendor? Mix of customers Types of institutions Similar sizes, range Similar resource mix User group Know your vendor(s), customer base NISO Metasearch Workshop

21. Gathering information: Use an RFI / RFP? • IMLS vendor vs. other federated search vendor • Purchasing / contracting requirements • Metasearch vs. portal – define scope • Keep it simple, focus on critical points • Don’t forget who your primary audience is • (hint: its not really the librarians…) • LC’s requirements list is only 11 pages… • Standards are critical NISO Metasearch Workshop

22. Gathering Information:Take a test drive • On-site demonstrations, include users • Hands-on test environment • Searching, include users • Administrative functions • Look at other installations in production • Talk to current customers NISO Metasearch Workshop

23. Metasearching: final thoughts • Remember: • Its a young and rapidly changing technology • Its a volatile environment • There are multiple players involved • The competition is Google • Its only one way to get users to your resources • Resource discovery vs. in-depth searching • Be flexible • You are buying a service as well as a system NISO Metasearch Workshop

24. Some additional resources • The Truth About Federated Searching, Information Today, 20(9), October 2003, http://www.infotoday.com/it/oct03/hane1.shtml • Luther, Judy, Meta Search Engines – The Librarians’ Answer to Google?,The Charleston Advisor, 4(4), April 2003, http://www.charlestonco.com/features.cfm?id=123&type=np • Luther, Judy, Trumping Google? Metasearching’s Promise, Library Journal, October 1, 2003, http://libraryjournal.reviewsnews.com/index.asp?layout=articlePrint&articleID=CA322627 • List of Portal Application Functionalities for the Library of Congress, First Draft for Public Comment, July 15, 2003, issued by the Library of Congress Portals Applications Issues Group,http://www.loc.gov/catdir/lcpaig/portalfunctionalitieslist4publiccomment1st7-22-03revcomp.pdf • Tennant, Roy, The Right Solution: Federated Search Tools, Library Journal, June 15, 2003, http://libraryjournal.reviewsnews.com/index.asp?layout=articlePrint&articleID=CA302427 • Blossom, John, Federated Foxholes: Moving Search Beyond Library Science to Content Science, Shore, October 6, 2003, http://shore.com/commentary/newsanal/items/20031006federated.html NISO Metasearch Workshop