Searching for NZ Information in the Virtual Library

1. Searching for NZ Information in the Virtual Library Alastair G Smith School of Information Management Victoria University of Wellington

2. Overview • Search engines: local vs global • Search engines: limitations • Searching for NZ info: effective strategies • Information Quality on the Web • Making NZ info more accessible: the role of librarians

3. NZ information online • Online access can mean that US, European Information is easier to access than NZ • E.g. Dialog • However Internet provides accessible infrastructure for making NZ information available • E.g. Knowledge Basket

4. Search tool definitions • Directories: resources categorised by human beings: e.g. • Yahoo! • Te Puna Web Directory • Search engines: automatically created databases of web pages, searchable by keyword e.g. • Google, SearchNZ

5. Role of Search Engines • Convenient, fast, usually find some information (if not most relevant) • Most people turn to a search engine first (GVU user survey: 85%) • For NZ Information we have a choice: • Global search engines, e.g. Google • Local search engines, e.g. SearchNZ

6. Comparing NZ and global search engines • Experiment compared NZ, global and metasearch engines • Test questions on NZ topics • Compared relative recall

7. Global Search Engines • AlltheWeb/FAST http://www.alltheweb.com/ • Google http://www.google.co.nz/ • HotBot http://hotbot.lycos.com/ • Altavista http://nz.altavista.com/

8. Local Search Engines • SearchNZ http://www.searchnz.co.nz/ • SearchNow http://www.searchnow.co.nz (no longer exists) • NZExplorer http://nzexplorer.co.nz/

9. Metasearch engines • Excite http://www.excite.com/ • Vivisimo http://vivisimo.com/ • Surfwax http://www.surfwax.com/

10. Examples of test questions • A description and image of the Maori flag • Information about the Otago Central Rail Trail • Information on the payment of British pensions in NZ

11. Recall • Recall: proportion of possible relevant documents found in search, e.g. • 100 relevant documents in database • Search finds 20 relevant documents • Recall is 20%

12. Problems in using recall to evaluate search engines: • Don’t know total number of relevant documents on Web • Ranking: Is document “found” if it appears in first 10, first 20…?

13. Relative Recall Pool results of search engines A, B, C: approximates to all relevant documents B A C

14. Recall in NZ search engine experiment • “First 20 relative recall” • Noted URLs of relevant documents found in first 20 hits for each search engine • Pooled results for all search engines • Used pooled list as approximation of all relevant documents

15. Recall results

16. Points arising from recall results • Only one local search engine equalled global search engines • No search engine found over half of relevant documents • Metasearch engines did not outperform standalone search engines

17. Comparison with 2000

18. Factors affecting performance of NZ search engines • Global search engines have similar or larger coverage of .nz sites • NZ search engines have less sophisticated search features • 36% of sites relevant to NZ topics were outside .nz domain • Global search engines update more rapidly

19. Overlap of search engine hits

20. Implications of overlap results • Most sites only found by one search engine • Few sites found by 7 or more search engines • Little overlap • Comprehensive searches require several search engines

21. Why aren’t metasearch engines better? • Metasearch engines select a few top ranked items from each search engine list • Search engine ranking imperfect • Looking at more results from one search engine may be as useful as looking at a few from each • Metasearch engines use “lowest common denominator” search • But can be useful for specific terminology

22. Limitations of Search Engines for finding NZ information • “hidden web” • How does a search engine work?…

23. Search engine architecture

24. Search engine limitations: • Spider can’t access some types of pages: database, frames, javascript… • Only 40% of pages are highly linked, others difficult for spider to locate • Search is of database: “some of the pages that once existed on the Web” • Spider may be optimised for popular sites rather than full coverage

25. Implications for Internet search strategy for NZ topics • Use several search engines • Avoid restricting search to .nz domain • Don’t rely on search engines to find everything • Use directories, subject resource guides • Use as many words as possible to describe your topic: optimise relevance

26. NZ directory examples

27. NZ Subject Resource Guides

28. Searching in practice…

29. Quality of NZ information on the Web • Like global information, and information in print: variable

30. NZ Information quality examples

31. Role of librarians in making NZ internet information available • Sharing our knowledge of web navigation…

32. …Creating search tools and information resources

33. …Preserving Internet information

34. Conclusion • NZ search engines do not offer advantages over global search engines • Comprehensive searches involve several search engines, directories, subject guides • Librarians have a role in creating local search tools, and in improving search skills