Internet Searching and Browsing in a Multilingual World

1. Internet Searching and Browsing in a Multilingual World An Experiment on the Chinese Business Intelligence Portal Acknowledgment: NSF/NIJ Grant

2. Outline • Motivation • The Chinese Business Intelligence Portal • System Description • Results of Usability Study • Conclusions

3. Introduction

4. Motivation • As the Internet grows in popularity worldwide, more users want to access Web content in their native languages • The majority of the total global online population (63.5%) lives in non-English-speaking areas (Global-Reach, 2002) • Such population is estimated to grow rapidly, much faster than English-speaking population • However, existing search engines may not serve their needs, because most technologies have been developed for English-speaking users

5. This Presentation • The following slides present our efforts in creating and evaluating intelligent Web portals that address the above needs • The Chinese business information serves as our research testbed • Through the studies, we aim to achieve better understanding of human interaction and analysis with automated systems developed for Internet searching and browsing in a multilingual world

6. The Chinese Business Intelligence Portal (CBizPort)

7. CBizPort • The Chinese Business Intelligence Portal (CBizPort) • Two versions of user interface: Simplified Chinese and Traditional Chinese • URLs • Introduction: http://ai.bpa.arizona.edu/go/dl/cbizport.html • Portal: http://ai17.bpa.arizona.edu:8080/big5biz/index.html • Each version has the same user interface and provides the same functions • Encoding conversion • Meta searching major Chinese information sources • Summarization, Categorization • Providing links to major Chinese business Web resources • The following slides show the system architecture and screen shots of CBizPort

9. Provides links to major Chinese business Web sites and resources Provides both Simplified and Traditional Chinese versions of user interface Allows input of multiple key terms Meta searches 8 major information sources of Mainland China, Hong Kong, and Taiwan Keywords:

10. A two-sentence summary on left, original page on right Search Page Result Page Web pages grouped by key phrases extracted by mutual information algorithm (non-exclusive categorization) Categorizer Summarizer

11. Evaluation of CBizPort Objectives • To evaluate the performance of summarizer as a preview function and categorizer as an overview function • To compare CBizPort with regional Chinese search engines to study its effectiveness and usability • To evaluate, in comparison with existing regional Chinese search engines, the information quality obtained from CBizPort and its capability of searching for cross-regional business information

12. Experimental Design • Searching and browsing were studied • Scenario-based, culturally oriented tasks, e.g., • A search task (4 min): “Find two cities in mainland China that Motorola has set up its manufacturing operations” • A browse task (5 min): “Describe, in a number of distinct themes, the economic impacts of removing trade barriers between mainland China and Taiwan towards Hong Kong ” • Theme identification method (Chen et al., 2001) • Pilot test: 3 subjects used up all the time in most tasks  only focused on effectiveness but not efficiency

13. S = search task; B = browse task; O = Basic searching (with neither summarizer nor analyzer); M = Basic searching + with summarizer only; A = Basic searching + with categorizer only; G = General searching and browsing; C = Cross-regional searching and browsing; same number signals the same question across different regions (Random assignment of tasks is used for different settings)

14. Browse Search CBizPort Compare Compare Openfind With or without categorizer or Browse YahooHK Search With or without summarizer or Sina.com Comparisons

15. Subjects • 30 subjects, 10 from each region, were recruited • Rationale: equal influence of regional impacts • Each subject used CBizPort and another search tool according to his/her origin

16. Experts • Three experts, one from each region, were recruited to provide answers to all browse tasks • First, the experts identify the set of relevant answers (organized into themes) to a browse task • Then, they modified the answers by adding some of subjects’ responses that they judged as relevant • The above two steps are repeated for all the other browse tasks Bla bla bla

17. Hypotheses • Three sets of hypotheses were tested • CBizPort’s Enhanced Analysis Capabilities • Searching and browsing • With or without summarizer/categorizer • SE Performance Comparison • Searching and browsing capabilities • Individual settings and combination* • Users’ Subjective Evaluation • Information quality • cross-regional searching capability • overall satisfaction • Auxiliary hypotheses: Performance of the three regions are not significantly different We tried to mimic a situation that each subject was allowed to use both CBizPort and benchmark search engine together to solve the same problem

18. CBizPort Experts’ answers Benchmark SE

20. Performance Measures • Accuracy = Percentage of correct answers • Precision = number of correct themes identified by users / total number of themes identified by users • Recall = number of correct themes identified by users / total number of themes identified by an expert • F value = 2*Recall*Precision / (Precision + Recall) • Information quality: accessibility, appropriateness of amount, believability, completeness, …, etc. (Wang & Strong, 2002) • Subjective evaluation: cross-regional searching capability, overall satisfaction, protocol analysis, post-hoc test (to study whether the three SEs yield significantly different results)

21. Accuracy of search tasks

22. Precision of browse tasks

23. Recall of browse tasks

24. F value of browse tasks

25. Information Quality

26. Users’ Subjective Evaluation

27. Subjects liked summarizer and categorizer Subj.#15: “… good performance in summarization and categorization, more focused results can be found”; #26: “… very handy”; #6: “…useful tools to enhance the searching ability” (11 subjects) CBizPort provides a wide coverage and variety of searching options Subj.#2: “… Yahoo Search Engine is more limited when search certain term in a specific region … While CBizport can fulfill what Yahoo couldn’t do.”; #4: “… more search engines to choose from” (4 subjects) Subjects’ Verbal Comments

28. Subjects are familiar with benchmark SEs Subj#27: “I am familiar with the format of Openfind. So that's the reason that I am more satisfied with it than CBizPort.”; (4 subjects) Benchmark SEs are not good at cross-regional information searching Subj#15: “Sina gives many results but they are not focused, and is poor at searching HK and Taiwan results”; #5: “provide more accurate regional searching” CBizPort is user friendly but slow #3: “Yahoo not as precise as CBizPort”; #28: “… easier to search” (7 subjects); “slow” (3 subjects) Subjects’ Verbal Comments (2)

29. Conclusions • CBizPort’s summarizer and categorizer provide helpful analysis capabilities for users’ search and browse tasks • CBizPort’s searching and browsing performance is comparable to that of regional Chinese search engines • CBizPort can significantly augment the searching and browsing ability of regional Chinese search engines, thus improving human integration of regional information and analysis • Information quality, cross-regional searching capability and overall satisfaction of CBizPort are comparable to those of regional Chinese search engines • CBizPort is better than regional Chinese search engines in terms of analysis functions, cross-regional searching capabilities and user-friendliness