Lecture 9: Rank Aggregation in MetaSearch

1. Lecture 9: Rank Aggregation in MetaSearch • MetaSearch Engine • Social Choice Rules • Rank Aggregation

2. Choices of Search Engines • Many search engines exist to compete for users • The results are not necessarily the same • Different users prefer different search engines • Search results may, in the future, be biased towards paid advertisements.

3. GOOGLE: City University • CityUniversity London - the University for business and the ...... The University for business and the professions. Contact Us | About CityUniversity| Maps & Directions, AZ Index | Site Map | Help. Prospective Students. ...Description: Official site with information about courses, research, schools, and departments. Includes details... • CityUniversity HomeA University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study. Description: A private, nonprofit institution founded to serve working adults wanting to pursue educational opportunit... • Welcome to Dublin CityUniversity... here... Dublin CityUniversity, Dublin 9, Ireland. Tel. +353 (0) 1700 5000, Fax. +353 (0) 1 836 0830. Page updated: 02/09/03 legal. ...Description: Information on facilities, services, degree courses, research, the campus, student life, the library... • The CityUniversity of New YorkDescription: The University's Main Website. • CityUniversity of Hong KongAD Working Group lends staff, students an ear. Several issues stillevoked strong emotions as staff members and students voiced their ...Description: Formerly the City Polytechnic of Hong Kong. Includes information on university, links to learning... • Welcome to Oklahoma CityUniversity... At Oklahoma CityUniversity, our students come first. Our ... Oklahoma CityUniversity offers a quality, values-centered education. United ...Description: Admissions, academic programs and alumni relations, sports, services, news, calendar of events and...

4. YAHOO • CityUniversity London - the University for business and the ...... The University for business and the professions. Contact Us | About CityUniversity | Maps & Directions, AZ Index | Site Map | Help. Prospective Students. ... • City UniversityA University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study.www.cityu.edu/ - 3k - Cached - More pages from this site • Dublin City University... here... Dublin CityUniversity, Dublin 9, Ireland. Tel. +353 (0) 1 700 5000, Fax. +353 (0) 1 836 0830. Page updated: 02/09/03 legal. ...www.dcu.ie/ - 6k - Cached - More pages from this site • City University of New YorkThe University's Main Website.www.cuny.edu/ - More pages from this site • City University of Hong KongAD Working Group lends staff, students an ear. Several issues still evoked strong emotions as staff members and students voiced their ...www.cityu.edu.hk/ - 26k - Cached - More pages from this site • Oklahoma City University... At Oklahoma CityUniversity, our students come first. Our ... Oklahoma CityUniversity offers a quality, values-centered education. United ...www.okcu.edu/ - 20k - Cached - More pages from this site

5. AllTheWeb • Apartment Listings in University City  (sponsored)St. Louis, Missouri area apartment listings at Apartments.com. Free nationwide apartment search with visual rental listings online.http://www.apartments.com • City University - Washington  (sponsored)Contact information and resources such as yellow page information, phone number, address, maps and directions as provided by QwestDex.http://service.bfast.com • City University Apartments - Rent.com  (sponsored)Rent.com has millions of free apartment listings nationwide. Get $100 when you sign a lease near your school - it's easy.http://www.rent.com • City University London - the University for business and the professions... Contact Us | About City University | Maps & Directions A-Z Index | Site Map | Help ... Description: Official site with information about courses, research, schools, and departments. Includes details of news and events.more hits from:  http://www.city.ac.uk/  -  13 KB • Flash UpgradeDescription: A University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study.http://www.cityu.edu/  -  27 KB • Welcome to Dublin City University... KnowledgeWorks, wins DCU Mallin-invent award Full text you can go anywhere in the world from here... Dublin City University, Dublin 9, Ireland. Tel. +353 (0) 1 700 5000, Fax. +353 (0) 1 836 0830. Page updated: 02/09/03 legal search ... Description: Information on facilities, services, degree courses, research, the campus, student life, the library and DCU news.

6. MSN Search • City University of New York Details the CUNY system's campuses and admissions requirements. Eye photos of notable graduates such as actor Judd Hirsch.www.cuny.edu • City University Offers prospectus, admissions, academic calendar, courses and degrees, and news.www.cityuniversity.net • City University Private, nonprofit institution located in Belleview, Washington, offers higher education opportunities to working professionals and lifelong learners.www.cityu.edu • Valley City University Admissions Site covers financial aid and tuition and offers a virtual campus tour. Site also provides access to an online application.www.vcsu.nodak.edu/admissions • Oklahoma City University The Oklahoma City University site contains information about admissions, academic departments and student life. The site also provides library resources, faculty information and news.frodo.okcu.edu • Dublin City University, Dublin - TripAdvisor Browse reviews of the Dublin City University in Dublin, County Dublin, Ireland. Check consumer reviews, articles and opinions about the Dublin City University.www.tripadvisor.com/Hotel_Review-g186605-d209093-Reviews-Dublin_City_U...

7. AOL Search • City University London - the University for business and the ... - ... The University for business and the professions. Contact Us | About City University | Maps & Directions, AZ Index | Site Map | Help. Prospective Students. ... http://www.city.ac.uk/ • City University Home - A University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study. http://www.cityu.edu/ • Welcome to Dublin City University - ... here... Dublin City University, Dublin 9, Ireland. Tel. +353 (0) 1 700 5000, Fax. +353 (0) 1 836 0830. Page updated: 02/09/03 legal. ... http://www.dcu.ie/ • http://www.cuny.edu/ - The University's Main Website.http://www.cuny.edu/ • City University of Hong Kong - AD Working Group lends staff, students an ear. Several issues still evoked strong emotions as staff members and students voiced their ... http://www.cityu.edu.hk/ • Welcome to Oklahoma City University - ... At Oklahoma City University, our students come first. Our ... Oklahoma City University offers a quality, values-centered education. United ... http://www.okcu.edu/

8. Ask Jeeves • CityUniversity of Hong Kong Formerly the City Polytechnic of Hong Kong. Includes information on university, links to learning resources, research and departmental information.....From:www.cityu.edu.hk/ • CityUniversity Text Version. The University for business and the professions ... Contact Us | About CityUniversity | Maps & Directions A-Z Index | Site Map | Help...From:www.city.ac.uk/ • New Jersey CityUniversity Official site includes graduate and undergraduate information, news, events, online learning, calendars, and athletics.From:www.njcu.edu/ • Elizabeth City State University Admission Application Introduction 2004-2005 APPLICATION FOR ADMISSION. Welcome to the Elizabeth City State University Online Application. New Freshmen...From:www.ncmentor.org/applications/unc/apply/elizabeth_city_state_univ... • CityUniversity A University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study.From:www.cityu.edu/ • The City College of New York City College of New York (CUNY) A senior college of The CityUniversity of New York. Located in Manhattan. 138th Street at Convent Avenue. A Harlem masterpiece in the neo-Gothic...From:www.ccny.cuny.edu/

9. HotBot 1.City University of New YorkDetails the CUNY system's campuses and admissions requirements. Eye photos of notable graduates such as actor Judd Hirsch.www.cuny.edu/ - August 10, 2003 - 25 KB 2.City UniversityPrivate, nonprofit institution located in Belleview, Washington, offers higher education opportunities to working professionals and lifelong learners.www.cityu.edu/ - August 23, 2003 - 27 KB 3.New Jersey City UniversityExplore the course requirements for this liberal arts institution. Link to admissions and financial aid information. ... New Jersey City University. 2039 Kennedy Boulevard Jersey City, New Jersey 07305-1597 ... www.njcu.edu/ - November 1, 2003 - 20 KB 4.City University London - the University for business and the...... Contact Us | About City University | Maps & Directions. A-Z Index | Site Map | Help ... www.city.ac.uk/ - September 28, 2003 - 14 KB 5.Oklahoma City UniversityTake a tour of the campus, and explore a roster of academic programs. ... At Oklahoma City University, our students come first. ... www.okcu.edu/ - October 26, 2003 - 20 KB 6.University of Missouri, Kansas CityUniversity based in Kansas City, Missouri, presents and overview of its academic programs, and offers campus news. ... UNIVERSITY OF MISSOURI-KANSAS CITY. Bulletin. Winter 2004 Fee Update, more... ... www.umkc.edu/ - October 23, 2003 - 16 KB

10. Lycos 1.City University London - the University for business and the...… Contact Us | About City University | Maps & Directions A-Z Index | Site Map | Help …More results from: www.city.ac.uk September 16, 2003 - 14 KB 2.Flash UpgradeA University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study.www.cityu.edu November 1, 2003 - 27 KB 3.The City University of New YorkThe University's Main Website.www.cuny.edu December 31, 1969 - 121 B 4.Welcome to Dublin City University… KnowledgeWorks, wins DCU Mallin-invent award Full text you can go anywhere in the world from here... Dublin City University , Dublin 9, Ireland. Tel. +353 (0) 1 700 5000, Fax. +353 (0) 1 836...More results from: www.dcu.ie October 29, 2003 - 5 KB 5.City University of Hong KongFormerly the City Polytechnic of Hong Kong. Includes information on university, links to learning resources, research and departmental information and student information.More results from: www.cityu.edu.hk October 29, 2003 - 26 KB 6.HCU HomePage -English-… contact us For any suggestion and requests to this web site, www-admin@hiroshima-cu.ac.jp Feel free to link this web site. Hiroshima City University More results from: www.hiroshima-cu.ac.jp May 11, 2003 - 26 KB

11. Teoma • City University of Hong Kong Formerly the City Polytechnic of Hong Kong. Includes information on university, links to learning resources, research and departmental information...www.cityu.edu.hk/[Related Pages][More Results from www.cityu.edu.hk] • City University Text Version. The University for business and the professions ... Contact Us | About City University | Maps & Directions A-Z Index | Site Map | Help...www.city.ac.uk/[More Results from www.city.ac.uk] • New Jersey City University Official site includes graduate and undergraduate information, news, events, online learning, calendars, and athletics.www.njcu.edu/[Related Pages][More Results from www.njcu.edu] • Elizabeth City State University Admission Application Introduction 2004-2005 APPLICATION FOR ADMISSION. Welcome to the Elizabeth City State University Online Application. New Freshmen...www.ncmentor.org/applications/unc/apply/el... • City University A University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study.www.cityu.edu/ • The City College of New York City College of New York (CUNY) A senior college of The City University of New York. Located in Manhattan. 138th Street at Convent Avenue. A Harlem masterpiece in the neo-Gothic...www.ccny.cuny.edu/[More Results from www.ccny.cuny.edu]

12. MetaSearch Engine • Metasearch Engines are designed to increase the coverage of web by forwarding users’ queries to multiple search engines • Users’ requests are sent to multiple search engines such as AlltheWeb, Google, MSN. • Then the results from the individual search engine are combined into a single result set to present to users.

13. Different Forms of MetaSearch • Submit different representations of the same query to the same search engine, then combine the results. • Submit the same query to several search engine adopting different information retrieval models, then combine the results.

14. Issues • How to combine the results retrieved by different source search engines is crucial for the success of a metasearch engine. • And this is the problem that social choice theory has been trying to answer.

15. Search Engine Watch • Interesting metat search engines are listed at • http://www.searchenginewatch.com/links/article.php/2156241

16. Award WinnersSource: www.searchenginewatch.com • VivisimoEnter a search term, and Vivismo will not only pull back matching responses from major search engines but also automatically organize the pages into categories. Slick and easy to use. Winner of Best Meta Search Engine award from Search Engine Watch in 2002. • EZ2WWWThis meta search engine provides results from AllTheWeb, AltaVista, Google, Open Directory, Teoma, Wisenut and Yahoo. Advanced search offers access to more than 1,000 specialized resources, searchable by category. Honorable mention for Best Meta Search Engine award from Search Engine Watch in 2002. • KartooIf you like the idea of seeing your web results visually, this meta search site shows the results with sites being interconnected by keywords. Honorable mention for Best Meta Search Engine award from Search Engine Watch in 2002. • SurfWaxSearches against major engines or provides those who open free accounts the ability to chose from a list of hundreds. Using the "SiteSnaps" feature, you can preview any page in the results and see where your terms appear in the document. Allows results or documents to be saved for future use. Honorable mention for Best Meta Search Engine award from Search Engine Watch in 2002. • FazzleFazzle offers a highly flexible and customizable interface to a wide variety of information sources, ranging from general web results to specialized search resources in a number of subject specific categories. Formerly called SearchOnline.

17. Other Top ChoicesSource: www.searchenginewatch.com • InfoGridIn a compact format, InfoGrid provides direct links to major search sites and topical web sites in different categories. Meta search and news searching is also offered. • Infonetware RealTerm SearchThis site is primarily designed to demonstrate classification technology from Infogistics. It's a meta search engine, and it does topical classification of results, like Vivisimo. However, it is unique in that you can select several different topics, then "drill down" to see results from all of them, rather than being restricted to the results from only one topic. • IthakiIthaki is probably the most "global" of all meta search engines, available in 14 languages and offering more than 35 different categories for limiting your search. In addition, Ithaki offers country specific search, querying only local search engines rather than the regional versions of the major search engines. • IxquickMeta search engine that ranks results based on the number of "top 10" rankings a site receives from the various search engines. • ProFusionBrings back listings from several major search engines as well as "Invisible Web" resources. Formerly based at the University of Kansas, the site was purchased by search company Intelliseek in April 2000. • qbSearchWant to get multiple pages of results from a search engine combined into one single page? QB-Search will quickly join up to 200 pages of listings from major search engines. • Query ServerSearch against major web-wide search engines, as well as major news, health, money and government search services. • Turbo10Turbo10 is a metasearch Engine accesses both traditional web search engines and some invisible web databases, with a very speedy interface.

18. Popular ServicesSource: www.searchenginewatch.com • DogpilePopular metasearch site owned by InfoSpace that sends a search to a customizable list of search engines, directories and specialty search sites, then displays results from each search engine individually. • ExciteFormerly a crawled-based search engine, Excite was acquired by InfoSpace in 2002 and uses the same underlying technology as the other InfoSpace meta search engines, but maintains its own portal features. • MetaCrawlerOne of the oldest meta search services, MetaCrawler began in July 1995 at the University of Washington. MetaCrawler was purchased by InfoSpace, an online content provider, in Feb. 97. • WebcrawlerFormerly a crawled-based search engine owned by Excite, Webcrawler was acquired by InfoSpace in 2002 and uses the same underlying technology as the other InfoSpace meta search engines, but offers a fast and clean, ad-free interface. • MammaFounded in 1996, Mamma.com is one of the oldest meta search engines on the web. Mamma searches against a variety of major crawlers, directories and specialty search sites. The service also provides a paid listings option for advertisers, Mamma Classifieds. • Search.comSearch.com is a meta search engine operated by CNET. It offers both web-wide search and a wide variety of specialty search options. Search.com uses technology from SavvySearch, which was acquired by CNET in October 1999. The SavvySearch site itself no longer operates. SavvySearch was one of the older metasearch services,

19. Special ServicesSource: www.searchenginewatch.com • Family Friendly SearchMeta search service that queries major kid-friendly search engines. • Searchy.co.ukSearches 15 U.K. engines. The advanced search form allows you to change the order that results are presented, either by speed or manually to suit your own preferences. • TeRespondoSpanish metacrawler that searches on the more popular search engines. Motor de busqueda que busca tu consulta en los buscadores mas populares en espanol. • Watson for the MacintoshWatson is a "Swiss Army Knife" with nineteen interfaces to web content and services -- an improvement on Sherlock, with nearly twice as many tools, including Google Searching.

20. Social Choice Theory • Studies on protocols that help a group of people make collective decisions, such as vote.

21. A Fundamental problem • Given a collection of agents (voters) • with preferences over different alternatives (allocations, outcomes), • how should society evaluate these alternatives and make a decision for all • that may be for the will of some voters but against that of others.

22. Applications • Voters elect president from several candidates. • National polls for economic or political policy of the government • The procedure or rule of election • The rank of metasearch engine obtained from those of search engines

23. Group Descisions How do we make decisions • Flip a coin? • Dictatorship? • Democracy (Majority rule)? in a group with • Heterogeneity of individual preference

24. Group Decision Rules • Majority rule , • Condorcet paradox (voting cycle) • Borda rule

25. Mathematical model • A set of voters V={v1,v2,v3,…,Vn} • A set of alternatives or outcomes S={s1,s2,s3,…Sm}, with |S|=m; and • A set of preference relation P={R1,R2,R3…Rn}, called a preference profile, • the preference relation Ri for each voter i is a permutation (order) of elements in S.

26. Example 1 Majority Rule • 3 rational people have rational preferences over 2 alternatives {x,y} Person 1 2 3 1stX YX 1 : X>Y Pref. i.e.Person 2 : Y>X 2ndYXY 3 : X>Y How to Aggregate their preferences? How to choose?

27. Using majority rule. • Since more than ½ people (two out of three) prefer x to y. • Then the group prefers x to y

28. Example 2 Condorcet Paradox • 3 rational people have rational preferences over 3 alternatives {x,y,z} Person 1 2 3 1stX YZ 1 : X>Y>Z Pref. 2ndYZ Xi.e. Person 2 : Y>Z>X 3rdZXY 3 : Z>X>Y

29. Binary/paired Comparison With Majority rule • Person 1 2 3 1stX YZ 1 : X>Y Pref. 2ndYZ X for (x,y) 2 : Y>X X>Y 3rdZXY 3 : X>Y • Similarly, for (Y,Z) we can get Y>Z; for (Z,X) we can get Z>X. • Then X>Y>Z>X (cycling) , Intransitive  Not rational

30. It was noted by Condorcet in the 18 century that no alternative can win a majority against all other alternatives. • Pairwise majority is not satisfactory in all cases.

31. Example 3 Borda Rule • For each voter, • associate the number 1 with the most preferred alternative, • 2 with the second and so on, • Assign to each alternative the number equal to • the sum of the numbers the individual voters assigned to the alternative.

32. Person 1 2 3 1stX(1) Y(1)X(1) X(4) X Pref. 2ndY(2)X(2) W(2) Y(7)  Y 3rdZ(3)W(3)Z(3) Z(10) W 4thW(4)Z(4)Y(4) W(9) Z Then We get choice X>Y>W>Z

33. For above example, if we use binary/paired comparison With majority rule . We can get X>Y in 2 out of 3, Y>W in 2 out of 3, W>Z in 2 out of 3, X>W in 3 out of 3, X>Z in 3 out of 3, Y>Z in 2 out of 3 Then we can achieve same choice X>Y>W>Z

34. For the previous example we had trouble with majority rule via binary/paired comparison, we get a tie between all three alternatives with the Borda’s rule: • All three alternatives get a sum of 6.

35. Some variations 1 with relevant scores available allotting each input system a point p to be distributed according to relevance scores of the documents. 2 Weighted Borda-rule Each voter may not have equal effectiveness to the final result. We may set more weight to good quality input systems.

36. Condorcet winner algorithm It also comes from social choice theory. The Condorcet algorithm says that any candidate that can beat all other candidates in a head-to-head contest (pair-wise comparison) should win the election.

37. Step 1, Construct Condorcet Graph. For each candidate pair (x,y), there exists an edge from x to y if x would receive at least as many votes as y in a head-to-head contest. In Condorcet graph, there is at least one directed edge between every pair of candidates. ( we call the graph is semi-complete) It may contains cycles in the graph. This is due to voting paradox of the condorcet voting.

38. Step 2, We form a new acyclic graph from an old cyclic one by contracting all of the nodes in a cycle into one. It is a strongly connected component graph (SCCG). Definition of Strongly connected component(SCC): A strongly connectedsubgraph, S, of a directed graph, D, such that no vertex or subset of vertices of D can be added to S such that the new subgraph is still strongly connected.

39. The graph is totally orderable at the level of the SCC’s and each SCC is a “pocket” of cycles, within which each candidate is tied. Step 3, The condorcet-consistent Hamiltonian path is any Hamiltonian path through Condorcet graph. Definition Hamiltonian path: A path between two vertices of a graph that visits each vertex exactly once.

40. Theorem 1. Every semi-complete graph contains a Hamiltonian path. • Theorem 2. Suppose x and y are nodes in a graph g, and that X and Y are nodes of the associated SCCG G such that x X and y Y. If there exists a path from X to Y in G, then every Condorcet path of g has x before y. Refer to[Javed A. Aslam, Mark Montague 2001] for proof.

41. Rank Aggregation in MetaSearch Here we discussed two cases which using algorithm rooted at social choice theory for MetaSearch rank aggregation. • Data fusion track in TREC [Javed A. Aslam, Mark Montague 2001] Models for Metasearch in SIGIR2001 • Rank aggregation for web search engine [Cynthia Dwork, Ravi Kumar, Moni Naor, D.Sivakumar 2001] Rank Aggregation Methods for the Web in WWW10

42. Data fusion track in TREC • TREC (Text Retrieval Conference ,see http://trec.nist.gov/) maintains about 6Gb of SGML tagged text, queries and respective answers for evaluation purposes. • The TREC organizers distribute data sets in advance and 50 new queries each year. • The competing teams then submit ranked lists of documents that their system gave in response to each query. And these retrieval systems will be evaluated.

43. These ranked lists are available for metasearch researchers to download and fuse. • For each query, every retrieval system will return top 1000 documents and relevant score is available. • Then given these results retrieved by many different retrieval systems, how to aggregate them for better performance?

44. Previous algorithms • Min, Max and Average Models [Fox and Shaw,1995] • Linear Combination Model [Bartell 1995] • Logistic Regression Model

45. Example • Min, Max and Average model The final score of each document d is based on the scores given to d by each input systems (voters). Algorithm Final score CombMin minimum of individual relevance scores CombMed median of individual relevance scores CombMax maximum of individual relevance scores CombSum sum of individual relevance scores CombANZ CombSum / num non-zero relevance scores CombMNZ CombSum * num non-zero relevance scores

46. Linear Combination Model The final score of document d is a simply linearly (each weighted differently) combining the normalized relevance scores given to each document.

47. Experiment result on TREC Model • The performance of rank aggregation is evaluated by average precision over the queries • Score-based borda-fuse is usually the best method among several borda variant algorithms. • It is better than best input system over most of data collection. Such as TREC3, TREC5

48. Experiment result II • The performance of rank aggregation is evaluated by average precision over the queries. • Condorcet-fusion is the only algorithm that , without training data, ever matches the performance of the best input system over TREC 9. • Condorcet-fusion seems particularly sensitive to the dependence of input systems. If the input systems (voters) are too similar, the performance will decrease.

49. Rank aggregation methods for web New Challenges: Different from the case in TREC data fusion, • The coverage of various search engine is different • Thus some highly relevant web pages may not be ranked by some search engines. • Therefore, each voter ranks a partial candidate list

50. Preliminaries • Given a universe U, an ordered list with respect to U is an ordering of a subset S U, i.e., ,with each and is some ordering relation on S. • If contains • all the elements in U, then it is said to be a full list, • otherwise it is called partial list.