Lecture 9: Rank  Aggregation in MetaSearch
Download
1 / 64

Lecture 9: Rank Aggregation in MetaSearch - PowerPoint PPT Presentation

Lecture 9: Rank Aggregation in MetaSearch MetaSearch Engine Social Choice Rules Rank Aggregation Choices of Search Engines Many search engines exist to compete for users The results are not necessarily the same Different users prefer different search engines

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha

Download Presentation

Lecture 9: Rank Aggregation in MetaSearch

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Slide1 l.jpg

Lecture 9: Rank Aggregation in MetaSearch

  • MetaSearch Engine

  • Social Choice Rules

  • Rank Aggregation


Choices of search engines l.jpg
Choices of Search Engines

  • Many search engines exist to compete for users

    • The results are not necessarily the same

    • Different users prefer different search engines

    • Search results may, in the future, be biased towards paid advertisements.


Google city university l.jpg
GOOGLE: City University

  • CityUniversity London - the University for business and the ...... The University for business and the professions. Contact Us | About CityUniversity| Maps & Directions, AZ Index | Site Map | Help. Prospective Students. ...Description: Official site with information about courses, research, schools, and departments. Includes details...

  • CityUniversity HomeA University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study. Description: A private, nonprofit institution founded to serve working adults wanting to pursue educational opportunit...

  • Welcome to Dublin CityUniversity... here... Dublin CityUniversity, Dublin 9, Ireland. Tel. +353 (0) 1700 5000, Fax. +353 (0) 1 836 0830. Page updated: 02/09/03 legal. ...Description: Information on facilities, services, degree courses, research, the campus, student life, the library...

  • The CityUniversity of New YorkDescription: The University's Main Website.

  • CityUniversity of Hong KongAD Working Group lends staff, students an ear. Several issues stillevoked strong emotions as staff members and students voiced their ...Description: Formerly the City Polytechnic of Hong Kong. Includes information on university, links to learning...

  • Welcome to Oklahoma CityUniversity... At Oklahoma CityUniversity, our students come first. Our ... Oklahoma CityUniversity offers a quality, values-centered education. United ...Description: Admissions, academic programs and alumni relations, sports, services, news, calendar of events and...


Yahoo l.jpg
YAHOO

  • CityUniversity London - the University for business and the ...... The University for business and the professions. Contact Us | About CityUniversity | Maps & Directions, AZ Index | Site Map | Help. Prospective Students. ...

  • City UniversityA University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study.www.cityu.edu/ - 3k - Cached - More pages from this site

  • Dublin City University... here... Dublin CityUniversity, Dublin 9, Ireland. Tel. +353 (0) 1 700 5000, Fax. +353 (0) 1 836 0830. Page updated: 02/09/03 legal. ...www.dcu.ie/ - 6k - Cached - More pages from this site

  • City University of New YorkThe University's Main Website.www.cuny.edu/ - More pages from this site

  • City University of Hong KongAD Working Group lends staff, students an ear. Several issues still evoked strong emotions as staff members and students voiced their ...www.cityu.edu.hk/ - 26k - Cached - More pages from this site

  • Oklahoma City University... At Oklahoma CityUniversity, our students come first. Our ... Oklahoma CityUniversity offers a quality, values-centered education. United ...www.okcu.edu/ - 20k - Cached - More pages from this site


Alltheweb l.jpg
AllTheWeb

  • Apartment Listings in University City  (sponsored)St. Louis, Missouri area apartment listings at Apartments.com. Free nationwide apartment search with visual rental listings online.http://www.apartments.com

  • City University - Washington  (sponsored)Contact information and resources such as yellow page information, phone number, address, maps and directions as provided by QwestDex.http://service.bfast.com

  • City University Apartments - Rent.com  (sponsored)Rent.com has millions of free apartment listings nationwide. Get $100 when you sign a lease near your school - it's easy.http://www.rent.com

  • City University London - the University for business and the professions... Contact Us | About City University | Maps & Directions A-Z Index | Site Map | Help ... Description: Official site with information about courses, research, schools, and departments. Includes details of news and events.more hits from:  http://www.city.ac.uk/  -  13 KB

  • Flash UpgradeDescription: A University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study.http://www.cityu.edu/  -  27 KB

  • Welcome to Dublin City University... KnowledgeWorks, wins DCU Mallin-invent award Full text you can go anywhere in the world from here... Dublin City University, Dublin 9, Ireland. Tel. +353 (0) 1 700 5000, Fax. +353 (0) 1 836 0830. Page updated: 02/09/03 legal search ... Description: Information on facilities, services, degree courses, research, the campus, student life, the library and DCU news.


Msn search l.jpg
MSN Search

  • City University of New York Details the CUNY system's campuses and admissions requirements. Eye photos of notable graduates such as actor Judd Hirsch.www.cuny.edu

  • City University Offers prospectus, admissions, academic calendar, courses and degrees, and news.www.cityuniversity.net

  • City University Private, nonprofit institution located in Belleview, Washington, offers higher education opportunities to working professionals and lifelong learners.www.cityu.edu

  • Valley City University Admissions Site covers financial aid and tuition and offers a virtual campus tour. Site also provides access to an online application.www.vcsu.nodak.edu/admissions

  • Oklahoma City University The Oklahoma City University site contains information about admissions, academic departments and student life. The site also provides library resources, faculty information and news.frodo.okcu.edu

  • Dublin City University, Dublin - TripAdvisor Browse reviews of the Dublin City University in Dublin, County Dublin, Ireland. Check consumer reviews, articles and opinions about the Dublin City University.www.tripadvisor.com/Hotel_Review-g186605-d209093-Reviews-Dublin_City_U...


Aol search l.jpg
AOL Search

  • City University London - the University for business and the ... - ... The University for business and the professions. Contact Us | About City University | Maps & Directions, AZ Index | Site Map | Help. Prospective Students. ... http://www.city.ac.uk/

  • City University Home - A University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study. http://www.cityu.edu/

  • Welcome to Dublin City University - ... here... Dublin City University, Dublin 9, Ireland. Tel. +353 (0) 1 700 5000, Fax. +353 (0) 1 836 0830. Page updated: 02/09/03 legal. ... http://www.dcu.ie/

  • http://www.cuny.edu/ - The University's Main Website.http://www.cuny.edu/

  • City University of Hong Kong - AD Working Group lends staff, students an ear. Several issues still evoked strong emotions as staff members and students voiced their ... http://www.cityu.edu.hk/

  • Welcome to Oklahoma City University - ... At Oklahoma City University, our students come first. Our ... Oklahoma City University offers a quality, values-centered education. United ... http://www.okcu.edu/


Ask jeeves l.jpg
Ask Jeeves

  • CityUniversity of Hong Kong Formerly the City Polytechnic of Hong Kong. Includes information on university, links to learning resources, research and departmental information.....From:www.cityu.edu.hk/

  • CityUniversity Text Version. The University for business and the professions ... Contact Us | About CityUniversity | Maps & Directions A-Z Index | Site Map | Help...From:www.city.ac.uk/

  • New Jersey CityUniversity Official site includes graduate and undergraduate information, news, events, online learning, calendars, and athletics.From:www.njcu.edu/

  • Elizabeth City State University Admission Application Introduction 2004-2005 APPLICATION FOR ADMISSION. Welcome to the Elizabeth City State University Online Application. New Freshmen...From:www.ncmentor.org/applications/unc/apply/elizabeth_city_state_univ...

  • CityUniversity A University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study.From:www.cityu.edu/

  • The City College of New York City College of New York (CUNY) A senior college of The CityUniversity of New York. Located in Manhattan. 138th Street at Convent Avenue. A Harlem masterpiece in the neo-Gothic...From:www.ccny.cuny.edu/


Hotbot l.jpg
HotBot

1.City University of New YorkDetails the CUNY system's campuses and admissions requirements. Eye photos of notable graduates such as actor Judd Hirsch.www.cuny.edu/ - August 10, 2003 - 25 KB

2.City UniversityPrivate, nonprofit institution located in Belleview, Washington, offers higher education opportunities to working professionals and lifelong learners.www.cityu.edu/ - August 23, 2003 - 27 KB

3.New Jersey City UniversityExplore the course requirements for this liberal arts institution. Link to admissions and financial aid information. ... New Jersey City University. 2039 Kennedy Boulevard Jersey City, New Jersey 07305-1597 ... www.njcu.edu/ - November 1, 2003 - 20 KB

4.City University London - the University for business and the...... Contact Us | About City University | Maps & Directions. A-Z Index | Site Map | Help ... www.city.ac.uk/ - September 28, 2003 - 14 KB

5.Oklahoma City UniversityTake a tour of the campus, and explore a roster of academic programs. ... At Oklahoma City University, our students come first. ... www.okcu.edu/ - October 26, 2003 - 20 KB

6.University of Missouri, Kansas CityUniversity based in Kansas City, Missouri, presents and overview of its academic programs, and offers campus news. ... UNIVERSITY OF MISSOURI-KANSAS CITY. Bulletin. Winter 2004 Fee Update, more... ... www.umkc.edu/ - October 23, 2003 - 16 KB


Lycos l.jpg
Lycos

1.City University London - the University for business and the...… Contact Us | About City University | Maps & Directions A-Z Index | Site Map | Help …More results from: www.city.ac.uk September 16, 2003 - 14 KB

2.Flash UpgradeA University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study.www.cityu.edu November 1, 2003 - 27 KB

3.The City University of New YorkThe University's Main Website.www.cuny.edu December 31, 1969 - 121 B

4.Welcome to Dublin City University… KnowledgeWorks, wins DCU Mallin-invent award Full text you can go anywhere in the world from here... Dublin City University , Dublin 9, Ireland. Tel. +353 (0) 1 700 5000, Fax. +353 (0) 1 836...More results from: www.dcu.ie October 29, 2003 - 5 KB

5.City University of Hong KongFormerly the City Polytechnic of Hong Kong. Includes information on university, links to learning resources, research and departmental information and student information.More results from: www.cityu.edu.hk October 29, 2003 - 26 KB

6.HCU HomePage -English-… contact us For any suggestion and requests to this web site, www-admin@hiroshima-cu.ac.jp Feel free to link this web site. Hiroshima City University More results from: www.hiroshima-cu.ac.jp May 11, 2003 - 26 KB


Teoma l.jpg
Teoma

  • City University of Hong Kong Formerly the City Polytechnic of Hong Kong. Includes information on university, links to learning resources, research and departmental information...www.cityu.edu.hk/[Related Pages][More Results from www.cityu.edu.hk]

  • City University Text Version. The University for business and the professions ... Contact Us | About City University | Maps & Directions A-Z Index | Site Map | Help...www.city.ac.uk/[More Results from www.city.ac.uk]

  • New Jersey City University Official site includes graduate and undergraduate information, news, events, online learning, calendars, and athletics.www.njcu.edu/[Related Pages][More Results from www.njcu.edu]

  • Elizabeth City State University Admission Application Introduction 2004-2005 APPLICATION FOR ADMISSION. Welcome to the Elizabeth City State University Online Application. New Freshmen...www.ncmentor.org/applications/unc/apply/el...

  • City University A University which believes in forward-thinking business and leadership skills, exposing students to the latest technology in all courses of study.www.cityu.edu/

  • The City College of New York City College of New York (CUNY) A senior college of The City University of New York. Located in Manhattan. 138th Street at Convent Avenue. A Harlem masterpiece in the neo-Gothic...www.ccny.cuny.edu/[More Results from www.ccny.cuny.edu]


Metasearch engine l.jpg
MetaSearch Engine

  • Metasearch Engines are designed to increase the coverage of web by forwarding users’ queries to multiple search engines

    • Users’ requests are sent to multiple search engines such as AlltheWeb, Google, MSN.

  • Then the results from the individual search engine are combined into a single result set to present to users.


Different forms of metasearch l.jpg
Different Forms of MetaSearch

  • Submit different representations of the same query to the same search engine, then combine the results.

  • Submit the same query to several search engine adopting different information retrieval models, then combine the results.


Issues l.jpg
Issues

  • How to combine the results retrieved by different source search engines is crucial for the success of a metasearch engine.

  • And this is the problem that social choice theory has been trying to answer.


Search engine watch l.jpg
Search Engine Watch

  • Interesting metat search engines are listed at

    • http://www.searchenginewatch.com/links/article.php/2156241


Award winners source www searchenginewatch com l.jpg
Award WinnersSource: www.searchenginewatch.com

  • VivisimoEnter a search term, and Vivismo will not only pull back matching responses from major search engines but also automatically organize the pages into categories. Slick and easy to use. Winner of Best Meta Search Engine award from Search Engine Watch in 2002.

  • EZ2WWWThis meta search engine provides results from AllTheWeb, AltaVista, Google, Open Directory, Teoma, Wisenut and Yahoo. Advanced search offers access to more than 1,000 specialized resources, searchable by category. Honorable mention for Best Meta Search Engine award from Search Engine Watch in 2002.

  • KartooIf you like the idea of seeing your web results visually, this meta search site shows the results with sites being interconnected by keywords. Honorable mention for Best Meta Search Engine award from Search Engine Watch in 2002.

  • SurfWaxSearches against major engines or provides those who open free accounts the ability to chose from a list of hundreds. Using the "SiteSnaps" feature, you can preview any page in the results and see where your terms appear in the document. Allows results or documents to be saved for future use. Honorable mention for Best Meta Search Engine award from Search Engine Watch in 2002.

  • FazzleFazzle offers a highly flexible and customizable interface to a wide variety of information sources, ranging from general web results to specialized search resources in a number of subject specific categories. Formerly called SearchOnline.


Other top choices source www searchenginewatch com l.jpg
Other Top ChoicesSource: www.searchenginewatch.com

  • InfoGridIn a compact format, InfoGrid provides direct links to major search sites and topical web sites in different categories. Meta search and news searching is also offered.

  • Infonetware RealTerm SearchThis site is primarily designed to demonstrate classification technology from Infogistics. It's a meta search engine, and it does topical classification of results, like Vivisimo. However, it is unique in that you can select several different topics, then "drill down" to see results from all of them, rather than being restricted to the results from only one topic.

  • IthakiIthaki is probably the most "global" of all meta search engines, available in 14 languages and offering more than 35 different categories for limiting your search. In addition, Ithaki offers country specific search, querying only local search engines rather than the regional versions of the major search engines.

  • IxquickMeta search engine that ranks results based on the number of "top 10" rankings a site receives from the various search engines.

  • ProFusionBrings back listings from several major search engines as well as "Invisible Web" resources. Formerly based at the University of Kansas, the site was purchased by search company Intelliseek in April 2000.

  • qbSearchWant to get multiple pages of results from a search engine combined into one single page? QB-Search will quickly join up to 200 pages of listings from major search engines.

  • Query ServerSearch against major web-wide search engines, as well as major news, health, money and government search services.

  • Turbo10Turbo10 is a metasearch Engine accesses both traditional web search engines and some invisible web databases, with a very speedy interface.


Popular services source www searchenginewatch com l.jpg
Popular ServicesSource: www.searchenginewatch.com

  • DogpilePopular metasearch site owned by InfoSpace that sends a search to a customizable list of search engines, directories and specialty search sites, then displays results from each search engine individually.

  • ExciteFormerly a crawled-based search engine, Excite was acquired by InfoSpace in 2002 and uses the same underlying technology as the other InfoSpace meta search engines, but maintains its own portal features.

  • MetaCrawlerOne of the oldest meta search services, MetaCrawler began in July 1995 at the University of Washington. MetaCrawler was purchased by InfoSpace, an online content provider, in Feb. 97.

  • WebcrawlerFormerly a crawled-based search engine owned by Excite, Webcrawler was acquired by InfoSpace in 2002 and uses the same underlying technology as the other InfoSpace meta search engines, but offers a fast and clean, ad-free interface.

  • MammaFounded in 1996, Mamma.com is one of the oldest meta search engines on the web. Mamma searches against a variety of major crawlers, directories and specialty search sites. The service also provides a paid listings option for advertisers, Mamma Classifieds.

  • Search.comSearch.com is a meta search engine operated by CNET. It offers both web-wide search and a wide variety of specialty search options. Search.com uses technology from SavvySearch, which was acquired by CNET in October 1999. The SavvySearch site itself no longer operates. SavvySearch was one of the older metasearch services,


Special services source www searchenginewatch com l.jpg
Special ServicesSource: www.searchenginewatch.com

  • Family Friendly SearchMeta search service that queries major kid-friendly search engines.

  • Searchy.co.ukSearches 15 U.K. engines. The advanced search form allows you to change the order that results are presented, either by speed or manually to suit your own preferences.

  • TeRespondoSpanish metacrawler that searches on the more popular search engines. Motor de busqueda que busca tu consulta en los buscadores mas populares en espanol.

  • Watson for the MacintoshWatson is a "Swiss Army Knife" with nineteen interfaces to web content and services -- an improvement on Sherlock, with nearly twice as many tools, including Google Searching.


Social choice theory l.jpg
Social Choice Theory

  • Studies on protocols that help a group of people make collective decisions, such as vote.


A fundamental problem l.jpg
A Fundamental problem

  • Given a collection of agents (voters)

    • with preferences over different alternatives (allocations, outcomes),

  • how should society evaluate these alternatives and make a decision for all

    • that may be for the will of some voters but against that of others.


Applications l.jpg
Applications

  • Voters elect president from several candidates.

  • National polls for economic or political policy of the government

  • The procedure or rule of election

  • The rank of metasearch engine obtained from those of search engines


Group descisions l.jpg
Group Descisions

How do we make decisions

  • Flip a coin?

  • Dictatorship?

  • Democracy (Majority rule)?

    in a group with

    • Heterogeneity of individual preference


Group decision rules l.jpg
Group Decision Rules

  • Majority rule ,

  • Condorcet paradox (voting cycle)

  • Borda rule


Mathematical model l.jpg
Mathematical model

  • A set of voters V={v1,v2,v3,…,Vn}

  • A set of alternatives or outcomes S={s1,s2,s3,…Sm}, with |S|=m; and

  • A set of preference relation P={R1,R2,R3…Rn}, called a preference profile,

    • the preference relation Ri for each voter i is a permutation (order) of elements in S.


Example 1 majority rule l.jpg
Example 1 Majority Rule

  • 3 rational people have rational preferences over 2 alternatives {x,y}

    Person

    1 2 3

    1stX YX 1 : X>Y

    Pref. i.e.Person 2 : Y>X

    2ndYXY 3 : X>Y

    How to Aggregate their preferences? How to choose?


Slide27 l.jpg

  • Using majority rule.

  • Since more than ½ people (two out of three) prefer x to y.

  • Then the group prefers x to y


Example 2 condorcet paradox l.jpg
Example 2 Condorcet Paradox

  • 3 rational people have rational preferences over 3 alternatives {x,y,z}

    Person

    1 2 3

    1stX YZ 1 : X>Y>Z

    Pref. 2ndYZ Xi.e. Person 2 : Y>Z>X

    3rdZXY 3 : Z>X>Y


Binary paired comparison with majority rule l.jpg
Binary/paired Comparison With Majority rule

  • Person

    1 2 3

    1stX YZ 1 : X>Y

    Pref. 2ndYZ X for (x,y) 2 : Y>X X>Y

    3rdZXY 3 : X>Y

  • Similarly, for (Y,Z) we can get Y>Z; for (Z,X) we can get Z>X.

  • Then X>Y>Z>X (cycling) , Intransitive  Not rational


Slide30 l.jpg


Example 3 borda rule l.jpg
Example 3 Borda Rule alternative can win a majority against all other alternatives.

  • For each voter,

    • associate the number 1 with the most preferred alternative,

    • 2 with the second and so on,

  • Assign to each alternative the number equal to

    • the sum of the numbers the individual voters assigned to the alternative.


Slide32 l.jpg

Person alternative can win a majority against all other alternatives.

1 2 3

1stX(1) Y(1)X(1) X(4) X

Pref. 2ndY(2)X(2) W(2) Y(7)  Y

3rdZ(3)W(3)Z(3) Z(10) W

4thW(4)Z(4)Y(4) W(9) Z

Then We get choice X>Y>W>Z


Slide33 l.jpg


Slide34 l.jpg


Slide35 l.jpg

  • Some variations via binary/paired comparison, we get a tie between all three alternatives with the Borda

    1 with relevant scores available

    allotting each input system a point p to be distributed according to relevance scores of the documents.

    2 Weighted Borda-rule

    Each voter may not have equal effectiveness to the final result. We may set more weight to good quality input systems.


Slide36 l.jpg

  • Condorcet winner algorithm via binary/paired comparison, we get a tie between all three alternatives with the Borda

    It also comes from social choice theory. The Condorcet algorithm says that any candidate that can beat all other candidates in a head-to-head contest (pair-wise comparison) should win the election.


Slide37 l.jpg

  • Step 1, via binary/paired comparison, we get a tie between all three alternatives with the Borda Construct Condorcet Graph.

    For each candidate pair (x,y), there exists an edge from x to y if x would receive at least as many votes as y in a head-to-head contest.

    In Condorcet graph, there is at least one directed edge between every pair of candidates. ( we call the graph is semi-complete)

    It may contains cycles in the graph. This is due to voting paradox of the condorcet voting.


Slide38 l.jpg

  • Step 2 via binary/paired comparison, we get a tie between all three alternatives with the Borda, We form a new acyclic graph from an old cyclic one by contracting all of the nodes in a cycle into one. It is a strongly connected component graph (SCCG).

    Definition of Strongly connected component(SCC):

    A strongly connectedsubgraph, S, of a directed graph, D, such that no vertex or subset of vertices of D can be added to S such that the new subgraph is still strongly connected.


Slide39 l.jpg

The graph is totally orderable at the level of the SCC via binary/paired comparison, we get a tie between all three alternatives with the Borda’s and each SCC is a “pocket” of cycles, within which each candidate is tied.

Step 3, The condorcet-consistent Hamiltonian path is any Hamiltonian path through Condorcet graph.

Definition Hamiltonian path: A path between two vertices of a graph that visits each vertex exactly once.


Slide40 l.jpg

  • Theorem 1 via binary/paired comparison, we get a tie between all three alternatives with the Borda. Every semi-complete graph contains a Hamiltonian path.

  • Theorem 2. Suppose x and y are nodes in a graph g, and that X and Y are nodes of the associated SCCG G such that x X and y Y. If there exists a path from X to Y in G, then every Condorcet path of g has x before y.

    Refer to[Javed A. Aslam, Mark Montague 2001] for proof.


Rank aggregation in metasearch l.jpg
Rank Aggregation in MetaSearch via binary/paired comparison, we get a tie between all three alternatives with the Borda

Here we discussed two cases which using algorithm rooted at social choice theory for MetaSearch rank aggregation.

  • Data fusion track in TREC

    [Javed A. Aslam, Mark Montague 2001] Models for Metasearch

    in SIGIR2001

  • Rank aggregation for web search engine

    [Cynthia Dwork, Ravi Kumar, Moni Naor, D.Sivakumar 2001]

    Rank Aggregation Methods for the Web in WWW10


Data fusion track in trec l.jpg
Data fusion track in TREC via binary/paired comparison, we get a tie between all three alternatives with the Borda

  • TREC (Text Retrieval Conference ,see http://trec.nist.gov/) maintains about 6Gb of SGML tagged text, queries and respective answers for evaluation purposes.

  • The TREC organizers distribute data sets in advance and 50 new queries each year.

  • The competing teams then submit ranked lists of documents that their system gave in response to each query. And these retrieval systems will be evaluated.


Slide43 l.jpg

  • These ranked lists are available for metasearch researchers to download and fuse.

  • For each query, every retrieval system will return top 1000 documents and relevant score is available.

  • Then given these results retrieved by many different retrieval systems, how to aggregate them for better performance?


Previous algorithms l.jpg
Previous algorithms to download and fuse.

  • Min, Max and Average Models

    [Fox and Shaw,1995]

  • Linear Combination Model

    [Bartell 1995]

  • Logistic Regression Model


Example l.jpg
Example to download and fuse.

  • Min, Max and Average model

    The final score of each document d is based on the scores given to d by each input systems (voters).

    Algorithm Final score

    CombMin minimum of individual relevance scores

    CombMed median of individual relevance scores

    CombMax maximum of individual relevance scores

    CombSum sum of individual relevance scores

    CombANZ CombSum / num non-zero relevance scores

    CombMNZ CombSum * num non-zero relevance scores


Slide46 l.jpg

  • Linear Combination Model to download and fuse.

    The final score of document d is a simply linearly (each weighted differently) combining the normalized relevance scores given to each document.


Experiment result on trec model l.jpg
Experiment result on TREC Model to download and fuse.

  • The performance of rank aggregation is evaluated by average precision over the queries

  • Score-based borda-fuse is usually the best method among several borda variant algorithms.

  • It is better than best input system over most of data collection. Such as TREC3, TREC5


Experiment result ii l.jpg
Experiment result II to download and fuse.

  • The performance of rank aggregation is evaluated by average precision over the queries.

  • Condorcet-fusion is the only algorithm that , without training data, ever matches the performance of the best input system over TREC 9.

  • Condorcet-fusion seems particularly sensitive to the dependence of input systems. If the input systems (voters) are too similar, the performance will decrease.


Rank aggregation methods for web l.jpg
Rank aggregation methods for web to download and fuse.

New Challenges: Different from the case in TREC data fusion,

  • The coverage of various search engine is different

  • Thus some highly relevant web pages may not be ranked by some search engines.

  • Therefore, each voter ranks a partial candidate list


Preliminaries l.jpg
Preliminaries to download and fuse.

  • Given a universe U, an ordered list with respect to U is an ordering of a subset S U, i.e., ,with each and is some ordering relation on S.

  • If contains

    • all the elements in U, then it is said to be a full list,

    • otherwise it is called partial list.


Slide51 l.jpg


Slide52 l.jpg


Slide53 l.jpg

  • Distance measures for more than 2 list set S

    Given several full lists , for instance, the normalized Footrule distance of to is given by

    If are partial lists, let U denote the union of elements in and let be a full list with respect to U. Considering the distance between and the projection of with respect to , we have the induced footrule distance


Slide54 l.jpg


Optimal rank aggregation l.jpg
Optimal rank aggregation set S

The question is

Given (full or partial) lists , find a such that is a full list with respect to the union of the elements of

and minimizes

The aggregation obtained by optimizing Kendall distance is called Kemeny optimal aggregation.


Slide56 l.jpg

  • When k>=4,computing the set SKemeny optimal aggregation is NP-hard.

    (please refer to [Cynthia Dwork, Ravi Kumar, Moni Naor, D.Sivakumar 2001] for detailed proof )

    We can use Spearman footrule distance to approximate the Kendall distance.


Slide57 l.jpg

  • Proposition 1 set S For any two full lists , ,

  • Proposition 2if is the Kemeny optimal aggregation of full lists , and optimizes the footrule aggregation, then


Slide58 l.jpg

  • Proposition 3 set SGiven full lists if the median positions of the candidates in the lists form a permutation, then this permutation is a footrule optimal aggregation.

  • Proposition 4 Footrule optimal aggregation of full lists can be computed in polynomial time, specifically, the time to find a minimum cost perfect matching in a bipartite graph.


Slide59 l.jpg

  • The footrule optimal aggregation for full lists can be recast as a minimum cost bipartite matching problem.

  • For partial list, we define the bipartite graph as before, except that the weights are given by scaled footrule distance .

  • The weight W(c,p) of a ranking that places element c at position p is


Scaled footrule aggregation l.jpg
Scaled Footrule aggregation recast as a minimum cost bipartite matching problem.

  • As before , we can solve the minimum cost maximum matching problem on this bipartite graph to obtain the footrule aggregation algorithm for partial lists.

  • We called this method the scaled footrule aggregation (SFO)


Locally kemeny optimal aggregation l.jpg
Locally Kemeny optimal aggregation recast as a minimum cost bipartite matching problem.

  • Since Kemeny optimal aggregation is NP-hard for k>=4, Here we introduce locally Kemeny optimal aggregation.

  • It ensures satisfaction of the extended Condorcet principle and yet remains computationally tractable.


Slide62 l.jpg

  • A full list is recast as a minimum cost bipartite matching problem.locally kemeny optimal aggregation of partial lists if there is no full list that can be obatined from by a single transposition of an adjacent pair of elements and for which


Slide63 l.jpg


Slide64 l.jpg

It has following characteristics. is a procedure that computes a locally Kemeny optimal aggregation of that is maximally consistent with .

  • The Condorcet losers receive low rank, while the Condorcet winners receive high rank.

  • The result disagrees with on the order of any given pair (i,j) of elements only if a majority of those ‘s expressing opinions disagree with on (i,j)

  • For every ,the length d prefix of the output is a local Kemenization of the top d elements in .


ad
  • Login