Random walks, eigenvectors, and their applications to Information Retrieval, Natural Language Processing, and Machine Le

1 / 65

# Random walks, eigenvectors, and their applications to Information Retrieval, Natural Language Processing, and Machine Le - PowerPoint PPT Presentation

Random walks, eigenvectors, and their applications to Information Retrieval, Natural Language Processing, and Machine Learning. Dragomir R. Radev University of Michigan radev@umich.edu Guest lecture in SI 614 March 7, 2006. INTRODUCTION. Social networks. Induced by a relation

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about 'Random walks, eigenvectors, and their applications to Information Retrieval, Natural Language Processing, and Machine Le' - apu

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### Random walks, eigenvectors, and their applications to Information Retrieval, Natural Language Processing, and Machine Learning

University of Michigan

Guest lecture in SI 614

March 7, 2006

Social networks
• Induced by a relation
• Symmetric or not
• Examples:
• Friendship networks
• Board membership
• Citations
• Power grid of the US
• WWW
Prestige and centrality
• Degree centrality: how many neighbors each node has.
• Closeness centrality: how close a node is to all of the other nodes
• Betweenness centrality: based on the role that a node plays by virtue of being on the path between two other nodes
• Eigenvector centrality: the paths in the random walk are weighted by the centrality of the nodes that the path connects.
• Prestige = same as centrality but for directed graphs.

MARKOV CHAINS

AND

RANDOM WALKS

1-d random walks
• Drunkard’s walk:
• Start at position 0 on a line
• What is the prob. of reaching 0 before reaching 5? Same for penny matching.
• Harmonic functions:
• P(0) = 0
• P(N) = 1
• P(x) = 1/2p(x-1)+1/2p(x+1), for 0<x<N

0

1

2

3

4

5

1

6

8

2

7

5

3

4

Graph-based representations

Square connectivity(incidence) matrix

Graph G (V,E)

Markov chains
• A homogeneous Markov chain is defined by an initial distribution x and a Markov kernel E.
• Path = sequence (x0, x1, …, xn).Xi = xi-1*E
• The probability of a path can be computed as a product of probabilities for each step i.
• Random walk = find Xjgiven x0, E, and j.
Stationary solutions
• The fundamental Ergodic Theorem for Markov chains [Grimmett and Stirzaker 1989] says that the Markov chain with kernel E has a stationary distribution p under three conditions:
• E is stochastic
• E is irreducible
• E is aperiodic
• To make these conditions true:
• All rows of E add up to 1 (and no value is negative)
• Make sure that E is strongly connected
• Make sure that E is not bipartite
• Example: PageRank [Brin and Page 1998]: use “teleportation”

t=0

1

6

8

2

7

t=1

5

3

4

Example

This graph E has a second graph E’(not drawn) superimposed on it:E’ is the uniform transition graph.

EIGENVALUES

ANDEIGENVECTORS

Eigenvectors and eigenvalues
• An eigenvector is an implicit “direction” for a matrix

where v (eigenvector)is non-zero, though λ (eigenvalue) can be any complex number in principle

• Computing eigenvalues:
• Example
Stochastic matrices
• Stochastic matrices: each row (or column) adds up to 1 and no value is less than 0. Example:
• The largest eigenvalue of a stochastic matrix E is real: λ1 = 1.
• For λ1, the left (principal) eigenvector is p, the right eigenvector = 1
• In other words, ETp = p.
Computing the stationary distribution

functionPowerStatDist (E):

begin

p(0) = u; (or p(0) = [1,0,…0])

i=1;

repeat

p(i) = ETp(i-1)

L = ||p(i)-p(i-1)||1;

i = i + 1;

untilL < 

returnp(i)

end

Solution for thestationary distribution

t=0

1

6

8

2

7

t=1

5

3

4

t=10

Example

PageRank
• Imagine a random walk on a strongly connected Web graph.
• Aimless surfer will reach any page after a high number of steps.
• Visiting “prestigious pages” increases the speed of convergence.
Prestige
• Adjacency matrix E where E[i, j]=1 if document i cites document j.
• Every node has a prestige value p[v]
PageRank
• Described in “The anatomy of a large-scale hypertextual web search engine” by Brin and Page (WWW1998)
• Independent of query (although more recent work by Haveliwala (WWW 2002) has also identified topic-based PageRank.
Co-citation
• If document u cites both v and w, then v and w are co-cited.
• The entry E(u,w) in the (ETE) matrix is the co-citation index of v and w.
HITS
• Query-dependent model (Kleinberg 97)
• Hubs and authorities (e.g., cars, Honda)
• Algorithm
• obtain root set using input query
• expanded the root set by radius one
• run iterations on the hub and authority scores together
• report top-ranking authorities and hubs
• Currently used in Teoma
Some pointers
• http://jung.sourceforge.net/applet/rankingdemo.html
• Highest pagerank scores:http://en.wikipedia.org/wiki/List_of_websites_with_a_high_PageRank
• http://www.pagerank.dk/
• http://www.scriptet.com/improve-pagerank.html
• http://en.wikipedia.org/wiki/Page_rank

LEXICAL CENTRALITY

Centrality in summarization
• Extractive summarization (pick k sentences that are most representative of a collection of n sentences
• Motivation: capture the most central words in a document or cluster
• Centroid score [Radev & al. 2000, 2004a]
• Alternative methods for computing centrality?
Sample multidocument cluster

(DUC cluster d1003t)

1 (d1s1) Iraqi Vice President Taha Yassin Ramadan announced today, Sunday, that Iraq refuses to back down from its decision to stop cooperating with disarmament inspectors before its demands are met.

2 (d2s1) Iraqi Vice president Taha Yassin Ramadan announced today, Thursday, that Iraq rejects cooperating with the United Nations except on the issue of lifting the blockade imposed upon it since the year 1990.

3 (d2s2) Ramadan told reporters in Baghdad that "Iraq cannot deal positively with whoever represents the Security Council unless there was a clear stance on the issue of lifting the blockade off of it.

4 (d2s3) Baghdad had decided late last October to completely cease cooperating with the inspectors of the United Nations Special Commission (UNSCOM), in charge of disarming Iraq's weapons, and whose work became very limited since the fifth of August, and announced it will not resume its cooperation with the Commission even if it were subjected to a military operation.

5 (d3s1) The Russian Foreign Minister, Igor Ivanov, warned today, Wednesday against using force against Iraq, which will destroy, according to him, seven years of difficult diplomatic work and will complicate the regional situation in the area.

6 (d3s2) Ivanov contended that carrying out air strikes against Iraq, who refuses to cooperate with the United Nations inspectors, ``will end the tremendous work achieved by the international group during the past seven years and will complicate the situation in the region.''

7 (d3s3) Nevertheless, Ivanov stressed that Baghdad must resume working with the Special Commission in charge of disarming the Iraqi weapons of mass destruction (UNSCOM).

8 (d4s1) The Special Representative of the United Nations Secretary-General in Baghdad, Prakash Shah, announced today, Wednesday, after meeting with the Iraqi Deputy Prime Minister Tariq Aziz, that Iraq refuses to back down from its decision to cut off cooperation with the disarmament inspectors.

9 (d5s1) British Prime Minister Tony Blair said today, Sunday, that the crisis between the international community and Iraq ``did not end'' and that Britain is still ``ready, prepared, and able to strike Iraq.''

10 (d5s2) In a gathering with the press held at the Prime Minister's office, Blair contended that the crisis with Iraq ``will not end until Iraq has absolutely and unconditionally respected its commitments'' towards the United Nations.

11 (d5s3) A spokesman for Tony Blair had indicated that the British Prime Minister gave permission to British Air Force Tornado planes stationed in Kuwait to join the aerial bombardment against Iraq.

Cosine between sentences
• Let s1 and s2 be two sentences.
• Let x and y be their representations in an n-dimensional vector space
• The cosine between is then computed based on the inner product of the two.
• The cosine ranges from 0 to 1.

d3s3

d2s3

d3s2

d3s1

d1s1

d4s1

d5s1

d2s1

d5s2

d5s3

d2s2

Lexical centrality (t=0.3)

d3s3

d2s3

d3s2

d3s1

d1s1

d4s1

d5s1

d2s1

d5s2

d5s3

d2s2

Lexical centrality (t=0.2)

d3s2

d3s3

d2s3

d3s2

d4s1

d3s1

d2s1

d1s1

d4s1

d5s1

d2s1

d5s2

d5s3

d2s2

Lexical centrality (t=0.1)

Sentences vote for the most central sentence!

Need to worry about diversity reranking.

LexRank

• T1…Tn are pages that link to A, c(Ti) is the outdegree of pageTi, and N is the total number of pages.
• d is the “damping factor”, or the probability that we “jump” to a far-away node during the random walk. It accounts for disconnected components or periodic graphs.
• When d = 0, we have a strict uniform distribution.When d = 1, the method is not guaranteed to converge to a unique solution.
• Typical value for d is between [0.1,0.2] (Brin and Page, 1998).

BIASED LEXRANK

A small plane has hit a skyscraper in central Milan, setting the top floors of the 30-story building on fire, an Italian journalist told CNN. The crash by the Piper tourist plane into the 26th floor occurred at 5:50 p.m. (1450 GMT) on Thursday, said journalist Desideria Cavina. The building houses government offices and is next to the city's central train station. Several storeys of the building were engulfed in fire, she said. Italian TV says the crash put a hole in the 25th floor of the Pirelli building, and that smoke is pouring from the opening. Police and ambulances are at the scene. Many people were on the streets as they left work for the evening at the time of the crash. Police were trying to keep people away, and many ambulances were on the scene. There is no word yet on casualties.

A small plane has hit a skyscraper in central Milan, setting the top floors of the 30-story building on fire, an Italian journalist told CNN. The crash by the Piper tourist plane into the 26th floor occurred at 5:50 p.m. (1450 GMT) on Thursday, said journalist Desideria Cavina. The building houses government offices and is next to the city's central train station. Several storeys of the building were engulfed in fire, she said. Italian TV showed a hole in the side of the Pirelli building with smoke pouring from the opening. RAI state TV reported that the plane had apparently radioed an SOS because of engine trouble. Earlier though, in Rome, the senate's president, Marcello Pera, said it "very probably" appeared to be a terrorist attack. Police and ambulances are at the scene. Many people were on the streets as they left work for the evening at the time of the crash. Police were trying to keep people away, and many ambulances were on the scene. There is no word yet on casualties. TV pictures from the scene evoked horrific memories of the September 11 attacks on the World Trade Center in New York and the collapse of the building's twin towers. "I heard a strange bang so I went to the window and outside I saw the windows of the Pirelli building blown out and then I saw smoke coming from them," said Gianluca Liberto, an engineer who was working in the area told Reuters. The building is known as the Pirelli skyscraper but the Italian tyre and cable company does not operate out of the building. It is one of the symbols of Italy's financial capital and is one of the world's tallest concrete buildings, designed between 1955 and 1960.

A small plane crashed into a skyscraper in downtown Milan today, setting several floors of the 30-story building on fire. The plane crashed into the 25th floor of the Pirelli building in downtown Milan. The weather was clear at the time of the crash. Smoke poured from the opening as police and ambulances rushed to the area. The president of the Italian Senate, Marcello Pera, told Italian television it "very probably" appeared to be a terrorist attack but soon afterwards his spokesman said it was probably an accident. A transport official told Reuters the plane had reported problems with its undercarriage and was circling the city ahead of trying to land at a local airport. The Pirelli building houses the administrative offices of the local Lombardy region and sits next to the city's central train station. It is constructed of concrete and glass. The crash happened just before rush hour, as office workers were closing their day.

A small airplane crashed into a government building in heart of Milan, setting the top floors on fire, Italian police reported. There were no immediate reports on casualties as rescue workers attempted to clear the area in the city’s financial district. Few details of the crash were available, but news reports about it immediately set off fears that it might be a terrorist act akin to the Sept. 11 attacks in the United States. Those fears sent U.S. stocks tumbling to session lows in late morning trading. Witnesses reported hearing a loud explosion from the 30-story office building, which houses the administrative off ices of the local Lombardy region and sits next to the city s central train station. Italian state television said the crash put a hole in the 25th floor of the Pirelli building. News reports said smoke poured from the opening. Police and ambulances rushed to the building in downtown Milan. No further details were immediately available.

Un aereo da turismo, un Piper si è schiantato questo pomeriggio a Milano, poco prima delle 18, contro il grattacielo Pirelli, sede anche della Regione Lombardia (il presidente della Regione, Roberto Formigoni, è in missione ufficiale in India con una delegazione della regione). Lo si è appreso in ambienti investigativi. L' impatto sarebbe avvenuto attorno al 25/o piano dei 30 del grattacielo. Almeno sei piani alla vista risultano sventrati. I detriti sono stati lanciati dal'esplosione a una quarantina di metri intorno all'edificio. In tutta l'area attorno al grattacielo Pirelli lecomunicazioni telefoniche anche via cellulare sono interrotte o quasi impossibili. La Borsa ha sospeso la seduta serale a Piazza Affari dopo lo schianto dell'aereo da turismo, anche il presidente Bush è stato subito avvertito dell'espolosione al Pirellone.«Con molta probabilità si tratta di un attentato». Lo ha detto Marcello Pera aprendo la seduta a Palazzo Madama. Ma secondo quanto si è appreso, l'aereo da turismo era probabilmente in avaria: il pilota, infatti, avrebbe lanciato l'SOS, raccolto dalla torre di controllo di Linate.

CNN 4/18/02 12:22pm; CNN 4/18/02 12:32pm; ABCNews 4/18/02 1:00pm;

MSNBC 4/18/02 1:00pm; La Stampa 4/18/02 12:45pm

Questions from the Milan cluster

1. How many people were injured?

2. How many people were killed? (age, number, gender, description)

3. Was the pilot killed?

4. Where was the plane coming from?

5. Was it an accident (technical problem, illness, terrorist act)?

6. Who was the pilot? (age, number, gender, description)

7. When did the plane crash?

8. How tall is the Pirelli building?

9. Who was on the plane with the pilot?

10. Did the plane catch fire before hitting the building?

11. What was the weather like at the time of the crash?

12. When was the building built?

13. What direction was the plane flying?

14. How many people work in the building?

15. How many people were in the building at the time of the crash?

16. How many people were taken to the hospital?

17. What kind of aircraft was used?

Wnt signaling

Glycogen synthase kinase-3 (GSK-3) and CK1 (casein kinase 1) alpha phosphorylate Arm (Armadillo, -catenin) andcause it to degrade.

Axin also binds to the phosphatase PP2A

PP2A activity inhibits Wnt signaling

Protein Regulatory Network Recognition

Hsu 1999, Li 2001, Yanagawa 2002, Liu2002, Nusse 2003

Biased lexrank
• Diversity-based summaries (cf. Carbonell&Goldstein)
• Query-based summaries: Given: a cluster of documents + a set of sample sentences that express certain facts (e.g., protein interactions or answers to questions like “What type of aircraft was involved?)

RW METHODS FOR

CLASSIFICATION

PP attachment

Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29. Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group. Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields PLC , was named a nonexecutive director of this British industrial conglomerate.

A form of asbestos once used to make Kent cigarette filters has caused a high percentage of cancer deaths among a group of workers exposed to it more than 30 years ago , researchers reported . The asbestos fiber , crocidolite , is unusually resilient once it enters the lungs , with even brief exposures to it causing symptoms that show up decades later , researchers said . Lorillard Inc. , the unit of New York-based Loews Corp. that makes Kent cigarettes , stopped using crocidolite in its Micronite cigarette filters in 1956 . Although preliminary findings were reported more than a year ago , the latest results appear in today 's New England Journal of Medicine , a forum likely to bring new attention to the problem .

• High vs. low attachment

V x02_join x01_board x0_as x11_director

N x02_is x01_chairman x0_of x11_entitynam

N x02_name x01_director x0_of x11_conglomer

N x02_caus x01_percentag x0_of x11_death

V x02_us x01_crocidolit x0_in x11_filter

V x02_bring x01_attent x0_to x11_problem

1 Ω

Electrical networks and random walks

c

• Ergodic (connected) Markov chain with transition matrix P

1 Ω

1 Ω

w=Pw

b

a

0.5 Ω

0.5 Ω

d

From Doyle and Snell 2000

1 Ω

Electrical networks and random walks

c

1 Ω

1 Ω

b

a

0.5 Ω

0.5 Ω

• vxis the probability that a random walk starting at x will reach a before reaching b.

d

• The random walk interpretation allows us to use Monte Carlo methods to solve electrical circuits.

1 V

Example

reported earnings for quarter

reported loss for quarter

posted loss for quarter

posted loss of quarter

posted loss of million

V

?

?

?

N

*

Example

V

reported earnings for quarter

posted earnings for quarter

n1

n2

v

N

posted loss of million

p

Wu and Huberman 2004. Finding communities in linear time: a physics approach. The European Physics Journal B, 38:331--338
• Kurland and Lee 2005 – random walks with generation probabilities
• Erkan 2006 – random walk based clustering
• Zhu and Ghahramani – work on ML methods on graphs
• Doyle and Snell – random walks and electric networks

ID LPR (0.1) LPR (0.2) LPR (0.3) Centroid

d1s1 0.6007 0.6944 1.0000 0.7209

d2s1 0.8466 0.7317 1.0000 0.7249

d2s2 0.3491 0.6773 1.0000 0.1356

d2s3 0.7520 0.6550 1.0000 0.5694

d3s1 0.5907 0.4344 1.0000 0.6331

d3s2 0.7993 0.8718 1.0000 0.7972

d3s3 0.3548 0.4993 1.0000 0.3328

d4s1 1.0000 1.0000 1.0000 0.9414

d5s1 0.5921 0.7399 1.0000 0.9580

d5s2 0.6910 0.6967 1.0000 1.0000

d5s3 0.5921 0.4501 1.0000 0.7902

Cosine centrality vs. centroid centrality
Evaluation metrics
• Difficult to evaluate summaries
• Intrinsic vs. extrinsic evaluations
• Extractive vs. non-extractive evaluations
• Manual vs. automatic evaluations
• ROUGE = n-gram recall for different values of n.
• Example:
• Reference = “The cat in the hat”
• System = “The cat wears a top hat”
• 1-gram recall = 3/5; 2-gram recall = 1/4;3,4-gram recall = 0
• ROUGE-W = longest common subsequence
• Example above: 3/5

Centroid

Degree

LexPageRank

Evaluation results

Centroid: C0.5, C10, C1.5, C1, C2.5, C2

Degree: D0.5T0.1, D0.5T0.2, D0.5T0.3, D1.5T0.1, D1.5T0.2, D1.5T0.3, D1T0.1, D1T0.2, D1T0.3

LexRank: Lr0.5T0.1, Lr0.5T0.2, Lr0.5t0.3, Lr1.5t0.1, Lr1.5t0.2, Lr1.5t0.3, Lr1T0.1, Lr1T0.2, Lr1T0.3

Rouge-1

Lr1.5t0.1 0.400

Lr1.5t0.2 0.400

Lr1T0.2 0.396

C1 0.382

Rouge-4

Lr1.5t0.1 0.124

Lr1.5t0.2 0.124

Lr1T0.2 0.124

C2 0.118

Rouge-2

Lr1.5t0.2 0.115

D1.5T0.2 0.114

D1T0.2 0.113

C1.5 0.099

Corpus
• 20 clusters: 11+3+6
• 341 total questions
• Interjudge agreement: Kappa = 0.68 (with 2 judges): does sentence X contain the answer to question Y.
TRDR Results
• TRDR = total reciprocal document rank
• Baseline: 0.867
• Mixture model: 0.991 (p-value = 0.062)
• For similarity = 0.20 and bias = 0.95 (estimated on the devtest)
Some statistics

20801 training, 3097 test, 66 different prepositions

Baselines and related work
• Always N: 59.0%
• Based on preposition only: 72.2%
• TBL [Brill & Resnik 94]: 81.8%
• Backoff [Collins & Brooks 95]: 84.5 %
• Boosting [Abney & al. 99]: 84.6%
• Dependency-based nearest neighbors [Zhao & Lin 04]: 86.5 %
• 3-hop random walk using wordnet and external noisy corpus [Toutanova & al. 04]: 87.5%
• Human (4 words only): 88.2 %
• Human (whole sent): 93.2 %
Current results
• PP attachment (full; 4039 test data points)

a

A

B

b

Models of the Web
• Evolving networks: fundamental object of statistical physics, social networks, mathematical biology, and epidemiology
• Erdös/Rényi 59, 60
• Barabási/Albert 99
• Watts/Strogatz 98
• Kleinberg 98
• Menczer 02