CS b553 : A lgorithms for Optimization and Learning

1 / 37

# CS b553 : A lgorithms for Optimization and Learning - PowerPoint PPT Presentation

CS b553 : A lgorithms for Optimization and Learning. Variable Elimination. Last Time. Variable elimination on polytrees Top down inference Linear in size of network Variable elimination in general No guarantees… NP hard in worst case… but when?. Variable Elimination in General Networks.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about ' CS b553 : A lgorithms for Optimization and Learning' - ping

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### CS b553: Algorithms for Optimization and Learning

Variable Elimination

Last Time
• Variable elimination on polytrees
• Top down inference
• Linear in size of network
• Variable elimination in general
• No guarantees…
• NP hard in worst case… but when?
Variable Elimination in General Networks

Coherence

Difficulty

Intelligence

SAT

Letter

Job

Happy

Variable Elimination in General Networks

Coherence

Difficulty

Intelligence

SAT

Letter

Job

Happy

Joint distribution
• P(X) = P(C)P(D|C)P(I)P(G|I,D)P(S|I)P(L|G) P(J|L,S)P(H|G,J)
• Apply elimination ordering C,D,I,H,G,S,L
Going through VE
• P(X) = P(C)P(D|C)P(I)P(G|I,D)P(S|I)P(L|G) P(J|L,S)P(H|G,J)
• Apply elimination ordering C,D,I,H,G,S,L
• 1(D)=SCP(C)P(D|C)
Going through VE
• SCP(X) = 1(D)P(I)P(G|I,D)P(S|I)P(L|G) P(J|L,S)P(H|G,J)
• Apply elimination ordering C,D,I,H,G,S,L
• 1(D)=SCP(C)P(D|C)
Going through VE
• SCP(X) = 1(D)P(I)P(G|I,D)P(S|I)P(L|G) P(J|L,S)P(H|G,J)
• Apply elimination ordering C,D,I,H,G,S,L
• 2(G,I)=SD1(D)P(G|I,D)
Going through VE
• SC,DP(X) = 2(G,I)P(I)P(S|I)P(L|G) P(J|L,S)P(H|G,J)
• Apply elimination ordering C,D,I,H,G,S,L
• 2(G,I)=SD1(D)P(G|I,D)
Going through VE
• SC,DP(X) = 2(G,I)P(I)P(S|I)P(L|G) P(J|L,S)P(H|G,J)
• Apply elimination ordering C,D,I,H,G,S,L
• 3(G,S)=SI2(G,I)P(I)P(S|I)
Going through VE
• SC,D,IP(X) = 3(G,S)P(L|G)P(J|L,S)P(H|G,J)
• Apply elimination ordering C,D,I,H,G,S,L
• 3(G,S)=SI2(G,I)P(I)P(S|I)
Going through VE
• SC,D,IP(X) = 3(G,S)P(L|G)P(J|L,S)P(H|G,J)
• Apply elimination ordering C,D,I,H,G,S,L
• 4(G,J)=SHP(H|G,J)

What values does this factor store?

Going through VE
• SC,D,I,HP(X) = 3(G,S)P(L|G)P(J|L,S)4(G,J)
• Apply elimination ordering C,D,I,H,G,S,L
• 4(G,J)=SHP(H|G,J)
Going through VE
• SC,D,I,HP(X) = 3(G,S)P(L|G)P(J|L,S)4(G,J)
• Apply elimination ordering C,D,I,H,G,S,L
• 5(S,L,J)=SG3(G,S)P(L|G)4(G,J)
Going through VE
• SC,D,I,H,GP(X) = 5(S,L,J)P(J|L,S)
• Apply elimination ordering C,D,I,H,G,S,L
• 5(S,L,J)=SG3(G,S)P(L|G)4(G,J)
Going through VE
• SC,D,I,H,GP(X) = 5(S,L,J)P(J|L,S)
• Apply elimination ordering C,D,I,H,G,S,L
• 6(L,J)=SS 5(S,L,J)P(J|L,S)
Going through VE
• SC,D,I,H,G,SP(X) = 6(L,J)
• Apply elimination ordering C,D,I,H,G,S,L
• 6(L,J)=SS 5(S,L,J)
Going through VE
• SC,D,I,H,G,SP(X) = 6(L,J)
• Apply elimination ordering C,D,I,H,G,S,L
• 7(J)=SL 6(S,L)
Going through VE
• SC,D,I,H,G,S,LP(X) = 7(J)
• Apply elimination ordering C,D,I,H,G,S,L
• 7(J)=SL 6(L,J)
Comparing Orderings
• Consider G,I,S,L,H,C,D
Understanding VE: From BNs to Undirected Graphs
• Consider each factor as a variable i
• Draw an edge between any variables appearing in the same factor
Building the Undirected Graph

P(C)

Coherence

P(I)

P(D|C)

Difficulty

Intelligence

P(S|I)

P(G|I,D)

SAT

P(L|G)

Letter

P(J|S,L)

Job

Happy

P(H|G,J)

Building the Undirected Graph

P(C)

Coherence

P(I)

P(D|C)

Difficulty

Intelligence

P(S|I)

P(G|I,D)

SAT

P(L|G)

Letter

P(J|S,L)

Job

Happy

P(H|G,J)

Building the Undirected Graph

Coherence

Difficulty

Intelligence

SAT

Letter

Job

Happy

Variable Elimination

Coherence

Difficulty

Intelligence

SAT

Letter

Job

Happy

Variable Elimination

Difficulty

Intelligence

SAT

Letter

Job

Happy

Variable Elimination

Difficulty

Intelligence

SAT

Letter

Job

Happy

Variable Elimination

Intelligence

SAT

Letter

Job

Happy

Variable Elimination

Intelligence

SAT

Letter

Job

Happy

Variable Elimination

New fill edge

SAT

Letter

Job

Happy

Variable Elimination

SAT

Letter

Job

Happy

Variable Elimination

SAT

Letter

Job

Variable Elimination

SAT

Letter

Job

Induced Graph from a VE ordering

Coherence

Difficulty

Intelligence

SAT

Letter

Job

Happy

Induced Graph from a VE ordering

Coherence

Difficulty

Intelligence

SAT

• Theorem:
• The scope of every intermediate factor in VE is a clique in the induced graph
• Every maximal clique in the induced graph is the scope of an intermediate factor

Letter

Job

Happy

Determining Optimal orderings
• Again, NP hard!
• Good heuristics in practice:
• Min-neighbors, min-fill, etc
• Search among elimination orderings while counting size of introduced factors
• Greedy search often works well