inside outside algorithm
Download
Skip this Video
Download Presentation
Inside-outside algorithm

Loading in 2 Seconds...

play fullscreen
1 / 42

Inside-outside algorithm - PowerPoint PPT Presentation


  • 338 Views
  • Uploaded on

Inside-outside algorithm. LING 572 Fei Xia 02/28/06. Outline. HMM, PFSA, and PCFG Inside and outside probability Expected counts and update formulae Relation to EM Relation between inside-outside and forward-backward algorithms. HMM, PFSA, and PCFG. PCFG. A PCFG is a tuple:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Inside-outside algorithm' - cala


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
inside outside algorithm

Inside-outside algorithm

LING 572

Fei Xia

02/28/06

outline
Outline
  • HMM, PFSA, and PCFG
  • Inside and outside probability
  • Expected counts and update formulae
  • Relation to EM
  • Relation between inside-outside and forward-backward algorithms
slide4
PCFG
  • A PCFG is a tuple:
    • N is a set of non-terminals:
    • is a set of terminals
    • N1 is the start symbol
    • R is a set of rules
    • P is the set of probabilities on rules
  • We assume PCFG is in Chomsky Norm Form
  • Parsing algorithms:
    • Earley (top-down)
    • CYK (bottom-up)
pfsa vs pcfg
a

b

S1

S2

S3

a

S2

b

S3

ε

PFSA vs.PCFG
  • PFSA can be seen as a special case of PCFG
    • State  non-terminal
    • Output symbol  terminal
    • Arc  context-free rule
    • Path  Parse tree (only right-branch binary tree)

S1

pfsa and hmm
Start

Finish

PFSA and HMM

HMM

Add a “Start” state and a transition from “Start” to any state in HMM.

Add a “Finish” state and a transition from any state in HMM to “Finish”.

the connection between two algorithms
The connection between two algorithms
  • HMM can (almost) be converted to a PFSA.
  • PFSA is a special case of PCFG.
  • Inside-outside is an algorithm for PCFG.
  • Inside-outside algorithm will work for HMM.
  • Forward-backward is an algorithm for HMM.
  • In fact, Inside-outside algorithm is the same as forward-backward when the PCFG is a PFSA.
forward and backward probabilities
on

o1

Ot-1

Xn+1

Xt

Xn

X1

X1

O1

Xt-1

Ot-1

Xt

Ot

Xn

Xn+1

On

Forward and backward probabilities
backward forward prob vs inside outside prob
X1

Xt=Ni

Xt=Ni

Ot-1

Ot

O1

Ol

On

O1

Ot-1

Ot

On

Backward/forward prob vs. Inside/outside prob

X1

PCFG:

PFSA:

Outside

Inside

Forward

Backward

notation
Notation

N1

Nj

wq

w1

wp-1

wp

Wq+1

wm

definitions
Definitions
  • Inside probability: total prob of generating words wp…wq from non-terminal Nj.
  • Outside probability: total prob of beginning with the start symbol N1 and generating and all the words outside wp…wq
  • When p>q,
recap so far
Recap so far
  • Inside probability: bottom-up
  • Outside probability: top-down using the same chart.
  • Probability of a sentence can be calculated in many ways.
inner loop of the inside outside algorithm
Inner loop of the Inside-outside algorithm

Given an input sequence and

  • Calculate inside probability:
    • Base case
    • Recursive case:
  • Calculate outside probability:
    • Base case:
    • Recursive case:
inside outside algorithm cont
Inside-outside algorithm (cont)

3. Collect the counts

4. Normalize and update the parameters

relation to em28
Relation to EM
  • PCFG is a PM (Product of Multi-nominal) Model
  • Inside-outside algorithm is a special case of the EM algorithm for PM Models.
  • X (observed data): each data point is a sentence w1m.
  • Y (hidden data): parse tree Tr.
  • Θ (parameters):
summary
Xt+1

Xt

Nj

Nr

Ns

wp

wd

Wd+1

wq

Summary

Ot

N1

summary cont
Summary (cont)
  • Topology is known:
    • (states, arcs, output symbols) in HMM
    • (non-terminals, rules, terminals) in PCFG
  • Probabilities of arcs/rules are unknown.
  • Estimating probs using EM (introducing hidden data Y)
converting hmm to pcfg
Converting HMM to PCFG
  • Given an HMM=(S, Σ, π, A, B), create a PCFG=(S1, Σ1,S0, R, P) as follows:
    • S1=
    • Σ1=
    • S0=Start
    • R=
    • P:
path parse tree
Path  Parse tree

oT

o1

o2

XT+1

XT

X1

X2

Start

D0

X1

D12

X2

BOS

o1

XT

DT,T+1

XT+1

ot

EOS

o utside probability
q=T

(j,i),(p,t)

Outside probability

Outside prob for Nj

Outside prob for Dij

q=p

(p,t)

inside probability
q=T

(j,i),(p,t)

Inside probability

Inside prob for Nj

Inside prob for Dij

q=p

(p,t)

slide38
Estimating

Renaming: (j,i), (s,j),(p,t),(m,T)

slide39
Estimating

Renaming: (j,i), (s,j),(p,t),(m,T)

slide40
Estimating

Renaming: (j,i), (s,j),(p,t),(m,T)

slide41
Calculating

Renaming: (j,i), (s,j),(w,o),(m,T)

calculating
Calculating

Renaming (j,i_j), (s,j),(p,t),(h,t),

(m,T),(w,O), (N,D)

ad