Incrementally learning parameter of stochastic cfg using summary stats
This presentation is the property of its rightful owner.
Sponsored Links
1 / 10

Incrementally Learning Parameter of Stochastic CFG using Summary Stats PowerPoint PPT Presentation


  • 47 Views
  • Uploaded on
  • Presentation posted in: General

Incrementally Learning Parameter of Stochastic CFG using Summary Stats. Written by:Brent Heeringa Tim Oates. Goals:. To learn the syntax of utterances Approach : SCFG (Stochastic Context Free Grammar) M=<V,E,R,S> V-finite set of non-terminal E-finite set of terminals

Download Presentation

Incrementally Learning Parameter of Stochastic CFG using Summary Stats

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Incrementally learning parameter of stochastic cfg using summary stats

Incrementally Learning Parameter of Stochastic CFG using Summary Stats

Written by:Brent Heeringa

Tim Oates


Goals

Goals:

  • To learn the syntax of utterances

    Approach:

  • SCFG (Stochastic Context Free Grammar)

    M=<V,E,R,S>

    V-finite set of non-terminal

    E-finite set of terminals

    R-finite set of rules, each r has p(r).

    Sum of p(r) of the same left-hand side = 1

    S-start symbol


Problems with most scfg learning algorithms

Problems with most SCFG Learning Algorithms

1)Expensive storage: need to store a corpus of complete sentences

2)Time-consuming: algorithms needs to repeat passes throughout all data


Learning scfg

Learning SCFG

  • Inducing context-free structure from corpus(sentences)

  • Learning – the production(rules) probabilities


Learning scfg cont

General method: Inside/Outside algorithm

Expectation-Maximization (EM)

Find expectation of rules

Maximize the likelihood given both expectation & corpus

Disadvantage of Inside/Outside algo.

Entire sentence corpus must be stored using some representation(eg. chart parse)

Expensive storage (unrealistic for human agent!)

Learning SCFG –Cont


Proposed algorithm

Proposed Algorithm

  • Use Unique Normal Form (UNF)

    • Replace all terminal A-z to 2 new rules

      • A->D p[A->D]=p[A->z]

      • D-> z p[D->z]=1

    • No two productions have the same right hand side


Learning scfg proposed algorithm cont

Learning SCFG- Proposed Algorithm -cont

  • Use Histogram

    • Each rule has 2 histograms (Hor, HLr)


Proposed algorithm cont

Proposed Algorithm -cont

  • Hor -contructed when parsing sentences in O

  • HLr- -will continue to be updated throughout learning process

  • HLr rescale to fixed size h

    • Why?!

    • Recently used rules has more impact on histogram


  • Comparing between h l r h o r

    Comparing between HLr & Hor

    • Relative entropy

    • T decrease- increase prob of rules used

      • (if s large, increase prob of rules used when parsing last sentence )

    • T increase- decrease prob of rules used

      (eg pt+1(r)=0.01* p t+1(r)


    Comparing inside outside algo with the proposed algorithm

    Inside/Outside

    O(n3)

    Good

    3-5 iterations

    Bad

    Need to store complete sentence corpus

    Proposed Algo

    O(n3)

    Bad

    500-1000 iterations

    Good

    Memory requirements is constant!

    Comparing Inside/Outside Algo with the proposed algorithm


  • Login