modular approaches to crossword puzzles and other language games l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Modular Approaches to Crossword Puzzles and Other Language Games PowerPoint Presentation
Download Presentation
Modular Approaches to Crossword Puzzles and Other Language Games

Loading in 2 Seconds...

play fullscreen
1 / 50

Modular Approaches to Crossword Puzzles and Other Language Games - PowerPoint PPT Presentation


  • 317 Views
  • Uploaded on

Modular Approaches to Crossword Puzzles and Other Language Games Michael L. Littman Rutgers University mlittman@cs.rutgers.edu Motivation Software that can understand language. Question answering Constructing databases from text Natural-language interaction

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Modular Approaches to Crossword Puzzles and Other Language Games' - MikeCarlo


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
modular approaches to crossword puzzles and other language games

Modular Approaches toCrossword Puzzles and Other Language Games

Michael L. Littman

Rutgers University

mlittman@cs.rutgers.edu

Crosswords and Constraint Satisfaction

motivation
Motivation

Software that can understand language.

  • Question answering
  • Constructing databases from text
  • Natural-language interaction
  • Automatic summarization/briefing

How can programs represent meaning?

Crosswords and Constraint Satisfaction

traditional approach
Traditional Approach

~like(me, bananas) ^

respect(me, x situation(x) ^ humorous(bananas,x))

I don’t really like bananas, but have long respected their humorous potential.

Semantics tricky; word meanings informal.

Behavioral approach: do something with it.

Crosswords and Constraint Satisfaction

language games
Language Games

Like other games:

  • Evaluation process clean.
  • Challenging (and fun!) for people.

Unlike logical games:

  • Meaning matters!
  • No closed world assumption; messy.
  • Learning necessary... moving target.

Machine performance far from humans’.

Crosswords and Constraint Satisfaction

word games
Word Games

Super-human performance common:

  • Scrabble™: Maven, near-perfect (Sheppard 02)
  • Boggle™: millisecond solutions (Boyan 98)
  • Hangman (Littman 00)
    • 99.97% 9-letter words under 5 guesses
    • 1.35 misses on average

Crosswords and Constraint Satisfaction

trivial pursuit
Trivial Pursuit™

Race around board, answer questions.

Categories: Geography, Entertainment, History, Literature, Science, Sports

Crosswords and Constraint Satisfaction

wigwam
Wigwam

QA via AQUA (Abney et al. 00)

  • back off: word match in order helps score.
  • “When was Amelia Earhart's last flight?”
    • 1937, 1897 (birth), 1997 (reenactment)
  • Named entities only, 100G of web pages

Move selection via MDP (Littman 00)

  • Estimate category accuracy.
  • Minimize expected turns to finish.

Crosswords and Constraint Satisfaction

modular approach to qa
Modular Approach to QA

High-performance question answering system uses a variety of approaches:

  • huge corpus of text on many topics
  • database of questions and answers
  • tables of facts
  • combines multiple extraction methods

Meaning has many faces

Crosswords and Constraint Satisfaction

wigwam s knowledge
Wigwam’s Knowledge

wigwam me trivia web

arts & literature .3 .6 .6 .9

entertainment .3 .3 .5 .9

science & nature .2 .7 .7 .7

geography .1 .2 .4 .9

history .1 .2 .5 .9

sports & leisure .025 .6 .7 .4

~turns/game 414 48 22 8

Crosswords and Constraint Satisfaction

who wants to be a millionaire
Who Wants to Be a Millionaire

“You know, we ought to enter her in one of those TV quiz shows. We could make a fortune.” (Danny Dunn in Williams & Abrashkin 58)

Mult. choice questions, increasing difficulty

  • 100, 200, 300, 500, 1000
  • 2000, 4000, 8000, 16000, 32000
  • 64000, 125000, 250000, 500000, 1000000

Crosswords and Constraint Satisfaction

question answering approach
Question Answering Approach

Choose highest ranked choice.

  • 75%, 68%, 56% (Clarke, Cormack & Lynam 01)

Expected value (always go on):

  • $3,689
  • Most value due to (rare) $1M.

People:

  • $97,357

Crosswords and Constraint Satisfaction

the humble crossword
The Humble Crossword

Crosswords and Constraint Satisfaction

nyt saturday october 10 th 1998
NYT, Saturday, October 10th, 1998

Crosswords and Constraint Satisfaction

variety of clue types
Variety of Clue Types

ThesaurusCut off _ _ _ _ _ _ _ _

Puns & WordplayMonk’s head? _ _ _ _ _

Arts & Literature“Foundation

Trilogy” author _ _ _ _ _ _

Popular CulturePal of Pooh _ _ _ _ _ _

EncyclopedicMountain known

locally as Chomolungma _ _ _ _ _ _ _

CrosswordeseKind of coal or coat _ _ _

ISOLATED

ABBOT

ASIMOV

TIGGER

EVE REST

PEA

Crosswords and Constraint Satisfaction

p roverb system design
PROVERB: System Design

Candidate generation(Keim et al. 99)

  • Like information retrieval: clue implies target
  • Variety of approaches used simultaneously

Merging

  • Like meta search engine: create master list

Grid filling(Shazeer et al. 99)

  • Like constraint satisfaction: fit answer to grid

Probabilities are the common language

Crosswords and Constraint Satisfaction

p roverb system architecture
PROVERB System Architecture

Crosswords and Constraint Satisfaction

modules cluedb
Modules: ClueDB

Nymph pursuer: SATYR Bugs pursuer: ELMER

Nymph chaser: SATYR Place for an ace: SLEEVE

Highball ingredient: RYEHighball ingredient: ICE

exact: Highball ingredient: RYE

partial: Ace place?: SLEEVE

TransModule: Bugs chaser: ELMER

AlsoDijkstra[1-2], d[1-2]c, lsicwdb

X chaser Xpursuer

Crosswords and Constraint Satisfaction

cluedb comparison

Available at

.com

ClueDB Comparison

exact TransModule partial

Coverage 40.3% 73.0% 92.6%

Accuracy 91.4% 79.8% 71.0%

# Returned 1.3 1.5 493.0

Crosswords and Constraint Satisfaction

modules other dbs
Modules: Other DBs

Database modules: Transform clue to DB query.

imdb: Warner of Hollywood: OLAND

wordnet: Fruitless: ARID

Syntactic: Variations of fill-in-the-blanks.

also blanks_{books, geo, movies, music, quotes}, kindof

blanks_movies: “Heavens ____!”: ABOVE

Web search: Not used in experimental system.

google: “The Way To Natural Beauty” author, 1980:

Also rogetsyns, geo, writers, compass, myth, TIEGS

altavista, yahoo, infoseek, EbModule, lsiency, etc.

Crosswords and Constraint Satisfaction

modules backstops
Modules: Backstops

Word lists: Ignore clue, return all words.

wordList: 10,000 words, perhaps: NOVELETTE

Implicit modules: Probability distributions over all strings of words (e.g., bigram).

segmenter: 1934 Hall and Nordhoff adventure novel: PITCAIRNISLAND

AlsobigWordList, wordList, DbList

Crosswords and Constraint Satisfaction

merging candidate lists
Merging Candidate Lists

Crosswords and Constraint Satisfaction

module performance
Module Performance

Crosswords and Constraint Satisfaction

slide23
CSPs

Constraint satisfaction is a core CS task. Apps:

  • planning and scheduling
  • design
  • vision
  • natural language understanding
  • temporal reasoning
  • protocol verification

Crossword puzzles the poster child.

Crosswords and Constraint Satisfaction

grid filling and csps
Grid Filling and CSPs

Crosswords and Constraint Satisfaction

csps and ir
CSPs and IR

Domain from ranked candidate list?

Tortellini topping:

TRATORIA, COUSCOUS, SEMOLINA, PARMESAN,

RIGATONI, PLATEFUL, FORDLTDS, SCOTTIES,

ASPIRINS, MACARONI, FROSTING, RYEBREAD,

STREUSEL, LASAGNAS, GRIFTERS, BAKERIES,…

MARINARA, REDMEATS, VESUVIUS, …

Standard recall/precision tradeoff.

Crosswords and Constraint Satisfaction

probabilities to the rescue
Probabilities to the Rescue?

Annotate domain with the bias.

Crosswords and Constraint Satisfaction

solution probability
Solution Probability

Proportional to the product of the probability of the individual choices.

Can pick sol’n with maximum probability.

Maximizes prob. of whole puzzle correct.

Won’t maximize number of words correct.

Crosswords and Constraint Satisfaction

posterior score
Posterior Score

Posterior probability of a candidate in a slot is sum of the solution probabilities.

Crosswords and Constraint Satisfaction

maximum expected overlap
Maximum Expected Overlap

Max words in common with random sol’n.

Q: expected overlap

qxi: prob. of word in slot i.

PP-complete.

Equivalent to stochastic satisfiability.

Crosswords and Constraint Satisfaction

fast approximation
Fast Approximation

Compute exact posterior quickly on trees.

  • Only consider slots reachable in d steps.
  • Assume independence of paths (tree).

Increase d iteratively, improve approx.

Cache intermediate results (DP). Polytime!

Crosswords and Constraint Satisfaction

connection to turbo decoding
Connection to Turbo Decoding

Turbo decoding (loopy belief propagation).

  • transmitting messages from deep space
  • true message = crossword solution
  • double encoding = across/down clues
  • corruption = answer uncertainty
  • 4-cycles

Same decoding algorithm in use!

Crosswords and Constraint Satisfaction

artificial problems
Artificial Problems

Random on 5x5 grids. Improves with d.

NYT: 52% to 90%.

Crosswords and Constraint Satisfaction

grid filling
Grid Filling

Crosswords and Constraint Satisfaction

grid filling34
Grid Filling

Crosswords and Constraint Satisfaction

grid filling35
Grid Filling

Crosswords and Constraint Satisfaction

grid filling36
Grid Filling

Crosswords and Constraint Satisfaction

grid filling37
Grid Filling

Crosswords and Constraint Satisfaction

grid filling38
Grid Filling

Crosswords and Constraint Satisfaction

grid filling39
Grid Filling

Crosswords and Constraint Satisfaction

grid filling40
Grid Filling

Crosswords and Constraint Satisfaction

grid filling41
Grid Filling

Crosswords and Constraint Satisfaction

grid filling42
Grid Filling

Crosswords and Constraint Satisfaction

grid filling43
Grid Filling

Crosswords and Constraint Satisfaction

final 88 words 97 letters
Final: 88% words, 97% letters

Crosswords and Constraint Satisfaction

p roverb results
PROVERB Results

Test collection (370 puzzles, @~15 min.)

  • 95% words, 98% letters, 46% puzzles
  • NYT: 89.5% (95.5% MTW, 85.0% TFSS)
  • Ablation: ClueDB only 88%, no ClueDB 27%

American Crossword Puzzle Tournament

  • 1998: 190/251, 80% words (vs. 100%)
    • tricks: letter pairs, words in single square
  • 1999: 147/261, 75% words
    • tricks: Home is near: ALASKA

Crosswords and Constraint Satisfaction

toefl synonyms
TOEFL Synonyms

Used in college applications.

fish

  • scale
  • angle
  • swim
  • dredge

Crosswords and Constraint Satisfaction

synonym approaches
Synonym Approaches

Latent Semantic Indexing (Landauer & Dumais 97)

  • Analyze 30k paragraphs, 300d embedding
  • 64% ~Boulder http://lsa.colorado.edu/

Pointwise Mutual Information-IR (Turney 01)

  • Counts in Altavista (350M); 74% (us: 77%)

Thesaurus

  • http://Wordsmyth.net: 98% prec.; 74% cov.
  • Combine with PMI-IR: 92%

Crosswords and Constraint Satisfaction

verbal analogies
Verbal Analogies

Used in college boards (SATs, GREs), and as an intelligence test.

cat : meow ::

  • mouse : scamper
  • bird : peck
  • dog : bark
  • horse : groom
  • lion : scratch

Crosswords and Constraint Satisfaction

wrap up
Wrap Up

Modular language-game systems.

PROVERB:

  • Human-competitive performance.
  • Components theoretically motivated.
  • Probabilistically grounded.

Crosswords and Constraint Satisfaction

what s next
What’s Next?

Better module merging: Much work has been ad hoc. Now evaluating a probabilistic combining rule.

RL: Corpus-based approach to behavior. Recognize how similarities to past experience.

Meaning comes from experience, not rules.

Crosswords and Constraint Satisfaction