Language games and other meaningful pursuits
1 / 43

Language Games and - PowerPoint PPT Presentation

  • Updated On :

Language Games and Other Meaningful Pursuits. Michael L. Littman [email protected] AT&T Labs  Research (sort of). Motivation. Software that can understanding language. Question answering Constructing databases from text Natural-language interaction

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Language Games and' - Michelle

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Language games and other meaningful pursuits l.jpg

Language Games andOther Meaningful Pursuits

Michael L. Littman

[email protected]

AT&T LabsResearch

(sort of)

Language Games

Motivation l.jpg

Software that can understanding language.

  • Question answering

  • Constructing databases from text

  • Natural-language interaction

  • Automatic summarization/briefing

    Semantics tricky; word meanings informal.

Language Games

What is meaning l.jpg
What is Meaning?

I don’t really like bananas, but have long respected their humorous potential.

How capture this sentence?

  • Predicate-argument structure?

  • Bag of words plus associated terms?

    Evaluate by use.

    Might need multiple reps. or none at all.

    Can we make incremental progress?

Language Games

Language games l.jpg
Language Games

Like other games:

  • Evaluation process clean.

  • Challenging (and fun!) for people.

    Unlike logical games:

  • Meaning matters!

  • No closed world assumption; messy.

  • Learning necessary... moving target.

    Machine performance far from humans’.

Language Games

Word games l.jpg
Word Games

Super-human performance common:

  • Scrabble™: Maven, near-perfect (Sheppard 02)

  • Boggle™: millisecond solutions (Boyan 98)

  • Hangman (Littman 00)

    • 99.97% 9-letter words under 5 guesses

    • 1.35 misses on average

Language Games

Language game examples l.jpg
Language Game Examples

  • Crossword puzzles Acrostics

  • Cryptic crosswords Wheel of Fortune™

  • Trivial Pursuit™ Tribond™

  • College Bowl Taboo™

  • Who Wants to Be a Scattergories™ Millionaire™

  • Weakest Link™ Dictionary

  • Synonyms/Analogies

Language Games

Slide7 l.jpg

Word meaning matters:

  • Pages semantically relevant to query

    Multiple methods of determining relevance:

  • query words near each other on page

  • query words appear anywhere on page

    Successful theme: combine

  • precision high, coverage low

  • precision low, coverage high

Language Games

Combination rule l.jpg
Combination Rule

Module set: coverage ci and precision pi

Let Rjbe the modules selecting response j.

  • Pick argmaxjPiin RjpiPiin URj’(1-pi)

    If modules have independent precision:

  • Expected accuracy beats maxicipi.

Language Games

Crossword puzzles shortz 98 l.jpg
Crossword Puzzles (Shortz 98)

ThesaurusCut off _ _ _ _ _ _ _ _

Puns & WordplayMonk’s head? _ _ _ _ _

Arts & Literature“Foundation

Trilogy” author _ _ _ _ _ _

Popular CulturePal of Pooh _ _ _ _ _ _

EncyclopedicMountain known

locally as Chomolungma _ _ _ _ _ _ _

CrosswordeseKind of coal or coat _ _ _







Language Games

P roverb system design l.jpg
PROVERB: System Design

Candidate generation(Keim et al. 99)

  • Like information retrieval: clue implies target

  • Variety of approaches used simultaneously


  • Like meta search engine: create master list

    Grid filling(Shazeer et al. 99)

  • Like constraint satisfaction: fit answer to grid

  • Solved via probabilistic optimization

Language Games

Modules cluedb l.jpg
Modules: ClueDB

Collection of published clues.

Nymph pursuer: SATYR Bugs pursuer: ELMER

Nymph chaser: SATYR Place for an ace: SLEEVE

Highball ingredient: RYEHighball ingredient: ICE

exact: Highball ingredient: RYE

partial: Ace place?: SLEEVE

TransModule: Bugs chaser: ELMER

X chaser Xpursuer

Language Games

Modules special purpose l.jpg
Modules: Special Purpose

Database modules: Transform clue to DB query.

imdb: Warner of Hollywood: OLAND

wordnet: Fruitless: ARID

Syntactic: Variations of fill-in-the-blanks.

blanks_movies: “Heavens ____!”: ABOVE

Web search: Not used in experimental system.

google: “The Way To Natural Beauty” author, 1980: TIEGS

Language Games

Modules backstops l.jpg
Modules: Backstops

Word lists: Ignore clue, return all words.

wordList: 10,000 words, perhaps: NOVELETTE

Implicit modules: Probability distributions over all strings of words (e.g., bigram).

segmenter: 1934 Hall and Nordhoff adventure novel: PITCAIRNISLAND

Language Games

Grid filling l.jpg
Grid Filling

Language Games

Grid filling15 l.jpg
Grid Filling

Language Games

Grid filling16 l.jpg
Grid Filling

Language Games

Grid filling17 l.jpg
Grid Filling

Language Games

Grid filling18 l.jpg
Grid Filling

Language Games

Grid filling19 l.jpg
Grid Filling

Language Games

Grid filling20 l.jpg
Grid Filling

Language Games

Grid filling21 l.jpg
Grid Filling

Language Games

Grid filling22 l.jpg
Grid Filling

Language Games

Grid filling23 l.jpg
Grid Filling

Language Games

Grid filling24 l.jpg
Grid Filling

Language Games

P roverb results l.jpg

Test collection

  • 95% words, 98% letters, 46% puzzles

  • NYT: 89.5% (95.5% MTW, 85.0% TFSS)

    American Crossword Puzzle Tournament

  • 1998: 190/251, 80% words (vs. 100%)

    • tricks: letter pairs, words in single square

  • 1999: 147/261, 75% words

    • tricks: cloonerized spues!

spoonerized clues!

Language Games

Cryptic crosswords l.jpg
Cryptic Crosswords

Popular in UK

A clue like this about Japanese religion (6)





about( HINT,SO)

Language Games

Crossword maestro tunstall pedoe l.jpg
Crossword Maestro (Tunstall-Pedoe)

Solves this clue:

`japanese religion' is the definition. I am not sure about the `japanese' bit but `Shinto' can be an answer for `religion'.

`a clue like this about' is the subsidiary indication. `a clue‘ becomes `hint', `like this' becomes `so', `about' means one lot of letters goes inside another. `hint' placed inside `so' is `shinto'.

Language Games

Cm analysis l.jpg
CM Analysis

CM “knows”:

  • Shinto is a religion (although not Japanese)

  • Back-off: “Japanese religion” kind of religion.

    • other classes: manmade objects, bodily activities, natural acts, verbs in base form, adjectives…

  • “a clue” is hint, “like this” is so

  • “about” indicates containment (game knowl.)


Language Games

Cm results l.jpg
CM Results

Ads claim 75% on individual clues (perhaps full puzzles).

New York Times puzzles, under 5%.

Very sophisticated:

  • NL explanations

  • broad coverage

  • real time

Language Games

Question answering l.jpg
Question Answering

Blend of IR and NLP.

  • query, tag type

  • find relevant passages

  • extract entities, sort by score (combination)

Language Games

Who invented the first computer l.jpg
Who invented the first computer?

  • The stunning shortsightedness and obtusity of XEROX's top-level suits has been well anatomized in “Fumbling The Future: How XEROX Invented, Then Ignored, the First Personal Computer” by Douglas K. Smith and Robert C. Alexander (William Morrow & Co., 1988, ISBN 0-688-09511-9).

  • 1876 Alexander Graham Bell invented the telephone. 1958 First electronic computers were built.

  • Steve Jobs and Steve Wozniak were two young men who invented the first-ready made computer, the Apple.

  • A Brief History of Computers and Data Communications. … 1820 Charles Babbage invented the difference engine … 1879 First private telephone switch board was installed.

Language Games

Trivial pursuit l.jpg
Trivial Pursuit™

Race around board, answer questions.

Categories: Geography, Entertainment, History, Literature, Science, Sports

Language Games

Wigwam l.jpg

QA via AQUA (Abney et al. 00)

  • back off: word match in order helps score.

  • “When was Amelia Earhart's last flight?”

    • 1937, 1897 (birth), 1997 (reenactment)

      Move selection via MDP (Littman 00)

  • Estimate category accuracy.

  • Minimize expected turns to finish.

Language Games

Simulation results l.jpg
Simulation Results

Language Games

Toefl synonyms l.jpg
TOEFL Synonyms

Used in college applications.


  • scale

  • angle

  • swim

  • dredge

Language Games

Synonym approaches l.jpg
Synonym Approaches

Latent Semantic Indexing (Landauer & Dumais 97)

  • Analyze 30k paragraphs, 300d embedding

  • 64% ~Boulder

    Pointwise Mutual Information-IR (Turney 01)

  • Counts in Altavista (350M); 74% (us: 77%)


  • 98% prec.; 74% cov.

  • Combine with PMI-IR: 92%

Language Games

Verbal analogies l.jpg
Verbal Analogies

Used in college boards (SATs, GREs), and as an intelligence test.

cat : meow ::

  • mouse : scamper

  • bird : peck

  • dog : bark

  • horse : groom

  • lion : scratch

Language Games

Preliminary results l.jpg
Preliminary Results

Undergraduate project at Princeton

  • 17 groups

  • 229 examples (ETS, web, class)

  • 76 test problems from SATs

Language Games

Analogy results sats l.jpg
Analogy Results: SATs

Three groups sig. better than chance.

Language Games

Some successful techniques l.jpg
Some Successful Techniques

Individual groups used combinations:

  • Wordnet for synonyms, antonyms, is-a

  • web search with “linking words”

    • x (characterized by OR possesses OR possessed OR possess OR characterizes) y

  • automatically identify linking words

  • word association in definitions

  • Language Games

    Conclusion l.jpg

    Language games interesting, challenging.

    Progress on solving “meaning” problems.

    • Word order

    • Words, as a backoff

    • Meaning-preserving transformations

    • Probabilistic models

    • Combinations!

      Exciting learning and reasoning to be done

    Language Games

    Challenges l.jpg

    What year was Dartmouth College founded?

    Dr. Koop graduated from DartmouthCollege in 1937 and received his M.D. degree from Cornell Medical College in 1941.

    Although DartmouthCollege was established in 1769 to educate American Indians and…

    Language Games