Lrec 2008 morocco
1 / 45

Information Extraction Tools and Methods for Understanding Dialogue in a Companion R. Catizone, A. Dingli, H. Pinto and - PowerPoint PPT Presentation

  • Uploaded on

Information Extraction Tools and Methods for Understanding Dialogue in a Companion R. Catizone, A. Dingli, H. Pinto and Y.Wilks University of Sheffield. LREC 2008 Morocco. Senior Companion. Developing a range of companions to assist the elderly. Photos. News. Yellow pages.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Information Extraction Tools and Methods for Understanding Dialogue in a Companion R. Catizone, A. Dingli, H. Pinto and' - bingham

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Lrec 2008 morocco

Information Extraction Tools and Methods for Understanding Dialogue in a CompanionR. Catizone, A. Dingli, H. Pinto and Y.WilksUniversity of Sheffield

LREC 2008


Senior companion
Senior Companion Dialogue in a Companion

Developing a range of companions to assist the elderly



Yellow pages

The Senior Companion is about… Dialogue in a Companion

Multimodal dialogue for building a picture of… Dialogue in a Companion

…through his photographs

the user

Behind the scenes using the Dialogue in a Companion dialogue to

…tag photos


family holiday




Photos Dialogue in a Companion

  • A Photograph application for reminiscing about personal photos.

  • The user begins with a set of photos which will be input into the system in advance.

  • The system engages in conversation with the user about the photos.

  • The system will create a life narrative by extracting key facts from the user about his/her photos and the people, places and events that are represented.

  • The information about the photos can later be retrieved and displayed in a future user session.

  • Semantic features extracted during the dialogue will be automatically associated with the photos ( may include audio recordings)

Life narrative through photos
Life Narrative through Photos Dialogue in a Companion

  • Life Narrative - Dynamically builds segments of a person’s life that correspond to relationships with:

    • People and Places with respect to

      • Life events (Birthdays, marriages, etc)

      • Journeys and Travels

      • Special memories

News Dialogue in a Companion

  • News reading

  • Live from BBC RSS feeds

  • Retrieved through categories

    • Sports, business, international, etc.

Yellow pages future
Yellow pages (future) Dialogue in a Companion

  • To incorporate helping with everyday interactions such as yellow pages assistant for finding information.

  • Example for finding a plumber

    User: “Hello Morgan, my toilet is broken, I really need to find a plumber. Can you help me.”

    System: “Sure I can,the broken toilet is at your home on Hayfield Rd right?

    User: “Yes”

    System: ”Let me have a look and see what I can find.”

    User: “Ok”

    System: “Yes, I’ve found a local plumber in your area. Would you like me to get him on the phone using Skype.”

    User: “Yes please”

    System: “Ok here you go”

The System Dialogue in a Companion

Interface Layer Dialogue in a Companion


Napier Interface


Input Queue









Senior Companion Architecture

Output Manager

Modules Dialogue in a Companion

  • Interface + Avatars

  • Face identification (Open CV)

  • Fusion

  • ASR

  • Natural Language Understanding

  • Dialogue Manager

  • Knowledge Representation/Reasoning

  • TTS

Interface Dialogue in a Companion

  • Multimodal - speech and touch (using a touchscreen tablet)

  • Includes photos, avatar and text box

  • Currently designed to discuss user’s photos and read news.

  • Displays multiple photos at a time

    • Photo management

    • Photo selection

Avatars Dialogue in a Companion

  • Experimenting with different avatars

    • Morgan (AAA)

    • Crazy Talk

      • Woman

      • Ken Dodd

      • Lion

      • A computer

Interface 2
Interface (2) Dialogue in a Companion

Interface 21
Interface (2) Dialogue in a Companion

Please read me the news

OpenCV Dialogue in a Companion

  • Face identification software

    • Finds the coordinates of all of the faces in a photo.

    • Cannot recognize the same face in more than one photo, but investigating face recognition software: Polar Rose

Photo applications
Photo Applications Dialogue in a Companion

  • Also use digital photo metadata such as

    • Date and time

    • GPS coordinates

  • Photos that have annotations (Facebook)

More information to start makes for more interesting dialogue

Samuel Dialogue in a CompanionCarter


Ed Bloch

Olivia Ford

Fusion Dialogue in a Companion

  • Merges speech and pointing input

“This is my daughter Octavia”

ASR Dialogue in a Companion

  • Dragon Naturally Speaking

    • requires 10 minute training session

    • accuracy is high - up to 99%

    • application integrated with the SC

    • full integration of the code using the Nuance SDK planned for next version

Natural language understanding
Natural Language Understanding Dialogue in a Companion

  • Sentence splitting

  • POS tagger

  • Parser

  • Annie Named Entity Recognizer (GATE, USFD)

    • person names (10,000)

    • locations (Web trawl)

    • 60 relation names

      • mother, brother, sister, friend, etc.

  • Dialogue Act Tagger (ALB).

  • Populates Ontology instances.

  • To use an IE approach for semantic interpretation

Nlu the task so far
NLU: The task so far … Dialogue in a Companion

  • Identify people and their relationships to the user and each other

  • Identify locations

The process
The process Dialogue in a Companion



Dialogue Act Tagger


Syntactic Parser

Ontology Processing

Syntactic and Semantic representation

Noun Chunks Dialogue in a Companion

Verb Phrases



I went to meet Lynne, my old flatmate, together with my sister Francesca.


Relationships Dialogue in a Companion



I went to meet Lynne, my old flatmate, together with my sister Francesca.

Syntactic parser
Syntactic Parser Dialogue in a Companion

I went to meet Lynne, my old flatmate, together with my friend Francesca.

Who is the sister and who is the flatmate?


( (together))((with)( (my)(sister)(Francesca))))))(.)))

(((I)(went((to meet Lynne , my old flatmate,)

( together)(with( my sister Francesca))))).))

(((I)(went((to meet Lynne , my old flatmate,)

( together)(with( my sister Francesca))))).))

Syntactic bindings will help us identify that!

Ontology Dialogue in a Companion

Reasoning Dialogue in a Companion

  • Allows us to infer new knowledge

    • Sister(My) = Francesca

    • Flatmate(My) = Lynne

    • If we have in the ontology:

      • Mother(My) = Mary

    • We can infer through the properties of the Sister relationship that

      • Mother(Francesca) = Mary

      • Daughter(Mary) = Francesca

Knowledge base
Knowledge Base Dialogue in a Companion

  • Relationships Ontology

    • Almost 40 person classes (mother, father, etc)

    • Almost 40 relationships (has friend, etc)

      Will be adding more …

  • Locations Ontology

    • Most continents, countries, regions, cities

    • Plus

      • several places of interests per region

      • top 10 things to do in cities

        Will be adding more …

DAMSL Dialogue Act set Dialogue in a Companion

* ASSERT ("This is my sister". Yes and no are also considered asserts, where they are responses to yes/no questions, thus abbreviated assertions.) * OFFER ("Shall we look at another picture?") * COMMIT ("Okay I'll do that") * EXPRESSION (All social expressions such as "you're welcome". Also things like "wow!" and "great!") * INFORMATION REQUEST (open question) * CONFIRMATION REQUEST (yes/no question) * REPEAT REQUEST ("Pardon?") * ACTION DIRECTIVE ("Show me another one." All imperatives.) * OPEN OPTION ("We could look at another picture." Stating an option in a way that doesn't demand an answer.) * OPENING ("Hi") * CLOSING ("Goodbye") * ANSWER (An answer is invariably also an assert. Yes/no answers are asserts.) * BACKCHANNEL ("Uhuh") * REPEAT REPHRASE (Expressing understanding by paraphrasing) * COMPLETION (Completing the utterance of the other speaker) * NON-UNDERSTANDING ("I don't understand") * CORRECTION (An assertion that corrects a previous assertion) * ACCEPT (Accepting a proposal) * REJECT (Rejecting a proposal)

Semantic specification
Semantic specification Dialogue in a Companion

  • Each dialogue begins with a user object and a picture object

    • User/Person objects have

      • name, people relations,age

    • Picture object

      • location, occasions, people, dates)

User object




Person: Zoe




Occasion: Hilary Duff concert

Picture object

Location: Birmingham

Person: Roisin, snooks

Information extraction in the sc
Information Extraction in the SC Dialogue in a Companion

  • Why do we want to use IE for Semantic representation?

    User utterances are unstructured

  • Using GATE IE tools to create templates

    • Relationship between the Named Entities and the significant Events.

      • Categorize events into meaningful classes

Information extraction 2
Information Extraction (2) Dialogue in a Companion

  • Example sentence

    ‘That is my daughter Zoe on the right’

  • Simple Example template :


    • Person1 Related-to Person2

      • Related-to : is-relation

        • is-relation: is-daughter, is-mother etc.

          is-daughter: lexical string: ‘is my daughter’

  • Filled IE Template

    Relation : [Person1=‘Zoe’], [Person2=‘Roberta’], [Related-to=is-daughter]

  • Sc ontology
    SC Ontology Dialogue in a Companion

    SC Ontology for Inference

    “Here are my daughters, Zoe and Octavia in New York City”

    Infer using the family relations Ontology that:

    -Zoe and Octavia are sisters

    -Roberta is the mother of Octavia and Zoe

    Dialogue manager 1
    Dialogue Manager (1) Dialogue in a Companion

    • Manages discussion of

      • User’s photos with respect to

        • Location

        • Time it was taken

        • Occasion

        • People in the Photo (exploits positional information of people)

          • Name

          • Age

          • Relation to the user

      • Photo Management

        • “Show me all the photos of my mother”

        • Photo selection

      • News

        • Reading and stopping

        • Choice of politics, sport or business

    Dialogue manager 2
    Dialogue Manager (2) Dialogue in a Companion

    • Accepts pre-annotated photos to allow the system to engage in more interesting conversation more quickly.

    • Handles basic photo management tasks:

      “Show me all the photos of my mother”

      “Please move on to the next photo”

      • Responds when same person is mentioned in more than one photo (using the user’s name)

    • Remembers user information from multiple sessions

    • Generation : template based

    Dialogue manager 3
    Dialogue manager (3) Dialogue in a Companion

    • Dialogue Manager adapted from COMIC DM

      • General purpose control structure that does the dialogue planning

      • Stack based system

        • Conforms to common behaviour of conversation: Discuss topic 1, move to topic 2, go back to topic 1.

      • Augmented transition networks, called Dialogue Action Forms (DAFs) for handling domain sub-tasks.

      • References the Dialogue History, Knowledge base and the User Model

    Dialogue action forms
    Dialogue Action Forms Dialogue in a Companion

    • GUI editor for creating DAFs

    • Composed of nodes and arcs containing tests and actions

    • DAFs pre-stacked, but can be overidden by matching indexing terms (semantic classes, significant words)

      • Essential for mixed-initiative conversation

    Sc dialogue manager stack
    SC Dialogue Manager Stack Dialogue in a Companion

    Run Greeting DAF

    Pop greeting DAF

    Push photo DAF

    Run people DAF

    Pop people DAF

    System start

    Run photo DAF

    People DAF



















    Dafs 1
    DAFs (1) Dialogue in a Companion

    Machine learning in the senior companion
    Machine Learning in the Senior Companion Dialogue in a Companion

    • Using first Senior Companion prototype to generate more data, augmenting WoZ data gathered by NAP and AAA and hand annotated.

    • Plan to “tile” (as in Hearst) the data to seek segmentations corresponding to topics and dialogue moves.

    • Plan to generalise across a set corresponding to “same” topic or move to generate a draft Dialogue Action.

    Www companions project org Dialogue in a Companion

    Thank you
    Thank You Dialogue in a Companion

    Evaluation of the senior companion
    Evaluation of the Senior Companion Dialogue in a Companion

    • Accuracy

      • Is the information that the system discusses accurate

        • Important when the user returns for repeat sessions (system needs to remember and recall collected information at the appropriate time).

      • Does the system discuss things in an efficient way? (not ask for clarification when the information is already known)

    • User satisfaction

      • Self-assessment: May include some testing of the user’s level of contentment with the system while running.