csetalk a spoken dialogue system for cse course information n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
CSETalk: A Spoken Dialogue System for CSE Course Information PowerPoint Presentation
Download Presentation
CSETalk: A Spoken Dialogue System for CSE Course Information

Loading in 2 Seconds...

play fullscreen
1 / 42

CSETalk: A Spoken Dialogue System for CSE Course Information - PowerPoint PPT Presentation


  • 93 Views
  • Uploaded on

CSETalk: A Spoken Dialogue System for CSE Course Information. Preethi Jyothi, Rohit Prabhavalkar, Thomas Lynch, Deepak Bal, Prateeti Mohapatra. Outline. Introduction Automatic Speech Recognition Language Understanding Galaxy Overview Backend Ravenclaw Architecture Speech Synthesis

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'CSETalk: A Spoken Dialogue System for CSE Course Information' - yetta-skinner


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
csetalk a spoken dialogue system for cse course information

CSETalk: A Spoken Dialogue System for CSE Course Information

Preethi Jyothi, Rohit Prabhavalkar, Thomas Lynch, Deepak Bal,

Prateeti Mohapatra

outline
Outline
  • Introduction
  • Automatic Speech Recognition
  • Language Understanding
  • Galaxy Overview
  • Backend
  • Ravenclaw Architecture
  • Speech Synthesis
  • Evaluation Metrics
  • Challenges Faced
  • Conclusions and Future Work
introduction
Introduction
  • Dialog systems seek to provide a natural conversational interaction between the user and the computer system
  • Spoken input from the human user -> meaning of the utterance -> results of the operation to the user
  • Two-way flow of information
    • User-to-system
    • System-to-user
introduction1
Introduction
  • Types of Dialogue Systems
    • System-initiative
    • User-initiative
    • Mixed-initiative
motivation
Motivation
  • All the components of a speech and language processing systems are integrated
  • Building a SDS is a challenging task due to the interaction of the various components that are involved
  • Goal: To implement an end to end spoken dialog system for the Course Information system using the RavenClaw dialog management framework
other dialogue systems
Other Dialogue Systems
  • Saplen system 1997 [R. López-Cózaret et al. Eurospeech1997]: Food ordering system
  • Let’s Go! Bus Information System [Raux et. al. Eurospeech 2000]
  • ITSpoke: Intelligent Tutoring SDS [Litman and Silliman HLT-NAACL 2004]
overall architecture

Recognition

SPHINX

Synthesis

THETA

Overall Architecture

Lang. Understand.

PHOENIX/HELIOS

Dialog Manag.

RAVENCLAW

Back-end

(various)

Lang. Generation

ROSETTA

csetalk course information system
CSETalk: Course Information System
  • Information-based
  • Mixed type of dialogue system
  • The user can seek information about course numbers, credits, instructors, call numbers, etc.
  • Typical query
    • What course is Prof. X offering?
    • What is the call number for CSE-333?
asr sphinx ii
ASR – Sphinx II
  • Using the Sphinx II decoding engine
  • Uses semi-continuous hidden markov models
  • Used an off the shelf acoustic model (male and female voice )
  • Pronunciation model and language model were built using Sphinx Knowledge Base Tool
  • Used an n-gram language model, built from a corpus of sentences generated randomly from system grammar
helios confidence annotation
Helios – Confidence Annotation
  • Generates a confidence score for the utterance based on information from ASR, Parser and Dialog manager
  • ASR
    • Number of words, 'unconfident' words
  • Parser
    • Uncovered words, transitions between parsed fragments, unparsed fragments, etc.
  • Dialog manager
    • State of dialog, concepts expected at current state, number of turns at current state, etc.
helios confidence annotation1
Helios – Confidence Annotation
  • Allows detection and recovering from misunderstandings
  • Unfortunately, due to lack of time, we were not able to explore it fully
phoenix parser
Phoenix - Parser
  • The parser parses the word sequence into a set of semantic frames
  • A frame is a set of named 'slots'
  • Frame: [CSETalk]

Nets: [QueryCallNum]

[QueryInstructor]

...

  • Nets are compiled into RTNs
slide13

Phoenix - Parser

RTN for [QueryCallNum]

galaxy communicator
Galaxy Communicator

http://communicator.sourceforge.net/sites/MITRE/distributions/GalaxyCommunicator/docs/manual/index.html

what is galaxy communicator
What is Galaxy Communicator
  • Per the Galaxy Website:
    • Distributed
    • Message based
    • Hub-and-spoke Infrastructure
    • Optimized for constructing spoken dialogue systems
  • Origins:
    • Based on MIT Galaxy System
    • Currently Mitre Corporation /Darpa Communicator (Now Concluded).
    • Available at Sourceforge

http://communicator.sourceforge.net/sites/MITRE/distributions/GalaxyCommunicator/docs/manual/index.html

galaxy overview
Galaxy Overview
  • Modular with Galaxy as the communications controller.
  • Everything is sent using sockets and frames.
  • Servers/modules can be located on the network or internet.
  • Galaxy transforms the information from text strings to sockets and back again.
  • Easy to interface with C++ or any other language with a sockets library.
  • Handles storage and forwarding for each independent module.
  • Monitors the status of the other modules.
  • Logs information pertaining to the other servers specified. (Very flexible.)
  • No need to negotiate communications parameters with the other modules.
  • Simple to add different package to handle some aspect of the system.
    • Write an interface from the new system to Galaxy.
    • For MySQL, changed the original backend server call to load and call MySQL.
    • The same strategy would work for other modules.
  • Downside: Galaxy is a single point of failure. If Galaxy goes down, everything is down and no one is notified.
the details
The Details
  • What its not:
    • Not an end-to-end dialogue system.
    • No run-time semantic standards
    • No Configuration-time semantic standards
    • But compatible with many standards like W3C Voice Browsers group's proposed specifications.
  • Knowledge Requirements:
    • You need to know C programming.
    • Some background in distributed processing.
      • RPC, CORBA, Java RM
    • Reasonable command of English - to read the documentation
    • Platforms: Sparc Solaris, Intel Linux and Win32
    • Compiler: GNU gcc and make.
the exchange
The Exchange

{

query course_query

courseId seven thirty three

credits

callNum

instructor fosler-lussier

)

Select *

From courses

Where

courseId =“730”

and

instructor like “fosler-lussier”

slide20

The Exchange

{

query refine_results

results : 1

{

{

courseId "730"

credits "3"

callNum "04581-7"

callNumM "04581-7"

dow "T R"

starttime "1130"

endtime "1248"

room "DL 0305"

instructor "Eric Fosler-Lussier"

title "Survey of Artificial Intelligence II: Advanced Topics U G 3"

description "A survey of advanced concepts, techniques, and applications of artificial intelligence, including knowledge-based systems, learning, natural language understanding, and vision."

)

}

}

aStr = Gal_GetObject(f, ":inframe");

inframe = Gal_StringValue(aStr);

// Our Stuff next line

GetCourseInfoC( inframe, &outframe );

aStr = Gal_StringObject(outframe);

Gal_SetProp(f, ":outframe", aStr);

backend server
Backend Server

MySQL is the database. Possibly a different choice will be made next time. Maybe something simple like SQLite.

Original Roomline calls a perl script’s for data retrieval.

Many of the components also use perl scripts for various functions.

galaxy summary
Galaxy Summary
  • Makes Life Easy
  • It Glues Everything Together (Seamlessly?)
  • Do Not Start From Scratch! You’ll be sorry.
  • Modify an Existing System.
  • Great Piece of Software to Allow Experimenting with various components.
  • However, too many single points of failure!
ravenclaw architecture
RavenClaw Architecture
  • Domain-specific dialog control logic specified
  • Most of the effort goes into creating this specification

Dialog Task Specification (Domain Specific)

Dialog Engine (Domain Independent)

  • Runs the Dialog Task Specification to specify the dialog control at runtime
  • Provides a large number of domain-independent conversational strategies
dialog task specification
Dialog Task Specification
  • Hierarchical plan for the dialog
  • Tree of dialog agents
    • Non-terminals - Dialog Agencies
    • Terminals - Fundamental Dialog Agents
      • Inform
      • Request
      • Expect and Execute
dialog agents
Dialog Agents
  • Concepts – associated with agents
    • Concepts have pre-defined types
    • Set of value/confidence pairs
  • Structure of an agent
    • Execute routine – dependent on the agent type
    • Preconditions
    • Success/Failure criteria
    • Trigger conditions/ Trigger commands
ravenclaw dialog engine
RavenClaw Dialog Engine
  • Functions using two data structures
    • Dialog Stack
    • Expectation Agenda
  • Functions in two phases
    • Execution Phase – Dialog agents executed from Dialog Stack
    • Input Phase – Uses Expectation Agenda to map user inputs to concept values
dialog task specification1

Dialog Stack

Expectation Agenda

Dialog Task Specification

Dialog Task Specification [captures domain-specific dialog control logic]

Dialog Engine [domain-independent reusable component]

Execution Phase

dialog task specification2

Dialog Stack

Expectation Agenda

Dialog Task Specification

Dialog Task Specification [captures domain-specific dialog control logic]

Dialog Engine [domain-independent reusable component]

Execution Phase

CSETalk

dialog task specification3

Dialog Stack

Expectation Agenda

Dialog Task Specification

Dialog Task Specification [captures domain-specific dialog control logic]

Dialog Engine [domain-independent reusable component]

Execution Phase

S: Welcome to CSE Talk ...

Welcome

CSETalk

dialog task specification4

Dialog Stack

Expectation Agenda

Dialog Task Specification

Dialog Task Specification [captures domain-specific dialog control logic]

Dialog Engine [domain-independent reusable component]

Execution Phase

S: Welcome to CSE Talk ...

CSETalk

dialog task specification5

Dialog Stack

Expectation Agenda

Dialog Task Specification

Dialog Task Specification [captures domain-specific dialog control logic]

Dialog Engine [domain-independent reusable component]

Execution Phase

S: Welcome to CSE Talk ...

Task

CSETalk

dialog task specification6

Dialog Stack

Expectation Agenda

Dialog Task Specification

Dialog Task Specification [captures domain-specific dialog control logic]

Dialog Engine [domain-independent reusable component]

Execution Phase

S: Welcome to CSE Talk ...

Course_Id: [courseNumber]

S: How can I help you?

How May I Help You

Task

CSETalk

dialog task specification7

Dialog Stack

Expectation Agenda

Dialog Task Specification

Dialog Task Specification [captures domain-specific dialog control logic]

Dialog Engine [domain-independent reusable component]

Execution Phase

S: Welcome to CSE Talk ...

Course_Id: [courseNumber]

Course_Id: [courseNumber]

Course_Inst: [professor]

S: How can I help you?

How May I Help You

Task

CSETalk

dialog task specification8

Dialog Stack

Expectation Agenda

Dialog Task Specification

Dialog Task Specification [captures domain-specific dialog control logic]

Dialog Engine [domain-independent reusable component]

Input Phase

S: Welcome to CSE Talk ...

Course_Id: [courseNumber]

Course_Id: [courseNumber]

Course_Inst: [professor]

S: How can I help you?

How May I Help You

Task

U: Who’s teaching seven thirty?

[courseNumber](seven thirty)

CSETalk

language generation rosetta
Language Generation - Rosetta
  • Template-Based
  • Attribute-Value pairs  Text for synthesis
  • Input from Dialog Manager, Output to TTS
    • Input Frame

{

act inform

object welcome

}

    • Output

Welcome to CSETalk Automated Course Information….

speech synthesis
Speech Synthesis

Within Galaxy, Kalliope manages the speech synthesis

Is the link between Festival and Galaxy Hub

Submits synthesis requests to Festival, holds a synthesis queue, etc

Formatted to work with multiple synthesis systems (Festival, Swift, Theta)

speech synthesis1
Speech Synthesis
  • Actual speech synthesized by Festival
    • Open Source speech synthesis software
    • Developed at University of Edinburgh and worked on extensively at CMU
    • Currently includes many types of voices
      • Diphone (what we use)
      • HMM based (referred to as HTS)
      • Unit Selection
speech synthesis2
Speech Synthesis
  • CMU’s Festvox project hopes to make the building of new voices more systematic and better documented
    • Extensive instructions and tools on the web to help with the creation of new voices
    • Limited domain could be useful for our project
      • Uses unit selection methods
performance evaluation
Performance Evaluation
  • Many facets to measuring performance
    • Efficiency
      • Processing time
      • How much time is wasted with corrections, etc.
    • Quality
      • How many times did the system misinterpret
      • How many times did the user have to correct the system
    • User Satisfaction
      • Does the user feel happy with their interaction
      • Have user fill out survey
    • Task Success
      • Did the user leave with the information that they came for
performance evaluation1
Performance Evaluation
  • One framework that tries to incorporate all of these facets is PARADISE (PARAdigm for DIalogue System Evaluation)
    • PARADISE proposes to compute a performance measure as a function of both task success and dialogue costs
    • If design changes were made, PARADISE could be used to evaluate the effectiveness of the changes.
challenges faced
Challenges Faced
  • NO DOCUMENTATION!! 
  • Hard to localize faults since ASR/DM/NLG are tightly coupled
  • Festival runs best on UNIX whereas the rest of the system runs on Windows
  • Limited Time
conclusions and future work
Conclusions and Future Work
  • Using more classes in the language model
  • Using other corpuses to obtain the acoustic model and the language model information
  • Creating or using better TTS voice
  • Actually evaluating performance as discussed