aquaint dialogue experiment
Download
Skip this Video
Download Presentation
AQUAINT Dialogue Experiment

Loading in 2 Seconds...

play fullscreen
1 / 11

AQUAINT Dialogue Experiment - PowerPoint PPT Presentation


  • 211 Views
  • Uploaded on

AQUAINT Dialogue Experiment. Jean Scholtz Information Access Division National Institute of Standards and Technology [email protected] Purpose of the Experiment. To investigate “dialogue” between a system and an analyst to learn what types of dialogue analysts expect to engage in

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' AQUAINT Dialogue Experiment' - eamon


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
aquaint dialogue experiment

AQUAINT Dialogue Experiment

Jean Scholtz

Information Access Division

National Institute of Standards and Technology

[email protected]

AQUAINT PI meeting

Dec. 3-6, 2002

purpose of the experiment
Purpose of the Experiment
  • To investigate “dialogue” between a system and an analyst
    • to learn what types of dialogue analysts expect to engage in
    • to learn how analysts react to different types of system responses

AQUAINT PI meeting

Dec. 3-6, 2002

wizard of oz experiment
Wizard of Oz Experiment
  • Used a web-based text chat to eliminate any possible confounds with usability of user interfaces
  • Experiment design
    • 5 systems participated
    • 2 analysts used each system
    • 10 scenarios were used; order was randomized for each system
    • analysts were given 15 minutes to explore each scenario
  • Data collection
    • logs of dialogues
    • rating questionnaires filled out by the analyst after each scenario
    • observation notes

AQUAINT PI meeting

Dec. 3-6, 2002

results systems
Results- Systems
  • Overall the systems were rated reasonably high by the analysts

AQUAINT PI meeting

Dec. 3-6, 2002

results systems1
Results - Systems
  • Analysts also judged most scenarios as “successful”

AQUAINT PI meeting

Dec. 3-6, 2002

results analysts
Results - Analysts
  • Initial queries and dialogues were extremely varied.
    • Most initial queries were phrased as questions but they also used statements such as
      • “I need”; “please provide information on”; “looking for background information on….”
    • Analysts at times provided context in the initial query
      • Analyst: "Subject is effect of pollution on black sea fishing industry, What are sources of pollution, trends in reducing pollution, and international cooperation in reducing pollution?”
  • Analysts did not always take turns. They asked questions as they occurred to them.
  • Analysts posed multipart questions. If the system did not understand, they broke these down into separate parts.
  • Analysts posed general questions. If the system did not understand, they asked more specific questions.

AQUAINT PI meeting

Dec. 3-6, 2002

results analysts1
Results - Analysts
  • Analysts expect the system to remember context
    • Example:
    • analyst, “good info. pls describe the how question."
    • Example:
    • Wizard, "the answer is 90 billion dollars"
    • analyst ,"The same for 2000, please."
    • Example:
    • Wizard,"I have no further information for the year 1998."
    • analyst,"OK on your anwser for 1998 can you do the same for 2000 and skip 1999"

AQUAINT PI meeting

Dec. 3-6, 2002

results analysts2
Results - Analysts
  • Self-clarification – Analysts interrupt to clarify their own questions or even to cancel them.
  • Example:
    • analyst, “iwould should read i would"
  • Example:
    • analyst,"Can you tell me which of these was the first joint venture?
    • Analyst,"forget that last question"
  • Analysts don’t just say no, or yes for that matter. When asked questions by the system that can be answered as yes or no, analysts often add constraints.
  • Example:
    • Wizard,"Would you like more figures?"
    • analyst, "Keep going but would like timeline on figures if possible."
    • analyst, “also do u have any projections?"
  • Example:
    • wizard,"Thank you. Are you interested in seeing information about The International Waters Assessment Team as it is related to Black Sea pollution?"
    • analyst, “Yes particularly about fishing if possible.”

AQUAINT PI meeting

Dec. 3-6, 2002

results analysts3
Results- Analysts
  • Responding to system questions is not always straight forward
    • Example:
    • Wizard, “We still have more, would you like to see?"
    • analyst,"can you give me info on when they were formed and the year they were formed”
    • Example:
    • Wizard, "There is information about inflation rate, GDP, exchange rate, and per capita income. Which indicator would you like?"
    • analyst, “i would like to see all indicators”
    • Example:
    • analyst, “give me allinformation concerning the economic situation in indonesia since the the year 1998"
    • Wizard,"I did not understand the word "allinformation". Please try using a different word"
    • analyst, “actually two words namely all information"
    • Wizard,"I did not understand the word "namely". Please try using a different word. "
    • analyst,” i would like all available information you have concerning the economic situation in indonesia"

AQUAINT PI meeting

Dec. 3-6, 2002

results analysts4
Results- Analysts
  • Analysts have high expectations – ask about content
  • Example
    • Wizard, "The cultivated opium production for 2001 is 7606.
    • Analyst, “Is this metric tons?"
    • Wizard,"The cultivated opium production for 2001 is 7606 hectares
  • Example:
    • wizard, “However, both the da Vinci and ZEUS systems must receive governmental approval for each procedure that a surgeon plans to use it for.\'"
    • wizard, “On July 11, 2000, the U.S. Food and Drug Administration (FDA) approved the da Vinci Surgical System, making it the first robotic system allowed to be used in American operating rooms.\'"
    • analyst, “When you say governmental approval do you mean federal or something else?"
    • Wizard," i think this refers to US federal approval"
    • analyst, "for each surgery?"

AQUAINT PI meeting

Dec. 3-6, 2002

next steps
Next Steps
  • Issues to be addressed for next experiment
    • Scenarios
      • need longer, more involved scenarios
    • Database
      • need a common database
    • User Interface
      • need to move to system supplied user interfaces
    • Metrics
      • handling of different classifications of dialogues/responses

AQUAINT PI meeting

Dec. 3-6, 2002

ad