Aquaint dialogue experiment
This presentation is the property of its rightful owner.
Sponsored Links
1 / 11

AQUAINT Dialogue Experiment PowerPoint PPT Presentation


  • 160 Views
  • Uploaded on
  • Presentation posted in: General

AQUAINT Dialogue Experiment. Jean Scholtz Information Access Division National Institute of Standards and Technology [email protected] Purpose of the Experiment. To investigate “dialogue” between a system and an analyst to learn what types of dialogue analysts expect to engage in

Download Presentation

AQUAINT Dialogue Experiment

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Aquaint dialogue experiment

AQUAINT Dialogue Experiment

Jean Scholtz

Information Access Division

National Institute of Standards and Technology

[email protected]

AQUAINT PI meeting

Dec. 3-6, 2002


Purpose of the experiment

Purpose of the Experiment

  • To investigate “dialogue” between a system and an analyst

    • to learn what types of dialogue analysts expect to engage in

    • to learn how analysts react to different types of system responses

AQUAINT PI meeting

Dec. 3-6, 2002


Wizard of oz experiment

Wizard of Oz Experiment

  • Used a web-based text chat to eliminate any possible confounds with usability of user interfaces

  • Experiment design

    • 5 systems participated

    • 2 analysts used each system

    • 10 scenarios were used; order was randomized for each system

    • analysts were given 15 minutes to explore each scenario

  • Data collection

    • logs of dialogues

    • rating questionnaires filled out by the analyst after each scenario

    • observation notes

AQUAINT PI meeting

Dec. 3-6, 2002


Results systems

Results- Systems

  • Overall the systems were rated reasonably high by the analysts

AQUAINT PI meeting

Dec. 3-6, 2002


Results systems1

Results - Systems

  • Analysts also judged most scenarios as “successful”

AQUAINT PI meeting

Dec. 3-6, 2002


Results analysts

Results - Analysts

  • Initial queries and dialogues were extremely varied.

    • Most initial queries were phrased as questions but they also used statements such as

      • “I need”; “please provide information on”; “looking for background information on….”

    • Analysts at times provided context in the initial query

      • Analyst: "Subject is effect of pollution on black sea fishing industry, What are sources of pollution, trends in reducing pollution, and international cooperation in reducing pollution?”

  • Analysts did not always take turns. They asked questions as they occurred to them.

  • Analysts posed multipart questions. If the system did not understand, they broke these down into separate parts.

  • Analysts posed general questions. If the system did not understand, they asked more specific questions.

AQUAINT PI meeting

Dec. 3-6, 2002


Results analysts1

Results - Analysts

  • Analysts expect the system to remember context

    • Example:

    • analyst, “good info. pls describe the how question."

    • Example:

    • Wizard, "the answer is 90 billion dollars"

    • analyst ,"The same for 2000, please."

    • Example:

    • Wizard,"I have no further information for the year 1998."

    • analyst,"OK on your anwser for 1998 can you do the same for 2000 and skip 1999"

AQUAINT PI meeting

Dec. 3-6, 2002


Results analysts2

Results - Analysts

  • Self-clarification – Analysts interrupt to clarify their own questions or even to cancel them.

  • Example:

    • analyst, “iwould should read i would"

  • Example:

    • analyst,"Can you tell me which of these was the first joint venture?

    • Analyst,"forget that last question"

  • Analysts don’t just say no, or yes for that matter. When asked questions by the system that can be answered as yes or no, analysts often add constraints.

  • Example:

    • Wizard,"Would you like more figures?"

    • analyst, "Keep going but would like timeline on figures if possible."

    • analyst, “also do u have any projections?"

  • Example:

    • wizard,"Thank you. Are you interested in seeing information about The International Waters Assessment Team as it is related to Black Sea pollution?"

    • analyst, “Yes particularly about fishing if possible.”

AQUAINT PI meeting

Dec. 3-6, 2002


Results analysts3

Results- Analysts

  • Responding to system questions is not always straight forward

    • Example:

    • Wizard, “We still have more, would you like to see?"

    • analyst,"can you give me info on when they were formed and the year they were formed”

    • Example:

    • Wizard, "There is information about inflation rate, GDP, exchange rate, and per capita income. Which indicator would you like?"

    • analyst, “i would like to see all indicators”

    • Example:

    • analyst, “give me allinformation concerning the economic situation in indonesia since the the year 1998"

    • Wizard,"I did not understand the word "allinformation". Please try using a different word"

    • analyst, “actually two words namely all information"

    • Wizard,"I did not understand the word "namely". Please try using a different word. "

    • analyst,” i would like all available information you have concerning the economic situation in indonesia"

AQUAINT PI meeting

Dec. 3-6, 2002


Results analysts4

Results- Analysts

  • Analysts have high expectations – ask about content

  • Example

    • Wizard, "The cultivated opium production for 2001 is 7606.

    • Analyst, “Is this metric tons?"

    • Wizard,"The cultivated opium production for 2001 is 7606 hectares

  • Example:

    • wizard, “However, both the da Vinci and ZEUS systems must receive governmental approval for each procedure that a surgeon plans to use it for.'"

    • wizard, “On July 11, 2000, the U.S. Food and Drug Administration (FDA) approved the da Vinci Surgical System, making it the first robotic system allowed to be used in American operating rooms.'"

    • analyst, “When you say governmental approval do you mean federal or something else?"

    • Wizard," i think this refers to US federal approval"

    • analyst, "for each surgery?"

AQUAINT PI meeting

Dec. 3-6, 2002


Next steps

Next Steps

  • Issues to be addressed for next experiment

    • Scenarios

      • need longer, more involved scenarios

    • Database

      • need a common database

    • User Interface

      • need to move to system supplied user interfaces

    • Metrics

      • handling of different classifications of dialogues/responses

AQUAINT PI meeting

Dec. 3-6, 2002


  • Login