aquaint dialogue experiment n.
Skip this Video
Loading SlideShow in 5 Seconds..
AQUAINT Dialogue Experiment PowerPoint Presentation
Download Presentation
AQUAINT Dialogue Experiment

Loading in 2 Seconds...

  share
play fullscreen
1 / 11
Download Presentation

AQUAINT Dialogue Experiment - PowerPoint PPT Presentation

eamon
224 Views
Download Presentation

AQUAINT Dialogue Experiment

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. AQUAINT Dialogue Experiment Jean Scholtz Information Access Division National Institute of Standards and Technology jean.scholtz@nist.gov AQUAINT PI meeting Dec. 3-6, 2002

  2. Purpose of the Experiment • To investigate “dialogue” between a system and an analyst • to learn what types of dialogue analysts expect to engage in • to learn how analysts react to different types of system responses AQUAINT PI meeting Dec. 3-6, 2002

  3. Wizard of Oz Experiment • Used a web-based text chat to eliminate any possible confounds with usability of user interfaces • Experiment design • 5 systems participated • 2 analysts used each system • 10 scenarios were used; order was randomized for each system • analysts were given 15 minutes to explore each scenario • Data collection • logs of dialogues • rating questionnaires filled out by the analyst after each scenario • observation notes AQUAINT PI meeting Dec. 3-6, 2002

  4. Results- Systems • Overall the systems were rated reasonably high by the analysts AQUAINT PI meeting Dec. 3-6, 2002

  5. Results - Systems • Analysts also judged most scenarios as “successful” AQUAINT PI meeting Dec. 3-6, 2002

  6. Results - Analysts • Initial queries and dialogues were extremely varied. • Most initial queries were phrased as questions but they also used statements such as • “I need”; “please provide information on”; “looking for background information on….” • Analysts at times provided context in the initial query • Analyst: "Subject is effect of pollution on black sea fishing industry, What are sources of pollution, trends in reducing pollution, and international cooperation in reducing pollution?” • Analysts did not always take turns. They asked questions as they occurred to them. • Analysts posed multipart questions. If the system did not understand, they broke these down into separate parts. • Analysts posed general questions. If the system did not understand, they asked more specific questions. AQUAINT PI meeting Dec. 3-6, 2002

  7. Results - Analysts • Analysts expect the system to remember context • Example: • analyst, “good info. pls describe the how question." • Example: • Wizard, "the answer is 90 billion dollars" • analyst ,"The same for 2000, please." • Example: • Wizard,"I have no further information for the year 1998." • analyst,"OK on your anwser for 1998 can you do the same for 2000 and skip 1999" AQUAINT PI meeting Dec. 3-6, 2002

  8. Results - Analysts • Self-clarification – Analysts interrupt to clarify their own questions or even to cancel them. • Example: • analyst, “iwould should read i would" • Example: • analyst,"Can you tell me which of these was the first joint venture? • Analyst,"forget that last question" • Analysts don’t just say no, or yes for that matter. When asked questions by the system that can be answered as yes or no, analysts often add constraints. • Example: • Wizard,"Would you like more figures?" • analyst, "Keep going but would like timeline on figures if possible." • analyst, “also do u have any projections?" • Example: • wizard,"Thank you. Are you interested in seeing information about The International Waters Assessment Team as it is related to Black Sea pollution?" • analyst, “Yes particularly about fishing if possible.” AQUAINT PI meeting Dec. 3-6, 2002

  9. Results- Analysts • Responding to system questions is not always straight forward • Example: • Wizard, “We still have more, would you like to see?" • analyst,"can you give me info on when they were formed and the year they were formed” • Example: • Wizard, "There is information about inflation rate, GDP, exchange rate, and per capita income. Which indicator would you like?" • analyst, “i would like to see all indicators” • Example: • analyst, “give me allinformation concerning the economic situation in indonesia since the the year 1998" • Wizard,"I did not understand the word "allinformation". Please try using a different word" • analyst, “actually two words namely all information" • Wizard,"I did not understand the word "namely". Please try using a different word. " • analyst,” i would like all available information you have concerning the economic situation in indonesia" AQUAINT PI meeting Dec. 3-6, 2002

  10. Results- Analysts • Analysts have high expectations – ask about content • Example • Wizard, "The cultivated opium production for 2001 is 7606. • Analyst, “Is this metric tons?" • Wizard,"The cultivated opium production for 2001 is 7606 hectares • Example: • wizard, “However, both the da Vinci and ZEUS systems must receive governmental approval for each procedure that a surgeon plans to use it for.'" • wizard, “On July 11, 2000, the U.S. Food and Drug Administration (FDA) approved the da Vinci Surgical System, making it the first robotic system allowed to be used in American operating rooms.'" • analyst, “When you say governmental approval do you mean federal or something else?" • Wizard," i think this refers to US federal approval" • analyst, "for each surgery?" AQUAINT PI meeting Dec. 3-6, 2002

  11. Next Steps • Issues to be addressed for next experiment • Scenarios • need longer, more involved scenarios • Database • need a common database • User Interface • need to move to system supplied user interfaces • Metrics • handling of different classifications of dialogues/responses AQUAINT PI meeting Dec. 3-6, 2002