AQUAINT R&D Program: Phase I 18-Month Workshop

1. AQUAINT R&D Program:Phase I 18-Month Workshop WELCOME

2. Advanced Question AnsweringScenario Exercise Dr. John D. Prange AQUAINT Program Manager jprange@nsa.gov 301-688-7092 http://www.ic-arda.org

3. What Functions Does It Include? Information Retrieval Analytic Knowledge Assessment and Interpretation Content Data Mark-up Presentation and Visualization Reporting and Dissemination Data Filtering & Selection IC Analysts Synthesis and Fusion Content Data Transformation Information Understanding Information Discovery Info-X is Focused on Informational Content & Its Meaning And on Enhancing the IC Analyst’s Interaction with these Info-X Functions Information Exploitation (Info-X)

4. I have plenty of Questions; It’s Answers that I need!! IC ANALYST “Pulling” Information • Accept complex “Questions” in a form natural to the analyst • Translate “Question” into multiple queries appropriate to the various data sets to be searched • Find relevant information in distributed, multimedia, multilingual, multi-agency data sets • Analyze, fuse and summarize information into a coherent “Answer” • Provide “Answer” to analyst in the form they want “Providing answers to the questions analysts ask”

5. AQUAINTAdvanced QUestion Answering for INTelligence • What it is and What it is not . . . • Question & Answering Aimed at the “Information Professional” --- Not just the Casual User

6. AQUAINTAdvanced QUestion Answering for INTelligence • What it is and What it is not . . . • Question & Answering Aimed at the “Information Professional” --- Not just the Casual User • Full Range of Questions --- Not just Factoid Questions

7. AQUAINTAdvanced QUestion Answering for INTelligence • What it is and What it is not . . . • Question & Answering Aimed at the “Information Professional” --- Not just the Casual User • Full Range of Questions --- Not just Factoid Questions • Rich, Contextually-based Question Scenarios --- Not just Isolated Questions

8. In a foreign news broadcast a team of analysts observe a previously unknown individual conferring with the Foreign Minister. They suspect that he/she is really a new senior advisor. What influence does he/she have on FM? Does this signal that other policy changes are coming? What are his/her views? What do we know about him/her? Who is this advisor? And still more questions ??? Overarching Context / Operational Requirement Intelligence Analysts AQUAINTAdvanced QUestion Answering for INTelligence

9. Judgement Questions? Predictive Questions? Interpretive Questions? Overarching Context / Operational Requirement Interpreting Complex QA Scenario within a Larger Context Why Questions? Other Questions? Factoid Questions? Voice Text System Specific Queries; Fully Tailored to Series of Questions Multi-Media Information Analysts Structured Other Extend Traditional Information Retrieval Ranked Lists of “Relevant” Data Objects Deeper Automated Understanding Extract & Analyze Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Multiple Heterogeneous Data Sources Advanced QA Provide Answers in a Form Analysts Want Interpret Results & Formulate the Answers Answers AQUAINTAdvanced QUestion Answering for INTelligence

10. AQUAINTAdvanced QUestion Answering for INTelligence • What it is and What it is not . . . • Question & Answering Aimed at the “Information Professional” --- Not just the Casual User • Full Range of Questions --- Not just Factoid Questions • Rich, Contextually-based Question Scenarios --- Not just Isolated Questions • Open Domain, Multiple Media, Multiple Languages, Multiple Genre, Structured and Unstructured Data --- Not just a Focused Data Environment

11. AQUAINTAdvanced QUestion Answering for INTelligence • What it is and What it is not . . . • Question & Answering Aimed at the “Information Professional” --- Not just the Casual User • Full Range of Questions --- Not just Factoid Questions • Rich, Contextually-based Question Scenarios --- Not just Isolated Questions • Open Domain, Multiple Media, Multiple Languages, Multiple Genre, Structured and Unstructured Data --- Not just a Focused Data Environment • Tackling fundamental, very difficult Information Exploitation Challenges

12. Today Level I Level II Future Level III Mulit-Valued Factual Questions Questions Cross Media Cross Document Simple Judgement Full Context-Based Question Scenario Full Context-Based Question Scenario Single Factual Isolated Questions Increasing Volumes (Petabyte & up) Data Chasm Synthesis Across “Documents”/Media Contradictory Data MANY Heterogeneous Data Sources; All Types, Sizes, Locations Multiple Perspectives Missing Data Reliability of Data Answers Fully Intersected; Automatically Generated; Variable Structure/ Format; Full Context Responses Fully Intersected; Automatically Generated; Variable Structure/ Format; Full Context Responses Variable Narrative Summary; Multi-Media Presentations; Simple Interpreted Results 50/250 Byte Passage from Single Text Document Fixed Templates or Tabular Lists Attacking the Data Chasm

13. Analytic Strategy Analytic Task / Scenario Analyst Analyst Questions 2 Recorders System Questions System Answers Participating Observers Researcher Advanced QA System Of the Future Facilitator Advanced Question AnsweringScenario Exercise

14. Analytic Strategy Analytic Task / Scenario Analyst Analyst Questions 2 Recorders System Questions System Answers Participating Observers Researcher 2 Oracles Facilitator Advanced Question AnsweringScenario Exercise

15. Analytic Strategy Analytic Task / Scenario Analyst Analyst Questions 2 Recorders System Questions System Answers Participating Observers Researcher Advanced QA System Of the Future 2 Oracles Facilitator Advanced Question AnsweringScenario Exercise

16. Analytic Tasks / Scenarios • Common Characteristics: • Description of multi-faceted Information Need • Description of type and scope of the Analytic Output desired; Who is the ultimate customer; What Timeframe is available • Description of the Analyst and his/her background; Previous work experience that is relevant to current task • Three Different Scenario Focuses • Scenario #1: Multiple Perspectives • Scenario #2: Individual of Interest • Scenario #3: Long Term Study of a Topical Area

17. Analytic Strategy • Analyst Describes Overarching Strategy for completing Assigned Analytic Task / Scenario; For Example this Strategy might include: • Types of information sources that might be consulted • What is his/her general expectation of the breadth/depth of information that needs to be collected, analyzed and then included in the final product • A breakdown of your total information needs required to complete the task – preferably in question form • The nature of the final product that is the target of the given scenario • How his/her time & effort would be allocated • The degree of review/consultation

18. Subtask 1:Analyst’s “Questions” • Using Analytic Strategy, Analyst chooses the first subtask to be completed and identifies a set of questions whose answers the analyst believes will allow the analyst to complete this subtask. • Analysts Questions are created without regard to the “capabilities” of the “Advanced QA System of the Future” • Analyst describes the type of answers that would satisfy the analyst • Analyst identifies (optionally) any knowledge that they are assuming that the system knows about the analyst or the assigned task

19. Subtask 1:System “Questions” • Since the Analyst has not worked with an “Advanced QA System of the Future” before, the analyst is fortunate to have the system’s developer, an AQUAINT Researcher, who knows the full capabilities of the system. • The Researcher may ask “Clarification” Questions of the Analyst • The Researcher will create a set of “System” Questions that the researcher believes can be successfully answered by the System. For instance the “System” Questions may be a decomposition of the “Analyst’s” Questions • Researcher will identify those Analyst questions that it believes are beyond the System’s Capabilities • Researcher will describe the type of answer that the Researcher is expecting the System to provide

20. Subtask 1:Oracle Generated Answers • Two Oracles (One Research and one Analyst) will take System “Questions” and will behind the scene in a Wizard-of-Oz fashion generate a System Response Answer • A summary of the Oracles’ activities be recorded but will not show the results of their internal browsing/searching to either the analyst or researcher. • “You didn’t specifically ask for this, but I thought you (the Analyst) might find this of interest – Do you want me to follow-up on this information?” • “I found multiple answers to your question. This is the answer that I believe is more likely correct and here is why I believe that or I can’t determine which answer is correct, so am reporting alternative answers. What do you want me to do next? • “I can not find an answer to your question. I did (or did not) find a partial answer. How would you like me to proceed?”

21. Subtask 1:Analyst Responds to “System Answers” • To each Answer provided by the System (aka “the Two Oracles”), the Analyst may respond • “What is the source/date of the information contained in the answer?” • “What is the justification for a given answer?” • Ask a direct follow-up question that provides “clarification” / “fills in the blanks”. • At the conclusion of given Subtask, the Analyst provides an assessment: • How well did the System’s Answers satisfy the given Subtask information needs • To what extent have future subtasks been altered because of the System’s Answers to date

22. Advanced QA Scenario ExerciseOther Roles • Facilitator: • Keeps things moving; Exercise Driven by a single Analyst and single Researcher • Goal is to complete 2-3 Complete Subtasks • Allow about 30-minutes at end of exercise for open Team-wide discussions / observations / comments • Make sure that focus remains on analytic process, not on technical, system-related issues • Two Recorders: • One records questions & answers on flip chart for all to see • Second one records other significant activities / comments / observations during exercise • Share duties during wrap-up • Participating Observers • Avoid attempting to redirect exercise • Offer observations / comments / alternatives during wrap-up

23. Advanced QA Scenario ExerciseLogistics • Nine total teams – each approximately 12-15 workshop attendees – everyone has been pre-assigned • Three Different Scenarios each having three separate teams completing that given scenario • Last Session on Wednesday – Each of the three teams that worked the same scenario will meet and compare / contrast their experiences • Thursday morning there will be a Plenary Session during which each Scenario will report out what they learned. • Assignments on Handout

24. Scenario Exercise Sample SCENARIO You are an expert on Latin America but have focused your efforts on Cuba and Brazil. You have been asked by the Under Secretary of State for Latin American Affairs to use your familiarity with Latin culture and politics to assess the current situation in Colombia and FARC’s role in destabilizing the country. The report is due by COB tomorrow. Your time is short but you need to familiarize yourself with the issues before feeling confident that you can report on FARC’s current situation.

25. Analyst’s Strategy • Example Sources that might be consulted: • Material from classified collection sources • Database of published intelligence products • Cable traffic from State Department and US Military • Open Internet Sources • Internal working aids • FBIS • Scope of Analysis • Definition of FARC; Background • Goals of FARC • Related/Affiliated Organizations • Political Influence of FARC • US Position on FARC vis a vis FARC’s role in Columbia; vis a vis Columbian Government attempts to control the FARC • Nature of Final Analytic Product • Position/Point Paper with supporting documentation

26. Subtask #1 - Analyst General Information About the FARC • What does FARC stand? • Where is it located? • What is the nature of the FARC? • When was the it founded? • How large is the organization and who are some of its members?

27. Subtask #1 - Researcher General Information About the FARC • What does FARC stand? • What does the acronym FARC stand for? • Where is it located? • In what countries where the FARC has members? … conducts activities? • In what geographic regions where the FARC has members? … conducts activities? In what departments …? In what cities … ? • In what country (geographic region, department, city) is the largest concentration of FARC members? … of FARC associated activities? • What is the nature of the FARC? • What type of organization is the FARC? • What type of activities have been associated with the FARC? • What organizations supports the FARC? • What organization oppose the FARC?

28. Subtask #1 - Researcher General Information About the FARC • When was the FARC founded? • When was the FARC founded? • Why was the FARC founded? • Was the FARC a splinter group of another organization? If so, what organization? What was the reason for the split? • Where was the FARC founded? • List the names of individuals who founded the FARC? Are any of these individuals still active members of the FARC? • How large is the organization and who are some of its members? • What is the total membership of the FARC? • What is the source of this information? • What is the source date of this information? • Who are the leaders of the FARC? • What are their positions? • List the names of other members of the FARC?

29. Other Possible Follow-on Subtasks • Subtask #2: • The Goals and Objectives of the FARC • Subtask #3: • Identify funding sources for the FARC and the FARC’s links with other organizations • Subtask #4: • The Political / Economic / Military influence of the FARC • Columbian Government’s policy towards the FARC; Columbian Government’s activities aimed at defeating the FARC • Subtask #5: • US Government’s evolving position on the FARC

30. Scenario #1 – Group 1Point Loma 1

31. Scenario #1 – Group 2Point Loma 2

32. Scenario #1 – Group 3Point Loma 1

33. Scenario #2 – Group 1Point Loma 1

34. Scenario #2 – Group 2Point Loma 2

35. Scenario #2 – Group 3Del Mar

36. Scenario #3 – Group 1Point Loma 1

37. Scenario #3 – Group 2La Jolla

38. Scenario #3 – Group 3The Bay

39. BY ROOM Point Loma 1 Group 1-1 Group 1-3 Group 2-1 Group 3-1 Point Loma 2 Group 1-2 Group 2-2 DelMar Group 2-3 LaJolla Group 3-2 The Bay Group 3-3 BY GROUP Group 1-1 Point Loma 1 Group 1-2 Point Loma 2 Group 1-3 Point Loma 1 Group 2-1 Point Loma 1 Group 2-2 Point Loma 2 Group 2-3 DelMar Group 3-1 Point Loma 1 Group 3-2 LaJolla Group 3-3 The Bay Breakout Locations

40. Questions / Comments

41. Contact Information Dr. John Prange, AQUAINT Program Director • ARDA Web Pages: http://www.ic-arda.org • Email arda@nsa.gov JPrange@nsa.gov • Phones: 301-688-7092 800-276-3747 301-688-7410 (Fax) • Mailing: ARDA (RA) Room 12A69 NBP#1 STE 6644 9800 Savage Road Fort Meade, MD 20755-6644