Scenario Wrapup “Shangri-la” – QA Support in 7-10 Year

1. Scenario Wrapup “Shangri-la” – QA Support in 7-10 Year Dr. John D. Prange AQUAINT Program Manager jprange@nsa.gov 301-688-7092 http://www.ic-arda.org

2. How many operatives remain in America & what forms of terror do they have in mind? Task • Two Approaches • Top Down • Bottom Up • Both Approaches worked simultaneously by Analysts • Goal is to see if the two approaches can be connected somewhere in the middle; Can we “Marshal the Evidence to Build a Case” that will directly addresses the assigned task.

3. Top Down • Top Level Questions: • What are the possible forms of terror? • Candidate: Man Portables • Who are the possible perpetrators/operatives? • Suspected Al-Qaeda operatives • Members of God’s Aryan Militia • What are the possible modes of operations? • Weapons theft from military • Surplus weapons purchases • Suspicious transactions • Transportation of suspicious materials / movement of suspects • Task: For each possibility – decompose into sufficiently fine sub-elements; indicators, plan outlines, time lines, and then search for data that supports these elements. • How does the data that the analyst has in hand or can find fit these finer-grained models?

4. Current US /Yemen Relations Terrorist Activities What recent terrorist activity/Incidents in Yemen? Against US Military world-wide? What are the potential threats? ? ? POSSIBLE SOURCES Current Intel Reports; Latest Terrorist Assessments; Historical Incident Reports POSSIBLE SOURCES Other Analysts; State Department/ Country Team Cables; Recent Contact with Embassy/Consulate What is the current state of US relations with Yemen Other countries in region? ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. ……………………….………………….. Harbor Facilities/ Security Plans Current Yemen Environment POSSIBLE SOURCES Imagery of Harbor; Annotated Maps; Military Attaché Video of Port Facilities; Naval Orders/ SOPS/Plans ? POSSIBLE SOURCES Local Print, Radio, Broadcast News Reports Other Analysts ? What is the Current Political/Social/ Cultural climate in Yemen? In the region? How secure is the Aden Harbor? Are our security plans adequate? How secure is the Aden Harbor? Are our security plans adequate? An Asymmetric Threat Scenario * How safe is the Muscat harbor for refueling US Navy ships? * Reference: SRI International AQUAINT Briefing

5. Bottom Up • Try to form a more coherent picture of the available data / evidence that has been obtained. Use “Shangri-la” QA system to: • Create detailed timelines of the available data • Create multiple Link Diagrams to display different types of relationships between events, people, places, materials • Create overlays of these timelines and link diagrams • Use “Shangri-la” QA system to • Develop detailed profiles, background information on each event, people involved, organizations, materials, weapons, vehicles etc • Cast a “Wide Net”; Follow Leads • Create a broad foundation of known / suspected facts that appear to be related to the current investigation • Identify gaps, missing data  What don’t I know? And if I knew it what good would it do me?

6. A Different Paradigm may be useful when handling QA Scenarios: Current Analytic Paradigm: • Sequentially “Filter Down” to the • final result • Cast a “wider net” while searching • for “golden nuggets” (Answers) Data Background Processing & Analysis Discarded Answers Space of Data Objects and Sources Results • Automatically Extract, Represent, • and Preserve “closely related” • background information within • context of the QA Scenario QA Scenarios: A Different Paradigm? How Wide to Cast the “Net”? What Info to Retain? In what form? For how long? • Works when QA’s are • independent, isolated activities

7. Top Down Approach Case Bottom Up Approach Marshal the Evidence to Build a Case Shangra-La QA System assists Analyst in finding closure between the two approaches

8. Advanced Reasoning: • Use Multi-level Plans • Create and evaluate • chains of reasoning • Reason across hetero- • geneous data sources • Infer answers from • data extracted from • multiple sources when • the answer is not • explicitly stated • Utilize Link Analysis & • Evidence Discovery • Plus other strategies Follow-up Leads Follow-up Leads Associates Associates Education TV & Radio Broadcasts, Newspapers & Other Archives Past Positions Raw “Bio” Information Collected Views Family Travels New Senior Advisor Other Activities Cross Fertilization Summarized Results Summarized Results “Views: Past & Present” .….… ….….. .……. ….….. .……. ….….. .……. ….….. .……. ….….. “Bio” ………..…. ……..……. ………..…. ……..……. ………..…. ……..……. …………... • Advanced Learning: • Automatically • learn new or modify • existing reasoning • strategies Improved Reasoning & Learning

9. Final Comments • Shangra La QA System: • Not really a system but an integrated suite of tools • Allows access to tools/capabilities that fall outside the scope of Advanced QA • Learns and adapts as the task unfolds • Capabilities in the small exist today but they need to be done far more automatically with little or no direct analysts involvement (e.g. the metadata hotlink tags done automatically in the large) • Knows its limitations and can adequately share those with analysts • Can provide explanation and maintains understandable audit trails • Guts of the system are transparent to the analyst • HCI issues/concerns have been significantly addressed

10. Final Comments • IC Analysts: • Understands the capabilities and limitations of the Shangra-la QA system; How the system interfaces with the Analyst’s methods • Knows how to ask questions that the system is capable of responding to • Uses “Evidence Marshalling Strategies” and knows how to use the Shangra-la System to: • Formulate hypotheses and then decompose them into subcomponents that can be researched and investigated • Develop appropriate arguments and build a “Defendable Case” • Seek both confirming and disconfirming evidence • Construct scenarios out of the available evidence

11. Final Comments • It’s a team effort • IC Analyst + Shangra-la QA System • Use a Football Analogy • Shangra-la QA System • “Franchise” Player BUT … • IC Analyst is the “Quarterback/Coach” • Knows the scouting report • Develops the game plan • Calls the plays • Make the tough decisions