Mining Social Media in Extreme Events: Lessons Learned from the DARPA Network Challenge

1. Mining Social Media in Extreme Events: Lessons Learned from the DARPA Network Challenge Nicklaus A. Giacobe, Hyun-Woo “Anthony” Kim and AvnerFaraz [nxg13, hxk263] @ist.psu.edu, fia5005@psu.edu

2. The DARPA Network Challenge • Conceived to better understand and harness the power of the Internet • How does one get a video to go viral on YouTube? • Leveraging social networks to solve “intractable” or “impossible” tasks • Short time frame • Announced on October 29, 2009 • Challenge occurred on December 5, 2009 (5 weeks of possible prep time)

3. The DARPA Network Challenge • DARPA launched 10 red weather balloons • Tethered to fixed locations in public places across the continental United States • Find them and report GPS coordinates of all 10 before anyone else does and win the $40,000 prize • The balloons would be “up” from 0900 EST until 1600 local time (7-10 hours) and then taken down

5. Final Results • 4600 “Teams” registered with DARPA • 58 Teams were “in the hunt” and submitted 2 or more correct locations • Top teams used mass marketing and mass motivation techniques (offered to share the prize $ with observers) • Each team has a “lesson” applicable to homeland security

6. Top Finalists – We’re #10! From: https://networkchallenge.darpa.mil/FinalStandings.pdf

7. Overview • Who are the ? • Caucus of academic institutions studying information science • The iSchools are interested in the relationship between information, peopleand technology. • There are 28 academic Institutions (Colleges or Departments) in various universities across 8 countries • Some iSchools have roots in Library Science, Computational Sciences, MIS, Business Management, Cognitive Science, Human-Computer Interaction (HCI) and other fields. Each iSchool has its own focus and competencies.

9. Team Organization • Command Structure • Attempted to follow ICS from Fire Service • Most Team Members unfamiliar with this organizational structure, therefore very limited success • Operational Section • 2 Branches – Direct Observation and Cyberspace Search • Facilities • 211 IST – EEL – Command Post • 208 IST – Classroom – Cyberspace Search

13. Methods • Direct Observation • Recruit observers from the iSchools Caucus • Report sightings through • Website • Phone / SMS • Email • Cyberspace Search • Multiple Intel Teams searching open communications online • Twitter, Competitor Websites, No Hacking • Confirmation and Decision Making

14. Methods • Technologies Used • Twitter Capture – Anthony Kim • Custom Crawler – Madian Khabsa • Maltego2 – AvnerFaraz

15. Methods

16. Methods • Dempster-Shafer Evidence Theory Combination • Combine evidence from multiple sources under uncertainty • Apply confidence weights to sensor data • Intended, but applied cognitively • Analysts were to provide report data with confidence values (0=low, 10=high) • Some algorithmic process would have been needed to combine large numbers of reports • … but we had *extremely low* # of reports

17. Pre-Challenge Org.

18. Team Organization • Other Universities • Various Schools to send recruiting messages • UNC – distributed own phone number and email address for their recruiting messages • Univ. of Illinois – single-person cyberspace search division

20. Team Organization • Finance and Logistics • Google Voice (814-4BALL01) • Website Design (balloon.ist.psu.edu) • Email Address (balloon@ist.psu.edu) • Google Wave (Intel and Command Comms) • Private Twitter (late attempt at outbound comms) • Coffee, Donuts, Pizza, Soda and Homemade Cookies ($100!) • Incentives and Rewards (10 GPS Systems Offered by iSchools)

21. Team Members • University of Illinois • John Unsworth • Maeve Reilly • Karyn Applegate • University of North Carolina • Aaron Brubaker • Kjersti Kyle • Other iSchools • Marketing/Communications Staff

22. Team Members • Penn State University • Command Post Team • Nick Giacobe • Wade Shumaker • Louis-Marie NgamassiTchouakeu • John Yen • Jon Becker (p/t) • Michelle Young (p/t) • Logistics / Website • Shannon Johnson • Lei Lei Zhu

23. Team Members • Penn State University • Operations (Cyberspace Search Branch) • Crawler Task Force • Madian Khabsa and Jian Huang • Twitter Capture Task Force • Hyun Woo “Anthony” Kim and Airy Guru • Intelligence Analysts • Chris Robuck • Greg Traylor • Anthony Maslowski • Gregory O’Neill • Joe Magobeth • AvnerFaraz • Matt Maisel • Earl Yun

24. Results • Recruiting Message Reports • University of Illinois – Email to iSchool alumni; to faculty, staff and students. • University of Pittsburgh – Email to all alumni; to faculty, staff and students; Webpage article on main page & alumni news page; LinkedIn announcement to iSchool at Pitt group members; and Facebook announcement to iSchool at Pitt group members. Alumni email distributed to 4,674 alumni. 114 opened the DARPA link. Email blasts to 936 students 41 faculty. • Penn State – 338 fans on Facebook and 377 followers on Twitter. Re-tweets, questions about the Challenge on Twitter, but no activity on Facebook. 2,125 alumni received the e-mail. • UCLA – Sent messages to faculty, staff, students and alumni. • Drexel – Alumni listserv, Facebook, Tweeter electronic newsletter for undergrads and grads, online learning system grad web site release.

25. Results • Direct Reporting Data • 1 Report From Observer • 1 Pre-recruited Observer tasked for confirmation • 8 Observers recruited for confirmation • Website Hit Data • 567 hits (Terrible!) • 16 Case Studies of Individual Balloon Reports…

26. Overview of Cases

27. Results: Case Study 1 • Location: Erie, PA • Original Report: 10balloons.com website • Method: Observer mobilized • Notes: • Early report – 9:30AM • Evaluated whether 10balloons.com website was going to be useful intel or not • Observer was not pre-registered, was personal friend of command post personnel

34. Results: Case Study 2 • Location: Albany, NY • Original Report: Twitter Feed • Method: Observer mobilized • Notes: • Observer provided convincing photo evidence – no balloon at this location • Subsequent photo analysis confirmed this was a manufactured image

35. Results: Case Study 2 • Evidence for • Reputability (3/10) • Established Twitter Acct • Content (6/10) • Photo of a balloon • Balloon Number 6? • Weather – match • Location – exact coords not provided, but discoverable • Evidence Against • Reputability (9/10) • Pre-recruited observer • Known person (PSU Alumni Assn chapter president) • Content (10/10) • Excellent quality photo • Same angle, confirming coincidental landmarks/features

42. Results: Case Study 3 • Location: Royal Oak, MI • Original Report: Twitter • Method: Observer Mobilized • Notes: • Observer went to the location and talked to the store owner who admitted that the balloon was a “publicity stunt” – knew about DARPA challenge and put up own balloon. • Observer was not pre-recruited

43. Results: Case Study 3 • Evidence for • Reputability (3/10) • Established Twitter Acct • Content (4/10) • Photo of a balloon • No “DARPA” pennant , not balloon 6 • Weather – match for location • Location – Had GPS coordinates for location from photo • Evidence Against • Reputability (9/10) • Observer recruited after the fact • Content (8/10) • No photos sent • Good report, verbal description of events

44. Results: Case Study 4 • Location: Providence, RI • Original Report: Twitter • Method: Observer Mobilized • Notes: • Observer was a friend of one of the analysts in the Intel Division – reported no balloon at that location • Photo analysis provided repeat of the exact fabricated image

45. Results: Case Study 4 • Evidence for • Reputability (3/10) • Established Twitter Acct • Content (4/10) • Photo of a balloon • Evidence Against • Reputability (9/10) • Observer recruited after the fact • Content (8/10) • No photos sent • Good report, verbal description of location • Photo Evaluation (10/10) • Reproduced shop job • Edge evaluation

48. Results: Case Study 5 • Location: Seattle, WA • Original Report: • Method: Observer Mobilized • Notes: • Report was for balloon over the University of Washington Library • Observer was Assoc. Dean of LIS College at UW

49. Results: Case Study 6 • Location: Champaign, IL • Original Report: 10Balloons.com • Method: Observer Mobilized • Notes: • No picture • Known observer/Team member in close proximity to this location

50. Results: Case Study 7 • Location: Des Moines, IA • Original Report: Twitter • Method: Self-Recant • Notes: • Tweet gave location/street address of reporters home • No photo evidence • Reporter then re-tweeted complaining about people running through her yard and then recanted