Big Data Symposium: Analytics and Applications for Federal Big Data - FEMA

1. Big Data Symposium:Analytics and Applications for Federal Big Data - FEMA Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community http://semanticommunity.info/ AOL Government Blogger http://gov.aol.com/bloggers/brand-niemann/ March 5-6, 2013 http://semanticommunity.info/Big_Data_Symposia

2. Begin With the End in Mind • Ms. Jo Strang, Associate Administrator, Safety, Federal Railroad Administration, Department of Transportation “Open Gov 2.0 and Safety.Data.Gov” • New safety data sources and challenge from the National Institute of Justice and new data from Open FEMA • Open FEMA • Could not find with Google search • Start with Fema.gov • http://www.fema.gov/ • Safety.Data.Gov • Could not find at that Web site • Start with Bureau of Justice Statistics • http://www.bjs.gov/

3. My Process • Found Data Feeds at Fema.gov • Documented the Web Site, Metadata, and Data Sources for: • USGS Earthquakes • World Map • FEMA Disaster Declarations Summary • FEMA Hazard Mitigation Program Summary • Public Assistance Subgrantee Funded Projects Detail • Public Assistance Funded Projects Summary • Used Google Earth for the FEMA and USGS KML Files • Imported the Above Data Sets Into Spotfire • Created a Separate Visualization Tab for Each Data Set in Spotfire • Started a Unified Data Architecture • Submitted a Question About the Data Definitions to FEMA • Documented My Data Science Work in PowerPoint Slides • Provided Conclusions and Recommendations

4. Fema.gov Home Page Data Feeds http://www.fema.gov/

5. Fema.gov Data Feeds http://www.fema.gov/data-feeds

6. Fema.gov Data Feeds Examples • RSS: • Disaster Declarations • Query Form and List of Links • KMZ: • Disaster Declared Counties - National View • Designated Counties • META and XLS: • Disaster Declarations Summary • Same as RSS Disaster Declarations • Hazard Mitigation Program Summary • Web Page • Public Assistance Funded Projects Detail • No Web Page • Public Assistance Funded Projects Summary • Web Page • KML and CSV: • Worldwide M1+ Earthquakes, Past 7 Days (USGS) • KMZ and TXT (No Web Page) My Note: But I found a Way Cool One! • http://earthquake.usgs.gov/earthquakes/map/

7. RSS: Disaster Declarations http://www.fema.gov/disasters

8. META: Disaster Declarations Summary XLS http://www.fema.gov/library/viewRecord.do?id=6292

9. XLS: Disaster Declarations Summary Instructions Metadata View / Download / Print  (xls 9004K)

10. Hazard Mitigation Program Summary http://www.fema.gov/hazard-mitigation-grant-program

11. META: Hazard MitigationProgram Summary XLS http://www.fema.gov/library/viewRecord.do?id=6293

12. XLS: Hazard MitigationProgram Summary View / Download / Print  (xls 16563K)

13. META: Public AssistanceFunded Projects Detail XLS http://www.fema.gov/library/viewRecord.do?id=6295

14. XLS: Public AssistanceFunded Projects Detail View / Download / Print  (xls 129914K)

15. Public Assistance FundedProjects Summary http://www.fema.gov/public-assistance-local-state-tribal-and-non-profit

16. META: Public Assistance FundedProjects Summary http://www.fema.gov/library/viewRecord.do?id=6299

17. XLS: Public Assistance FundedProjects Summary Instructions Metadata Projects 1-9 Applicants View / Download / Print  (xls 18935K)

18. Worldwide M1+ Earthquakes,Past 7 Days (USGS) TXT not CSV Spotfire handles that! http://earthquake.usgs.gov/earthquakes/catalogs/eqs7day-M1.txt

19. Import All Data Sets (15) Into Spotfire

20. Default View in Spotfire Illustrates Data Visualization Design Using Shneiderman’s Mantra: Overview First, Zoom and Filter, Then Details-on-Demand

21. KMZ: FEMA Designated Counties and USGS Worldwide M1+ Earthquakes, Past 7 Days (USGS) https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

22. USGS Earthquakes: Spotfire My Note: It is hard to see at this scale, but the data points are all related in each visualization. My Note: Could Spotfire do this? Can USGS do what Spotfire does? https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

23. World Map and Data Tables: Spotfire My Note: I needed a world map and to see how to integrate it with the Earthquake and other data tables. This Spotfire functionality helps me do this and is the beginning of a “unified data architecture”(Scott Gnau, President, Terra Data Labs). https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

24. FEMA Disaster Declarations Program Summary: Spotfire My Note: The visualizations are connected! https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

25. FEMA Hazards Mitigation Program Summary: Spotfire My Note: The visualizations are connected! https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

26. FEMA Public AssistanceFunded Projects Detail: Spotfire https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

27. FEMA Public Assistance FundedProjects Summary: Spotfire https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?FEMA.gov-Spotfire

28. Conclusions and Recommendations • USGS Earthquakes: • Could Spotfire do this? Can USGS do what Spotfire does? • Google Earth of FEMA and USGS: • Need a way to access Google Earth within Spotfire. • World Map: • One can easily find and import geospatial data files (Shape) into Spotfire. • Unified Data Architecture: • Spotfire provides a way to produce this by displaying all the data tables and their columns. • FEMA Disaster Declarations Summary: • I submitted a question about the data definitions to FEMA and am waiting for an answer. • FEMA Hazard Mitigation Program Summary: • There is county data, but the names do not match those used in the standard Shape file for county maps so I had to use the State designation. • Public Assistance Subgrantee Funded Projects Detail: • Parts 1 and 2 of the Data Set need to be merged. • Public Assistance Funded Projects Summary: • The nine additional tabs of spreadsheet data should be used in a separate Spotfire file.