Build EPA’s BP Oil Spill Data Tools in the Cloud

1. Build EPA’s BP Oil Spill Data Tools in the Cloud Brand Niemann US EPA July 10, 2010 http://semanticommunity.net Disclaimer: These slides do not reflect the views of the U.S. Environmental Protection Agency and does not constitute endorsement by the EPA of the standards or products mentioned.

2. Overview • The Challenge • The Socrata and Google Earth Programs • The Expert and His Advice • The Cloud Tools • The Inspiration • The Data Sources • Other Sources of Data • The Process • The Results • Comments • Acknowledgements • References • Footnotes

3. The Challenge • BP Oil Spill Data Tools: • As part of our response to the ongoing BP oil spill in the Gulf of Mexico, we’ve been analyzing environmental conditions, including air, water, sediment, and oily wastes.  Right from the beginning, we’ve followed our open government approach, which means providing the data as soon as we have it.  We’ve posted data in CSV files, a format that people could download and open in a spreadsheet.  We’ve also provided printable PDF tables and summaries on our BP spill site. • But we knew that people needed other ways to get the data, and we’ve been working on several options.  We launched Google Earth a few weeks ago, and today we launched Socrata. • Source: Jeffrey Levy, EPA’s Director of Web Communications, Greenversations, July 8, 2010. http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Challenge

4. The Challenge • BP Oil Spill Data Tools (continued): • I’m excited to share these tools, but we can always improve them.  In the comments section of this post, please give us your suggestions.  Some examples: • Filtered views to provide beyond just what we’ve detected • Different ways of sorting the data • Mashups (ways to combine the data with other information) • And if you create your own views or download the data and produce interesting stuff, let us know! • Source: Jeffrey Levy, EPA’s Director of Web Communications, Greenversations, July 8, 2010. http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Challenge

5. The Socrata Program See next two slides http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Socrata_and_Google_Earth_Programs

6. The Socrata Program http://www.epa.gov/bpspill/socrata.html

7. The Socrata Program See next slide http://www.socrata.com/profile/U-S-Environmental-Protection-Agency/yahm-crqu

8. The Socrata Program Download data http://www.socrata.com/Government/EPA-Air-Sampling-Response-to-BP-Oil-Spill/498c-fwrs

9. The Socrata Program Metadata Also download data http://www.socrata.com/Government/EPA-Air-Sampling-Response-to-BP-Oil-Spill/498c-fwrs/about

10. The Google Earth Program See next slide http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Socrata_and_Google_Earth_Programs

11. The Google Earth Program Click to download File in Google Earth. Note: Need to install Google Earth client first. http://earth.google.com/ http://www.epa.gov/bpspill/popups/googledatapopup.html

12. The Google Earth Program http://www.epa.gov/bpspill/mapfiles/BPGulfSpillResponse.kmz in Google Earth

13. The Expert and His Advice • Edward Tufte Presidential appointment announced by White House, March 5, 2010. • Tufte Comment on iPhone interface design: Better to have users looking over material adjacent in space within our eyespan rather than stacked in time. This is especially the case for statistical data, where the fundamental analytical task is to make comparisons. Also see page 159 in the book reference below. • Edward R. Tufte, Beautiful Evidence (2006), Graphics Press LLC. http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Expert_and_His_Advice

14. The Cloud Tools http://cloud.mindtouch.com/

15. The Cloud Tools http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools

16. The Cloud Tools http://spotfire.tibco.com/

17. The Cloud Tools http://ondemand.spotfire.com/public/Help/index.htm

18. The Inspiration H1N1 Spread Courtesy of TIBCO Spotfire. See Web Player. http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Inspiration

19. The Data Sources http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Data_Sources

20. Other Sources of Data • NOAA interactive map tracks Gulf oil spill • Online mapping tool allows public to follow closures, spill trajectory • The public can now track Deepwater Horizon-BP oil spill recovery data online via a near-real-time interactive map at a new Web site created by the National Oceanic and Atmospheric Administration. • The GeoPlatform Web site, launched June 15, includes regularly updated geospatial data from several federal and state agencies on the oil spill trajectory, closed fishery areas, impact on wildlife and Gulf resources, daily position of research ships, and affected shorelines. • Data published on the site is received from NOAA, the U.S. Coast Guard, the Environmental Protection Agency, the U.S. Fish and Wildlife Service, the U.S. Geological Survey, the Homeland Security Department, NASA and several states. • “This Web site provides users with an expansive, yet detailed geographic picture of what’s going on with the spill,” Jane Lubchenco, NOAA administrator, said in the news release June 15. “It’s a common operational picture that allows the American people to see how their government is responding to the crisis.” • NOAA worked with the University of New Hampshire's Coastal Response Research Center to develop the Web-based geospatial platform designed for federal and local response activities and adapted for public use. • A separate public Web site, Deepwater Horizon Response sponsored by federal agencies, has been operating for several weeks offering news, announcements and information about the disaster. It is operated by the Deepwater Horizon Unified Command, which consists of DHS and the Defense and Interior departments, as well as other federal agencies, BP and other private entities.  http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#Other_Sources_of_Data

21. Other Sources of Data http://www.geoplatform.gov/gulfresponse/

22. Other Sources of Data http://www.geoplatform.gov/gulfresponse/

23. The Process • The Basic Steps: • Inventory Data Sources and Plan Application • Prepare and Import Data and Metadata • Implement Layout and Analytics • Add Bookmarks and Create Data Stories • Publish and Test in Web Player • Get Feedback and Improve • First create visualizations, faceted search (filters), and analytics for each individual data source and then look for relationships between the data sources. http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Process

24. The Results Spotfire on the PC http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Results

25. The Results Spotfire on the PC http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Results

26. The Results Spotfire on the PC http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Results

27. The Results Spotfire Webplayer http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Results

28. Comments • The initial objective to see how fast one could create this basic application using Spotfire's Multiple Visualizations (Cross Table, Table, and Scatter Plot) and Hyperlinks (Between Multiple Tabs and to the Web) and mash it up with the the NOAA GeoPlatform layers. It provides the same results as Socrata's More Views and builds on the NOAA GeoPlatform layers by adding statistical visualizations of the data (see The Challenge and The Socrata and Google Programs previously). • Please use the Add Comment feature at the bottom of this wiki page to provide feedback and suggest additional analyses you would like to see. To use the Add Comment feature you first need to register by providing your email address. Your privacy will be respected and your email address will not be available to others or used for any other purpose. You can also download the Spotfire File (about 2 MB) from this Wiki and a 30-day free evaluation copy from http://spotfire.tibco.com/ and reuse these analyses, add your own data to this file or new Spotfire files that you create. Have fun and give us your feedback! http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#Comments

29. Acknowledgements • The author acknowledges gratefully Dean Allemang, Cory Casanave, Sean Connors, Mills Davis, Li Ding, David Eng, Lee Feigenbaum, Aaron Fulkerson, Jim Hendler, Ralph Hodgson, Kevin Kirby, Kevin Jackson, Bob Marcus, John McMahon, Richard Murphy, Brand Niemann, Jr., Barry Nussbaum, Matthew Phoenix, Tony Shaw, Jeff Stein, George Strawn, George Thomas, Pete Tseronis, and Edward Tufte. http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#Acknowledgements

30. References • Brand L. Niemann, Put Your Desktop in the Cloud to Support the Open Government Directive and Data.gov/semantic, Semantic Universe, April 19, 2010. • Brand L. Niemann, Build Your Own Data.gov (Spotfire) and EPA Microsite (Spotfire) with Semantics and Statistics in the Cloud, Slides, May 15, 2010. • Brand L. Niemann, Build Your Community Health Information "Design for America" Using Mindtouch and Spotfire, Slides, May 17, 2010. • Brand L. Niemann, Build EPA's CASTNET in less that two hours, see Mindtouch and Spotfire, Slides, May 21, 2010. Completed in 8 more hours by May 30, 2010. • Brand L. Niemann, Build Your Own Data.gov/semantic with Mindtouch and Spotfire in the Cloud: The White House Visitor Database, Slides, May 22, 2010. See Data.gov takes the 'Mumsy' test, FCW, May 26, 2010. See Mindtouch and Spotfire, Updated July 7, 2010 (see below). Compare to Findthebest's WH Visitor Logs Comparison Tool. • Brand L. Niemann, Build EPA's EPA's Facility Registry System (FRS) and Locational Reference Database with Mindtouch and Spotfire in the Cloud: Virginia, No Slides, June 1, 2010. • Brand L. Niemann, Build the UK’s COINS in the Data Science Library Cloud (Mindtouch and Spotfire), Slides, June 9, 2010. • Brand L. Niemann, Build EPA's Envirofacts in the Cloud: Virginia FRS, NPL, and TRI (Mindtouch and Spotfire), Slides. June 14, 2010. • Brand L. Niemann, Build the SemTech 2010 in the Cloud (Mindtouch and Spotfire), No Slides, July 2, 2010. • Brand L. Niemann, Build the White House Staff Salaries in the Cloud (Mindtouch and Spotfire). Slides, July 3, 2010. http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#References

31. Footnotes 1 • From: Jeffrey Levy/DC/USEPA/US@EPA • To: CONTENT-MANAGERS-L@HUDLIST.HUD.GOV • Date: Thursday, July 08, 2010 06:21PM • Subject: We're using Google Earth and Socrata to provide data about the oil spill • Hi everyone. On our BP oil spill site, we've been providing CSV and PDF files from the beginning.  We're working on some enhancements to our database's capabilities, too, especially around providing context and summaries. • But we're also using two tools to expand how people can access and download our data, epsecially raw data in the spirit of open government. • Here's the blog post I just wrote about it: • http://blog.epa.gov/blog/2010/07/08/...ll-data-tools/ • Jeffrey Levy • Director of Web Communications • Office of Public Affairs • U.S. Environmental Protection Agency • levy.jeffrey@epa.gov • 202-564-9727 (cell for emergencies: 202-309-9445) • EPA's blog: http://blog.epa.gov | Twitter: • http://twitter.com/greenversations • EPA news: http://www.epa.gov/newsroom | mobile: http://m.epa.gov/news | • Twitter: http://twitter.com/epanews http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#Footnotes

32. Footnotes 2 • EPA's Work on the Gulf Coast • http://epa.wik.is/EPA's_Work_on_the_Gulf_Coast http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#Footnotes