Connecting the Silos

1. Connecting the Silos Strategies and Technologies for Sharing and Integrating Geospatial Information

2. Overview… • California Resources Agency’s approach to sharing environmental data and information • Strategies and technologies that address our needs • Plans for the future

3. CERES California Environmental Resources Evaluation System In the beginning…

4. http://www.ceres.ca.gov/ • A program in the California Resources Agency • Started in 1995 when there was very little use of the Internet by state agencies • To address loss of Agency data/information • Resources Agency & conservancy web sites • California Environmental Information Catalog • California Spatial Information Library • Interagency GIS coordination Working with public and private partners as part of the California Environmental Digital Library Network (www.caledln.org)

5. Catalog, Navigation & Discovery Tools Metadata Catalog Spatially Enable Search Tools Web Map Services Registry Geospatial Data In Development In Production Environmental Information Clearinghouse California Environmental Information Clearinghouse Atlas State Library & Archives Geospatial Data Library Web Map Services Documents Library File Exchange Application Programming Interface (API) Search/Retrieve Catalog/Publish/Harvest California’s Environmental Community Studies, articles, books, reports, findings, surveys, databases, photos, collections, etc. Data, Catalogs, Web Sites & Services

6. http://gis.ca.gov/catalog/ The CEIC Catalog is… • Standards based metadata catalog (FGDC & Dublin Core) • Spatially enabled via bounding box and spatial gazetteer • Use of keyword thesaurus for more effective discovery • 228 catalogs containing 11,116 records (5,298 from NRPI) from 201 organizations • Easy entry • Online form • Automated harvest • Content published to Geodata.Gov portal

7. http://gis.ca.gov/ • Started in 2001 • Goal to “Democratize” geospatial holdings • Broaden use of GIS data • Seek stewardship (maintenance) of data • Make data easy to find and understand appropriate use • Expand GIS “services” • Encourage partnerships for data development and enhancement

8. Accomplishments • More than 75 Terabytes Served Since 2002 • “High water mark” to date was more than 225 GB/day in November 2006 • Majority (~85%) of recent demand is for NAIP CCMs • I’m sure we’ll soon receive a call from the UC president’s office about bandwidth congestion. . . • More than 20 TB delivered via sneaker-net • Downloads and sneaker-net deliveries will likely explode with the distribution of the 2005 DOQQs (NAIP Imagery).

9. Accomplishments • Nearly 160,000 files comprising more than 4 terabytes of geospatial data online • Served via HTTP and FTP • Now offered in an easy to navigate thematic folder structure • CaSIL Mirror at UC Davis • Partial Mirror at CalEPA • Working on finding other partners (e.g. San Diego Super Computer Center?)

10. Spatially enabled tools for information discovery and navigation… Watershed Browser GeoFinder http://cwp.resources.ca.gov/browser/ http://casil.ucdavis.edu/cgi-bin/gb/geofinder

11. Working on a new search interface using new technologies… As you zoom out… More data is found A simple interface that can make use of 3rd party web services (e.g., Google Maps)

12. http://atlas.resources.ca.gov/atlas/ ArcIMS client from California Department of Fish and Game Data from many agencies

13. Its time to move on. Data & information need to be… • Made accessible…not just discoverable • Optimized and leveraged (customized to support local processes) • Stewarded (updated & of good quality) • Completed (comprehensive for the time, region and subject of interest) • Standardized (made consistent for combining across regions and the state) • Liberated (effectively licensed, not locked away in proprietary systems) • Prioritized (what do we need first/most) • Funded (for the long term, not just snapshots)

14. Today Far too many “silos” What people really want - Services - Information CPR Vision Still too many “silos” The Governor wanted to “blow up the boxes” but how many boxes should there be? Appropriate technologies wisely applied can help us transcend institutional barriers www Great institutional & political resistance to taking apart and rebuilding the “boxes”

15. The Most Challenging Barriers are Organizational not Technical • Organizational behavior makes data sharing difficult… • Obstreperous • Cantankerous • Uncooperative • Internally focused • Competitive • Special purpose and limited funding • Best practices seen as “unfunded” mandate

16. What is Service Oriented Architecture (SOA)? SOA Platform Event Module Routing Module Report Module Map Module Geocoding Module Application “A” Application “B” A software design that uses programs and data distributed over a network.

17. Organizations can publish and integrate their geospatial data services… Rather than making copies of data, using SOA organizations can continue to maintain their own data bases while offering access to these as web services that can be used by any number of desktop GIS clients and Internet map viewers. Integrated Map Base Map Fire History Public Lands Stressors

18. Base maps from CERES Public land ownership from USBLM Fire history data from CDF Data on environmental stressors from CalEPA organizations We need a better way to share and integrate geospatial data Traditionally, Internet GIS applications take copies of data from others and house them at a single location. For example, the California Digital Conservation Atlas… Typically done as one off applications that can only offer “snap shots” in time as copies of data quickly become out of date.

19. Web Map Viewer A secure, easy to use map viewer capable of accessing federated web map services delivered to any computer connected to the Internet through its web browser. I N S I G H T nteragency etworked ervices ntegrating eospatial osted echnology Web Services Registry A registry of web map services and associated standards for data exchange that organizations can use to create cross departmental information services. Technologies for mobilizing geospatial data from across the Internet

20. INSIGHT makes it safe to share GIS data over the Internet… ArcGIS, Geomedia, MapInfo Pro, open source, etc. Your Organization DMZ User Internet ArcGIS, Geomedia, MapInfo Pro, open source, etc. Spatial Server Web Server Can be configured so that users do not directly interact with data service providers Internal process for building and maintaining GIS data Map Client Registry CERES

21. Organizations remain in firm control, deciding & managing… • Appropriate use • Intended use of the data • Licensing and Intellectual Property Rights • Control who can access the data • Metadata • Information on all of the above to accompany each data set • Data maintenance • Keeping the data up-to-date and accurate • Service levels • Level of availability that a user/subscriber can expect Give a copy of your data away and there is no telling where it will end up or how it will be used. It is better to control it at the source.

22. What does the future hold?

23. 3/16/07 CERES Area of Responsibility Prop 84 & 1E Web Site Project Seeker Full Project Report Thick Map Client Web Services Full Project Report XML Exchange Services Address Matching Services All spatial attributes stored here Project ID used to link data CERES Web Enabled Geo-Warehouse Geospatial Data Error Check Map Validation Map Editor Client Validation Requests Bond Projects Accountability Database DPR Area of Responsibility Query & Report Map Display Client Bond project data entered directly into DPR system by some agencies Data Entry Forms Map Editor Client All tabular attributes stored here DPR Web Enabled Tabular Database Internal “Bonds” databases from other agencies Extract, Transform, Load & Error Check Exception Reports Data Contacts

24. Marshaling Local Data Federal The National Map Extract, Transform, Load State Emergency Response, Resource Conservation, Health Services, Transportation, etc. 1 2 n , … Extract, Transform, Load Services & Funds Regional 1 2 n , … Extract, Transform, Load Local 1 2 n 1 2 n , … , … 1 2 n Parcels, Roads, Critical Infrastructure, etc. , …

25. Suggested changes communicated back to data “owners” and they decide whether to accept or reject. Changes made to source databases and propagated back to GeoCommons. Edit Layer (“sandbox”) Org 1 Objects copied into Edit layer to change Org 2 Data Extraction, Standardization and Loading Master (read only) Provide option to expose any layer as a web map service . . . Once acted on, suggested changes are removed from Edit layer and moved into the Journal layer Journal (change history, read only) Org n Base Map Layers (read only) DATA COLLBORATION GeoCommons Consideration may be given to enabling the “borrowing” of line work from base layers for work in the Edit layer California GeoCommons • Members should be able to: • See all layers including other proposed changes • Select objects from Master to change • Import shapefile into Edit layer • Import GPS coordinates into Edit layer • Do heads up digitizing in Edit layer • Edit copy of attributes Information on action taken on suggested edits communicated back to the GeoCommons. 7-25-06

26. California GeoCommons • A collective view is expressed in the GeoCommons’ Master layer. • Source systems continue to maintain their own unique structures and designs. • A common “view” or core record standard must be negotiated or determined prior to starting. • Accounts within the GeoCommons are created and administered by collaboration leads.

27. Questions? John Ellison Agency Technology Officer & CERES Director john.ellison@resources.ca.gov (916) 653-2238