ICEDS – Creating a continental-scale on-line geographic information service

1. ICEDS – Creating a continental-scale on-line geographic information service Jeremy Morley, Nuno Gil, Jan-Peter Muller, Cristiano Giovando, Kashif MunirDept. Geomatic Engineering, UCL Oliver Greening, Rob ScottESYS plc

2. On-line GIS • Fast developing area • New distributed processing technologies: • Web services components (SOAP etc) • Grid processing • Aided by standards • Data transfer formats • Data access methods • OpenGeospatial Consortium central + ISO TC/211 • ICEDS: Integrated CEOS European Data Server • Demonstrator sponsored by British National Space Centre International Co-operation Programme

3. ICEDS Aims • To promote Web access to large datasets • Europe (to Urals) and Africa • SRTM & Landsat mosaics • To prepare: • An OpenGeospatial Web Map Service (WMS) • Web Coverage Service (WCS) • Portal website • To provide feedback on data preparation process • To use open access (i.e. open source / freeware / free to download) software • MapServer: http://mapserver.gis.umn.edu/

4. OGC Web mapping • Open Geospatial Consortium (new name for OpenGIS Consortium from September 2004) • Provides practical software standards for data and processing interchange of geospatial information • Web Map Service (WMS) • Provides rendered maps (i.e. a portrayal service) • Web Coverage Service (WCS) • Provides user-definable extracts of raster (coverage) data (i.e. live data connection) • Web Terrain Service (WTS) • Provides rendered images of 3D terrain • Web Feature Service • Provides access to vector and annotation data • Interface is simply through URLs on HTTP • Stateless • No need for low-level SOAP infrastructure, but experiments with this will commence shortly

5. Hardware • Primary ICEDS server: • Linux RAID Server with ~2TB capacity • Mandrake Linux • Apache WebServer and Tomcat servlet container • Processing carried out on a P4 laptop machine and P3 desktop with 20GB local disk. • Test web-server: IIS 6 on Win2003 Server - Dell Precision 210 - Pentium 3, 497MHz, 512MB RAM, 60GB local disk. • Sun Sparc Ultra 60, with a 20GB local disk, for editing in IDL/ENVI and ArcInfo • Further RAID server for additional data storage (Landsat, SRTM, ArcCoverages, Grids)

6. Software • UCL GE Dept. software: • ENVI 4.0 (SRTM and Landsat processing and editing) • ArcView 3 and ArcGIS 8.3 (coverages and vector data processing) • Free software: • DLGV32 Pro NASA release (SRTM colourised hillshading) • MicroDEM • Libraries and tools • GDAL libraries: GDALtileindex (shapefile indexing) • Global Mapper (full version of DLGV32 – low cost) • Free libraries for use with MapServer (including GDAL) • Map servers tested • MapServer 4.4.0 beta1 • IONIC RedSpiderWeb 3.1 (WMS setup and website design) • Various IONIC tools (indexers, editors, pyramid-creator) • DeeGree v.1 • Deegree tools: runtileimage2

7. Current datasets • Hosted datasets: • 3 arcsec. unedited SRTM DEM • Downloadable from Eros Data Center (EDS) • GSHHS global coastlines dataset (based on WVS) • Downloadable from NOAA website • JPL Landsat 5 full-res mosaic of Africa • 141GB on RAID Server • Courtesy of Nevin Bryant (JPL) • NASA MODIS "Blue Marble" global composite • Cascaded datasets: • NASA MODIS "Blue Marble" global composite • http://onearth.jpl.nasa.gov/gis-wms.cgi? • NASA LANDSAT 7 Global mosaic • http://onearth.jpl.nasa.gov/gis-wms.cgi? • ESRI dataset of inland waters (from DEMIS) • http://www.demis.nl/mapserver/request.asp? • “Bathymetry” derived from MODIS “Blue Marble” data • http://maps1.intergraph.com/wms/world/request.asp? • Any OGC compliant services worldwide can be cascaded

8. Data & server preparation

9. Guidelines document • A guidelines ‘cookbook’ is available on the ICEDS website http://iceds.ge.ucl.ac.uk • The latest release (v 1.1) details the deployment of the Minnesota MapServer • Full details of server software setup and implementation • Full details of data preparation and hosting • This document will be kept up to date as the service evolves. Expect a new draft release in September 2005

10. JPL Landsat 5 Africa mosaic contrast enhancement • Linear stretch proven to be better • Upper limit 98th percentile • To handle the possible (numerous) presence of black pixels around the image edges, without compromising the stretch quality, the lower limit is automatically adjusted. Default value is the 2nd percentile

11. JPL Landsat 5 Africa mosaic –retiling & pyramidal structure • Retiling operation was performed in • 5ºx5º contrast stretched composite tiles (WMS) • 5ºx5º original tiles in separate bands (WCS) • Pyramidal structure for WMS: • 0.0002778°/pixel from scale 1:1 to 1:1 000 000(14302 tiles) • 0.002778°/pixel from scale 1:1 000 000 to 1:8 000 000 (1020 tiles) • 0.02778°/pixel from scale 1:8 000 000 to max (1 tile) • Each level has its own index file to allow the map-server application to find and get the right files

12. SRTM retiling & hill shading • For portrayal in WMS, preferred portrayal for SRTM was colourised & hill shaded. • Preferred software: DLGV32 or Global Mapper • Handles SRTM HGT; can take user-defined colour table • Unedited SRTM data contains elevations over ocean – masked out in hill shaded images • An additional issue: raw HGT files from EDC contain additional overlap pixels • Overlap pixels need to be removed for processing in Global Mapper / DLGV32

13. SRTM in DLGV32 / Global Mapper • DLGV32 reads the native *.hgt files and displays the data with a user-defined elevation colour LUT and greyscale hillshading • Missing pixels are displayed as transparent - when exported they appear with the background colour (red) • Possible to export the data into different raster image or elevation formats

14. Hill shading colour table • A colour table has been defined reflecting atlas-like colour ranges • The “lowest green” ranges from +20m to -∞m, to include all possible areas below sea level • The resulting “green-sea” is then masked with vector layers

15. India SRTM hill shading • Same colour table used for Africa and Europe Area of Tibet has an elevation consistently greater than 5,000m and appears all in purple, the RGB values combination previously assigned to the highest elevation range

16. The global coastlines dataset • ~1:100 000 scale coastlines dataset (NGDC GSHHS) • Made by Hawaii University, centred on 180°E !! • Edited so mosaic centred on Greenwich meridian (ArcINFO processing CLIP, CLEAN, BUILD, UNION, etc) • Polygon for the sea mask extracted

17. The masking process • Vector polygon layer displayed in white (in most WMS clients, white is usually portrayed as transparent) • The background colour (red) shows the voids present in the SRTM. • The coastlines dataset has a lower resolution than SRTM, but sufficient to include many small islands (better than VMAP0 coastlines dataset -> 1km!)

18. SRTM shading & masking • Hillshaded and masked SRTM 1°x1° tiles • EU25: 1,118 files = 4.5GB • Uncompressed GeoTiff tiles • faster access • geo-info preserved • Index file created • Pyramidal structure to be accessed by the map-server • Lower res./bigger tiles created

19. Africa SRTM Masking SRTM tiles intersecting coastline shown for NE & S areas Inland tiles do not need masking DEM tiled in 4 regions (NW, NE, S, Azores) Azores included in Europe and Africa – political versus geographic affiliations

20. MapServer setup • Recently IONIC and Deegree servers were replaced by the University of Minnesota MapServer v 4.4.0-beta1 (http://mapserver.gis.umn.edu/) • UMMS pros • Compiles on most versions of UNIX/Linux, MacOS and Windows • Made available via a CGI application (no need to have Tomcat) • Very flexible data setup with the GDAL utility (no need to re-engineer data and easy to move to new server software) • Open source software supports WMS & WCS • DescribeCoverage request is supported on the WCS • WCS capable of easily serving 16-bit GeoTIFF coverages (SRTM) • WMS & WCS setup • WMS of hillshaded STRM pyramid & Landsat African stretched mosaic • WCS of 16-bit GEOTIFF SRTM and 8-bit Landsat mosaic • Web application interface same as IONIC & Deegree demo http://iceds.ge.ucl.ac.uk/cgi-bin/wms?map=wms.map&SERVICE=WMS&VERSION=1.1.1& REQUEST=GetCapabilities

21. MapServer versus IONIC & Deegree • IONIC in current pre-release WCS version cannot access a layer from a hierarchical directory of files • Requires a spatial image database (such as Oracle): beyond scope of ICEDS project • IONIC Red Spider is proprietary • Dedicated support, quicker development times • Cost! However, IONIC kindly donated licence(s). • Deegree is fussy, for example in the relationship between pyramid and full-resolution layers • Deegree 16-bit support inclusion would require major changes • MapServer has a very flexible data setup • Uses shape file as index • Data can be stored in disk in a simple directory structure or in a spatial database • Has utilities to ease data configuration: e.g. editing of configuration files

22. Connecting to the service

23. Current portal SRTM (unedited) data gaps visible in red

24. WCS client • Simple client based on the client/server architecture developed with IDL • Code is easily portable via an IDL virtual machine which is freely available • Allows on-line connection to the ICEDS WCS server • Signed 16-bit data retrieval • Display 3D perspective views of SRTM DEM • Vertical exaggeration control parameter • Possibility to choose desired area by BBox

25. WMS Connection from ArcGIS

26. WCS Connection from Cadcorp SIS

27. Comsine: WCS demonstration - orthorectification wcs_url = 'http://iceds.ge.ucl.ac.uk/cgi-bin/wcs?map=wcs.map&SERVICE=WCS&REQUEST=GetCoverage&COVERAGE=srtm&FORMAT=GEOTIFFINT16' res = max(tif_pixsize_x, tif_pixsize_y) wcsRequest = wcs_url + "&CRS=EPSG:" + str(epsg)+ "&BBOX=" + bbox + "&RESX=" + str(res)+ "&RESY=" + str(res) logger.info("WCS REQUEST " + wcsRequest) if(dem ==None): demFilename = str(time.clock())+ ".tif" urllib.urlretrieve(wcsRequest, filename) cleanup = 1

28. Current portal

29. Current portal

30. Current portal Additional annotation cascaded from GLOBE server

31. Current portal Cascaded layers only. GLOBE annotation plus JPL OnEarth Landsat 7 mosaic

32. Current portal JPL Landsat 5 low-res., stretched mosaic

33. Current portal Hill-shaded, colourised SRTM DEM

34. Current portal MODIS Blue Marble

35. Current portal Cascaded OnEarth Landsat 7 mosaic (NB: geoloc. shift c.f. SRTM)

36. Current portal JPL Landsat 5 full-res., stretched mosaic

37. Current portal

38. Current portal

39. ICEDS view of Cordoba

40. ICEDS optimal architecture Web Browser (Thin Client) Web Browser (Thin Client) Applications (Thick Client) Applications (Thick Client) Internet Internet • Integrated services: • - gazetteer • catalog • registry ICEDS Portal at http://iceds.ge.ucl.ac.uk portal host site host site WFS WMS WMS WCS Internet Internet Services Data UCL G.E. … ESA JPL … UN Data Providers

41. ICEDS Portal objectives • CEOS Geo-Portal publishing services compliant to OGC specs and ISOTC211 standards • Supports GSDI, ESDI, National, Regional and Local SDIs • Promotes migration to an environment that enables spatial data maintained by various collaborating communities to be readily accessed, integrated, fused and applied to support critical decision-making • Possible link to Reuters AlertNet humanitarian aid web mapping service • Free direct access to SRTM and Landsat data, through an easy and intuitive interface

42. ICEDS Next Phase • Next phase to address user needs and further develop the services – just begun • Main effort will run from Feb until Nov 2005, with final presentations in Jan 2006. • Publicise the service to user groups in Africa • Collate and analyse use cases, and extract requirements for further development • Parallel onward development of the services and portal • Accommodate new requirements for Web Service capability from use cases as budget, time and feasibility allows

43. Initial use cases • National Remote Sensing Agencies in Africa • Web Services for publishing of national geospatial data archives • Reuters Alertnet & NGOs • Evolution of the Alertnet WMS for improved map making and access to example data sets (e.g. those related to the recent Asian Tsunami event) • Awaiting input from Reuters • CEOS WGCV • Web Services to provide data of WGCV test sites • Addressed by Prof J-P Muller • Not restricted to African users • We are interested to hear from any group who use this service, and have opinions on how to improve it

44. Contacting African Users • WGISS contacts – Lorant and John • Links back to UNEP Nairobi, awaiting contact with UNEP • UNOOSA – David Stevens • Has agreed to support ICEDS project, and particularly likes the availability of the cookbook and scripts for local implementations • Has reservations about users ability to access the service, due to poor local infrastructure • AARSE – Jide Kufoniyi • Director of Regional Centre for Training in Aerospace Surveys (RECTAS), Obafemi Awolowo University Nigeria • Strongly supportive of the project and is propagating the ICEDS URL throughout AARSE membership, and other African user groups • UN FAO – Jelle Hielkema • Very interested in ICEDS in Tromso w.r.t. Madagascar, awaiting his input.

45. Candidate technology developments • Better printing support - online interactive map composition to enable high resolution PDF files to be produced for download to the client (combining any available WMS data sets) • Data exploration - additional WMS datasets added by a user, and manual annotation with standard menu of graphical icons • Support for identification of temporal changes (flickering between data sets, transparency, animation and flicker of multiple data sets in a user’s display list. To include display list management functions • Web Terrain Service – rendered imagery of 3D terrain • WGCV-TMSG extensions • Portal to link separate ICEDS-based services

46. Candidate Data Sets • Near-global SRTM JPL unedited DEM in WMS/WCS format (sourced from USGS) • Near-global SRTM JPL edited in WMS/WCS format (also USGS) • Near-global SRTM SAR amplitude derived water body mask in WMS format (source NGA to be supplied) • Near-global X-SRTM coverage map and colourised hill-shaded image mosaics in WMS format (source DLR) • Global ASTER stereo DEM coverage map in WMS format (source JPL/USGS) • Global ASTER stereo image (not yet processed to DEM) coverage map in WMS format (source JPL/USGS) • Global Landsat 5 Image mosaics (1”, bands 1-3, 4, where available (source JPL) • Coverage maps from other higher resolution sensors where available as WMS coverage. • Hot links to CEOS-WTF test site datasets (source EDC/GSFC)

47. Promotion plans • CEOS WGISS 18,19 and Plenary • African Journal of GeoInformation (AARSE Journal), and presentation to AARSE 2006 conference in Cairo. • UK EO Community – BNSC ICP2 workshop in November 2005 • Digital Earth – presentation to conference in Tokyo April 2005 • OGC technical and planning committee meeting in Frascati May/June 2005

48. Conclusions • Implementations of continental-scale high-resolution data serving using OGC technologies • Providing data & portrayal (WCS/WMS) services • User-selectable data freely and “instantly” available at full resolution • Guidelines, scripts and software for setting up your own server available now • Significantly more functionality and data to be added • Use cases with African users will be used to assess service and refine requirements for development • Initial user assessment very positive, but access limited for many users due to poor infrastructure • Updated service, guidelines and scripts/software will be made available September – November 2005