1 / 39

Metadata, Provenance and Web Service for Spatial Analysis -- the case of spatial weights

Metadata, Provenance and Web Service for Spatial Analysis -- the case of spatial weights. Luc Anselin, Sergio Rey, Wenwen Li GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University. Some Specific Project Goals

giolla
Download Presentation

Metadata, Provenance and Web Service for Spatial Analysis -- the case of spatial weights

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Metadata, Provenance and Web Service for Spatial Analysis --the case of spatial weights Luc Anselin, Sergio Rey, Wenwen Li GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University

  2. Some Specific Project Goals • Integrate and sustain a core set of composable, interoperable, manageable, and reusable CyberGIS software elements based on community-driven and open source strategies

  3. Challenge • most current spatial analysis/spatial econometrics software written for single CPU • rethink and rewrite analytical, algorithmic and processing facilities to integrate into a cyberinfrastructure • address lack of interoperability

  4. Spatial Econometrics Workbench • framework for supporting spatial econometric research in a cyberscience era (Anselin and Rey, IJGIS 2012) • Leverage PySALand CyberGIS • Support scientific workflow

  5. PySAL • open source library of Python routines for spatial analysis: geocomputation, spatial weights, spatial autocorrelation, spatial econometrics, regionalization • http://pysal.org • hosted on github

  6. PySAL Progress Report • current version is 1.6 (7th release) • 3.5 years of on-time bi-annual releases • 20,000+ downloads (10,000 in 2012) • recognized in open source scientific community - Anaconda

  7. Anaconda for big data analytics

  8. Migrating to CyberGIS • performance = need for parallelization + refined algorithms • interoperability = provide functionality as web services • replicability: need for metadata and provenance tracking

  9. Example: Spatial Weights • includes spatial data source, type of weights (e.g., contiguity, distance), any standardization or manipulation (e.g., higher order)

  10. Lack of Interoperability • different implementations • no standards • duplication of efforts • hinders interoperability and workflow chaining

  11. Example: Weights Formats in PySAL

  12. Example: PySAL spgreg what do we know about south_k6.gwt and south_ep_k20.kwt

  13. Conceptual Framework • separate data source from operations • data source: polygon or coordinate files with standard metadata (projection, origin, etc.) • operations: weights metadata

  14. weights vocabulary

  15. weights metadata structure (wmd)

  16. Web service implementation(OGC WPS) • wraps PySAL weights module • (re)creates weights object from information in wmd file • makes weights object available as a file

  17. wmd file (json) Weights Parser PySAL Dispatcher Weights Output Metadata Workflow

  18. Illustration

  19. Generate Weights from Shapefile • NAT.shp available on server • output format = gal

  20. Get Request • http://spatial.gdta.asu.edu/cgi-bin/wps.cgi?request=Execute&service=WPS&version=1.0.0&identifier=weights_ws&status=false&datainputs=[outputformat=gal;metadata={"input1":{"type":"shp","uri":"http://toae.org/pub/NAT.shp"},"weight_type":"rook","transform":"O","parameters":{"p":2,"k":4}}] metadata input

  21. Server Response

  22. Sample gal output http://spatial.gdta.asu.edu/wpsoutput/e66df128-14ed-11e3-bde9-0050455c0671.gal

  23. metadata (wmd) file http://spatial.gdta.asu.edu/wpsoutput/e66df128-14ed-11e3-bde9-0050455c0671.wmd

  24. Performance Evaluation • How does PySAL scale when the amount of input data increases? • Is the overhead of web service framework acceptable? • How does the web service framework scale in handling massive concurrent requests?

  25. Scale-up vs. Scale-out solution • Scale-up • High-end computer • Configuration • Processor  2 x 2.93 GHz Quad-Core Intel Xeon • Memory  16 GB 1066 MHz DDR3 ECC • Software  Mac OS X Lion 10.7.4 (11E53) • Scale-out: • Web server cluster

  26. Web Server Cluster

  27. Performance • experiment using grid layout for N = 10,000 to N = 100,000 • rook contiguity and k nearest neighbors (k = 4) • input shape files on server in Utah, web service on server at ASU

  28. Experiment 1 • Timing: average over 5 experiments • web server overhead, data transfer and computation • explore effect of data size

  29. time for rook and KNN contiguity

  30. Experiment 2 Scalability of web service framework High-end computer (8-cores) Cluster (4 computing nodes, each has 2-core) Total processing time Speed up

  31. Total processing time

  32. Speed-up

  33. Experiment 3 Scalability of the cluster by adding more computing nodes Average response time 128 concurrent requests Dataset: 10,000 polygons

  34. Scalability - cluster

  35. Next Steps

  36. Towards a Standard • refine specification: flexible, expandable, deal with edge cases • improve performance (parallelization) • implement seek operations on distributed files • interoperability with other software

  37. Thank you!

More Related