1 / 26

C. Meertens and J. McWhirter , UNAVCO S-J. S.Khalsa and T. Haran, NSIDC/CU

NASA ROSES ACCESS Project Review Science Data Systems for Satellite and Airborne LiDAR Data 23 February 2011. C. Meertens and J. McWhirter , UNAVCO S-J. S.Khalsa and T. Haran, NSIDC/CU C. Baru and C. Crosby, SDSC/UCSD D. Harding and B. Blair, NASA/GSFC and M. Hofton, U. Maryland.

zaria
Download Presentation

C. Meertens and J. McWhirter , UNAVCO S-J. S.Khalsa and T. Haran, NSIDC/CU

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NASA ROSES ACCESS Project ReviewScience Data Systems for Satellite and Airborne LiDAR Data23 February 2011 C. Meertens and J. McWhirter, UNAVCO S-J. S.Khalsa and T. Haran, NSIDC/CU C. Baru and C. Crosby, SDSC/UCSD D. Harding and B. Blair, NASA/GSFC and M. Hofton, U. Maryland ACCESS: Advancing Collaborative Connections for Earth System Science

  2. Science Data Systems for Satellite and Airborne LiDAR Data PI: Charles Meertens, UNAVCO • Provide an integrated system (NLAS) for web services-based access to distributed NASA airborne (LVIS) and satellite (ICESAT/GLAS) LiDAR data archives using OpenTopography technologies. • Define enhanced quality control (QC) metrics and a standard data format to improve evaluation of data quality, data usability, and compatibility with commercial LiDAR software packages. • Provide users access to point cloud and waveform LiDAR data, on-demand digital elevation models (DEMs) generation, and derivative products and visualizations. Portal provides user access to LiDAR point cloud, standard and on-demand digital elevation models generation. • Collaborate to define an industry-based LiDAR data format for NASA LiDAR data; to define enhanced QC metrics; and to design web services. • Develop NSIDC ICESAT/GLAS web service. • Develop UNAVCO/GSFC LVIS data repository system and web service. • Provide integrated user access using OpenTopography Portal. • Science partner testing of enhanced data and QC products and portal access. • Complete data format, QC, and web services • definitions. 12/10 • Implementation of Data Web Services. 04/11 • Beta release of NLAS/OpenTopography Portal • with connectivity to NSIDC and UNAVCO. 10/11 • Production release of NLAS Portal. 03/12 S-J. S.Khalsa, NSIDC/CU; C.Baru, SDSC/UCSD; D.Harding, B.Blair, NASA/GSFC; and M.Hofton, U.MD. TRLin=5 TRLcurrent=5 19 April 2010

  3. LiDAR Data Platforms Distributed and Large Collections from a diverse set of LiDAR Platforms 5 million points/day 10 km altitude 10 m footprint ~500 thousand points/day 600 km altitude 70 m footprint High Altitude Airborne Laser Vegetation Imaging System ( LVIS) Satellite - Icesat 500 million-1 billion/day 0.5 km altitude 20 cm footprint 100-500 million points/day tripod 1 cm footprint Low Altitude Airborne Laser Scanning Terrestrial LiDAR Scanning

  4. NLAS Project Overview Objectives of NLAS Project (NASA Laser Access System) • Provide an integrated system (NLAS) for web services-based access to distributed NASA airborne (LVIS) and satellite (ICESAT/GLAS) LiDAR data archives using OpenTopography technologies. • Define enhanced quality control (QC) metrics and a standard data format to improve evaluation of data quality, data usability, and compatibility with commercial LiDAR software packages. • Provide users access to point cloud and waveform LiDAR data, on-demand digital elevation models (DEMs) generation, and derivative products and visualizations. Public Website (search for “NLAS Opentopography”) Hosted at OpenTopography Project Collaborative Wiki (uses Google sites nlaswiki)

  5. Project Partners UNAVCO PI: Charles Meertens, Project management Software Engineer: Jeff McWhirter, Web services, API, and UI development San Diego Supercomputing Center, University of California, San Diego Co-PI: Chaitan Baru, Project management support Science Specialist: Chris Crosby, Algorithm Development, Science Evaluation Software Engineers: Vishu Nandigam, Minh Phan Web services, API, and UI development National Snow and Ice Data Center, NSIDC, CU Co-PI: Siri Jodha Singh Khalsa, Project management support, ESDSWG Software Engineers: Terry Haran, Nitin Khanna Web services, API, UI and Visualization development Goddard Space Flight Center Co-PI: David Harding and Collaborator: Bryan Blair Define data format, QA/QC, generate new products, science evaluation University of Maryland Collaborator: Michelle Hofton Define data format, generate new products, science evaluation

  6. Major Accomplishments Major Accomplishments • AGU NLAS presentation and NASA booth demo with end-to-end proof of concept for LVIS, ICESat GLAS, and TLS data • Ingest of select LVIS and ICESAT/GLAS data sets into Open Topography portal • Definition of initial new GLAS GLANL format and LVIS Comprehensive format • NLAS/RAMADDA supports LVIS, LAS, GLAS and ASCII data formats as well as aggregations of those files • Provisional support for new ICEBRIDGE data sets – ATM, MCORDS, PARIS • OpenTopography federated web service query to NLAS/RAMADDA for remote data repository collection summary metadata using ATOM caster. • Public OpenTopography website at http://www.opentopography.org/index.php/about/nlas (sample LVIS collections, LVIS point cloud, DEM service)

  7. Major Milestones Major Milestones • Complete data format, QC, and web services definitions. 12/10 • Draft definitions • Final definitions will evolve in stages over next two quarters • Develop file conversion software for LVIS, GLAS 12/10 • Note major step to use NLAS/RAMADDA to provide common data repository and richer web services for LVIS and GLAS • Sample data sets ingested in OT; Presentations at 2010 AGU 04/11 • Implementation of Data Web Services. 04/11 • Prototype • Catalog of services (connectivity between UNAVCO/SDSC OT) • Full data service implementation at NSIDC, OT • Beta release of NLAS/OpenTopography Portal • with connectivity to NSIDC and UNAVCO. 10/11 • Production release of NLAS Portal. 03/12

  8. Project Schedule Summary Project Schedule Summary • Prototyping of NLAS workflow from data repositories to OT samples for LVIS and GLAS: slightly ahead of schedule • Expedited by decision to utilize RAMADDA framework for data repositories and universal, flexible data translation web service • Definition of data formats and enhanced QC/QA: on schedule • Definition of full data web services API and implementation : behind schedule • Processing of complete datasets into comprehensive formats: behind schedule • Science partner utilization of NLAS services: slightly behind schedule • Publicizing NLAS project: ahead of schedule

  9. NLAS System Overview • OpenTopography Currently: • Provides users access to ten’s of billions of LiDAR point cloud measurements and in industry standard formats (e.g. ascii, LAS) • Provides on-demand processing services to enable generation of digital elevation models (DEMs), derivative products and visualizations. • NLAS will provide: • Web Services access to distributed archives using a downloadable data repository package (NLAS/RAMADDA) • Integrated access to NLAS data via OpenTopography • An extensible Java package for reading and writing various LiDAR formats • Add full waveform access and enhanced quality control (QC) metrics for ICESat/GLAS and LVIS USERS OpenTopography NLAS RAMADDA LVIS ICESat/GLAS

  10. Open Topography • OpenTopography Search • Map Interface • Metadata about Dataset • Access to download and processing interface • Public website http://www.opentopography.org/index.php/about/nlas OpenTopography is funded through the National Science Foundation

  11. Open Topography – Data Access Data Downloading and Processing User specifies Download/Processing Parameters 1.) Select area from Map Interface 2.) Point Cloud - Select format (ASCII or LAS) 3.) DEM Generation -Select Gridding Method, parameters, and format

  12. Open Topography - Results Page • OpenTopography • On-demand Processing Products: • Subset of point cloud data • Digital Elevation (DEM) Map • Visualization product and .kmz file if desired • DEM on browser embedded GoogleEarth LVIS example in Sierra Nevada

  13. Open Topography - GLAS LiDAR Examples • ICESat/GLAS example • Subset area of interest (below left) • Download ASCII Lat, Long, Elev (X,Y,Z) • Example of subsequent import into Excel and plot of shot number vs elevation (below right) OT query of NLAS/RAMADDA product catalog using web services with ATOM casting just completed (see beta server) Beta Server at: http://ot-beta.sdsc.edu:8080/gridsphere/gridsphere?cid=datasets

  14. Open Topography - Other LiDAR Examples • Terrestrial Laser Scanning example (NSF OT Development) • Subset area of interest • Download options include: ASCII Point Cloud, DEM, Hillshade & Slope Products, kmz, and image • Note that UNAVCO will implement TLS into RAMADDA for NSF

  15. Interoperability with Data Repository Web Service Open Topography Visualization/Analysis Clients NLAS Interoperability HTML Interface NLAS Web Service NLAS/RAMADDA NLAS Web Service

  16. NLAS – ICESat/GLAS GLANL/QCQA • ICESat Consolidated and Enhanced Altimetry Data Structure • Consolidation of parameters from 8 GLA product files • Eliminating numerous redundant, invalid and unused parameters • Producing a single data record for each laser shot • Selection of subsets from full ICESat GLA product files • Currently ~160 unique parameters in products • NLAS/GLANL 100 parameters in “long” subset for advanced users • NLAS/GLANL 18 parameters in “short” subset for casual users • Definition of 25 additional derived parameters for improved ease of use • e.g., Conversion of waveform-derived relative range offsets to elevations • Definition of 6 QA/QC parameters to guide selection of appropriate data • Source of return: Earth surface, cloud, noise, indeterminate • Cause when no geolocation result reported: no return detected, instrument anomaly • Non-nadir data acquired during spacecraft maneuver • Possible presence of forward scattering range delay • Qualitative range precision assessment from S/N and signal fitting residual • Data reliability: composite yes/no flag for 8 parameters effecting quality

  17. NLAS/RAMADDA RAMADDA Overview • Data repository framework developed at Unidata/UCAR • Freely available Java based open source package • Extensible plugin architecture • Basic content management functionality NLAS/RAMADDA plugin to support LiDAR data services • LiDAR data ingest, metadata harvesting and publication • Services include: • Map overview, time series and waveform displays • Spatial subset, decimation • Conversion to multiple data format: original binary, LAS, ASCII CSV, … • Product generation - Image, hill shade image, KMZ

  18. NLAS Data Repository – LVIS Example Collection Query Open Topography Portal NLAS/RAMADDA (LVIS or GLAS – same package) Web services • Spatial queries • Subsetting • Data Conversion • Product generation ATOM Response Metadata Database Data Request LAS, ASCII CSV, etc. Metadata harvesting LiDAR Tools File store Ingest New Comprehensive NLAS/LVIS Format (50 parameter) LVIS Data System LVIS Product Generation Other Clients

  19. NLAS Data Repository – ICESat/GLAS Example Collection Query Open Topography Portal NLAS/RAMMADA Web services • Spatial queries • Subsetting • Data Conversion • Product generation ATOM Response Metadata Database Data Request LAS, ASCII CSV, etc. Metadata harvesting LiDAR Tools File store Ingest New GLANL “long” 100 para “short” 18 para +25 additional para +6 QA/QC ICESat/GLAS Data System GLAS Data Pool Other Clients

  20. NLAS/LiDAR Tools NLAS/LiDAR Tools Stand-alone Extensible Java package for reading and writing various LiDAR formats (made available via SourceForge) • LAS • LVIS (including waveform) • ICESAT/GLAS products • Other Icebridge formats – ATM, MCORDS, PARIS • Terrestrial Laser Scanning formats – text, PTX • Transforms global XYZ to Lat, Long, Elevation • …

  21. LiDAR Subset Form Data Access direct from NLAS/RAMADDA (may run demo instead if time allows)

  22. LiDAR Waveform Data Full Waveform LiDAR Data Transmit Pulse Alternate Signal Start Outer Canopy Relief Canopy Inflection Standard Gaussian Fit Within Canopy Light Extinction Ground Relief Alternate Signal End 1064 nm laser pulse footprint ~ 70 m diameter spaced 175 m along profile Amplitude (volts)

  23. NLAS/RAMADDA Waveform Display NLAS/RAMADDA Waveform Display

  24. Project Metrics Metrics Plan • Custom and web log statistics from OpenTopography • http://www.opentopography.org/index.php/about/pcmetrics • NLAS/RAMADDA Repository implementation includes web service API logging • Web log analysis software will be used to parse and summarize information on service requests (e.g. Domain analysis such as IP address, unique visitors, volume of data transferred, number of points processed, jobs executed) • UNAVCO will aggregate repository statistics and will be responsible for upload to NASA metrics system • Additional impact and software reuse metrics separate from the web logs are planned • OT has guest user (no login) and login access that allows increased processing limits and a custom workspace (mylidarjobs). NLAS will determine login and services access requirements and apply to RAMADDA and OT. NLAS will also need to pass through user id or IP through web service API and will bring this up with broader ESDSWG. • Example • 2008 LVIS Sierra Nevada, CA: • Dec 2010 9 jobs 23,341,301 points • Jan 2011 4 jobs 2,186,947 points • Feb 2011 11 jobs 14,797,042 points

  25. ESDSWG Participation ESDSWG Participation • Siri Jodha Khalsa is the representative to the ESDSWG • Highlights: • Standards Process Group (SPG) activities • Participation in monthly teleconferences • Service on two SPG Technical Working Groups evaluating proposed standards and reviewer on third. • New Orleans meeting in October, 2010 (McWhirter also attended) • Incorporated Atom-based Service Cast output format into the NLAS web service implementation. • Feedback planned to ESDSWG: • Federated distributed search/processing from NLAS Repositories • Web service pass-through of user id/IP • Khalsa is tracking developments of a proposed US national lidar data encoding standard and will provide feedback to ESDSWG.

  26. Status and Next Steps Accomplishments • LiDAR data repository framework • NLAS/RAMADDA Data Repository • LiDAR Data translation software tool • End-to-end proof of concept • LVIS, ICESat GLAS, TLS • New GLAS product • GLANL containing enhanced QC Some Next Steps • Define NLAS web service API • Data and metadata discovery • Data service access • Implement NLAS production servers for GLAS and LVIS data sets • Implement full wave form service into OpenTopography • Explore potential application of NLAS services to other data sets – ATM, TLS, etc. GSFC LVIS Antarctic Glacier Scan

More Related